Queue Size limited?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Queue Size limited?

Abid Hussain
Hi all,

last restart of our cluster caused a heavy load as a couple of rivers are triggered which load alltogether about 40 GB of data into the (single-node) cluster.

This lead to massive exceptions like shown in the logs:
[6726]: index [my_index], type [my_doc_type], id [2390342], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@4a94944]]

We didn't change any of the threadpools queue size settings (like threadpool.bulk.queue_size). AFAIK in this case the queue size is unlimited.

So how is it possible that the exception above can occur? Why does the log message indicate a queue size of 50?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7f095b58-e471-43a7-9d20-d35dabd1a697%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Queue Size limited?

joergprante@gmail.com
Is this with JDBC river? If so, you can increase the number of bulk actions per request, to decrease the concurrent bulk actions, to remedy situations where many rivers are active.

Jörg

On Mon, Feb 23, 2015 at 9:40 AM, Abid Hussain <[hidden email]> wrote:
Hi all,

last restart of our cluster caused a heavy load as a couple of rivers are triggered which load alltogether about 40 GB of data into the (single-node) cluster.

This lead to massive exceptions like shown in the logs:
[6726]: index [my_index], type [my_doc_type], id [2390342], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@4a94944]]

We didn't change any of the threadpools queue size settings (like threadpool.bulk.queue_size). AFAIK in this case the queue size is unlimited.

So how is it possible that the exception above can occur? Why does the log message indicate a queue size of 50?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7f095b58-e471-43a7-9d20-d35dabd1a697%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE6wXT%2BstLnyadVxXvi0-pzycMn9Nfbjv7AR7eABjaw7Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Queue Size limited?

Abid Hussain
Yes it happened with jdbc river. Thanks for clarification.

Am Montag, 23. Februar 2015 17:16:09 UTC+1 schrieb Jörg Prante:
Is this with JDBC river? If so, you can increase the number of bulk actions per request, to decrease the concurrent bulk actions, to remedy situations where many rivers are active.

Jörg

On Mon, Feb 23, 2015 at 9:40 AM, Abid Hussain <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="CCBUjlFmW9EJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hus...@...> wrote:
Hi all,

last restart of our cluster caused a heavy load as a couple of rivers are triggered which load alltogether about 40 GB of data into the (single-node) cluster.

This lead to massive exceptions like shown in the logs:
[6726]: index [my_index], type [my_doc_type], id [2390342], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@4a94944]]

We didn't change any of the threadpools queue size settings (like threadpool.bulk.queue_size). AFAIK in this case the queue size is unlimited.

So how is it possible that the exception above can occur? Why does the log message indicate a queue size of 50?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="CCBUjlFmW9EJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/7f095b58-e471-43a7-9d20-d35dabd1a697%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/7f095b58-e471-43a7-9d20-d35dabd1a697%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/7f095b58-e471-43a7-9d20-d35dabd1a697%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/7f095b58-e471-43a7-9d20-d35dabd1a697%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b7edddf8-a619-4a3d-a25f-8f8a40af3e36%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.