best practice for thread pool queue size

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

best practice for thread pool queue size

Zaki Agha
Hi
We have several elastic search clusters
Recently we faced an issue in which one of our nodes experienced queueing.
In fact, the queue length was greater than 1000.
Subsequent requests were rejected as the queue was full.

Should we increase the default queue size?

I understand that there are several queue's within elastic search.

  1. Queues in Elastic Search

    1. Index            - default 200

    2. Bulk              - default 50

    3. Get               - default 1000

    4. Search         - default 1000

    5. Suggest       - default 1000

    6. Percolate     - default 1000

    7. ThreadPool queue_size:  1000




Errors:

  1. Error # 1

[[LApp45][SiyuJOHVRRG1udLiFwM9Yw][es1][inet[/xxx.xxx.xxx.xxx:9300]]], id [84124759]

[2014-07-13 04:13:35,332][WARN ][transport                ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[2014-07-13 04:13:35,332][WARN ][transport                ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[[LApp37][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]], id [80874233]

  1. Error # 2

[2014-07-13 06:28:26,043][WARN ]

[transport                ]

[es2] Received response for a request that has timed out,

sent [55795ms] ago,

timed out [25795ms] ago,

action [discovery/zen/fd/ping],

node

  1. Error # 3

[2014-07-13 06:28:26,049][WARN ][transport                ]

[es2] Received response for a request that has timed out,

sent [56023ms] ago,

timed out [26023ms] ago,

action [discovery/zen/fd/ping],

node [[es3][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]], id [84124758]

  1. Error # 4

There are several errors of this type all for the same index aggregated_user_game_points

[2014-07-13 06:28:26,153][DEBUG][action.search.type       ]

[es2] [aggregated_user_game_points][3], node[8qI5LGo2TxG1S-mQUgEA_w], [P], s

[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3367563e] lastShard [true]

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on                   

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4@71bd1bf

<omitted the rest of the error message>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: best practice for thread pool queue size

Mark Walkom
Increasing queues isn't going to help if there are underlying problems stopping the processing.

Based on those errors it looks like you may have network issues, but they are from July?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email]
web: www.campaignmonitor.com

On 14 October 2014 08:16, Zaki Agha <[hidden email]> wrote:
Hi
We have several elastic search clusters
Recently we faced an issue in which one of our nodes experienced queueing.
In fact, the queue length was greater than 1000.
Subsequent requests were rejected as the queue was full.

Should we increase the default queue size?

I understand that there are several queue's within elastic search.

  1. Queues in Elastic Search

    1. Index            - default 200

    2. Bulk              - default 50

    3. Get               - default 1000

    4. Search         - default 1000

    5. Suggest       - default 1000

    6. Percolate     - default 1000

    7. ThreadPool queue_size:  1000




Errors:

  1. Error # 1

[[LApp45][SiyuJOHVRRG1udLiFwM9Yw][es1][inet[/xxx.xxx.xxx.xxx:9300]]], id [84124759]

[2014-07-13 04:13:35,332][WARN ][transport                ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[2014-07-13 04:13:35,332][WARN ][transport                ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[[LApp37][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]], id [80874233]

  1. Error # 2

[2014-07-13 06:28:26,043][WARN ]

[transport                ]

[es2] Received response for a request that has timed out,

sent [55795ms] ago,

timed out [25795ms] ago,

action [discovery/zen/fd/ping],

node

  1. Error # 3

[2014-07-13 06:28:26,049][WARN ][transport                ]

[es2] Received response for a request that has timed out,

sent [56023ms] ago,

timed out [26023ms] ago,

action [discovery/zen/fd/ping],

node [[es3][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]], id [84124758]

  1. Error # 4

There are several errors of this type all for the same index aggregated_user_game_points

[2014-07-13 06:28:26,153][DEBUG][action.search.type       ]

[es2] [aggregated_user_game_points][3], node[8qI5LGo2TxG1S-mQUgEA_w], [P], s

[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3367563e] lastShard [true]

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on                   

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4@71bd1bf

<omitted the rest of the error message>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YdD2vWb-aPj2ySqR%2B%3D8K0fH7NTFjHjF426Gvcmj4mKUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: best practice for thread pool queue size

Zaki Agha
Yes the particular error is from July.
How can I determine the optimal setting for queue size?

On Monday, October 13, 2014 3:21:32 PM UTC-7, Mark Walkom wrote:
Increasing queues isn't going to help if there are underlying problems stopping the processing.

Based on those errors it looks like you may have network issues, but they are from July?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LbyuT_YorUYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">ma...@...
web: <a href="http://www.campaignmonitor.com" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.campaignmonitor.com\46sa\75D\46sntz\0751\46usg\75AFQjCNFv30c-WBiP6sfBmxXaWBP5YBZg1Q';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.campaignmonitor.com\46sa\75D\46sntz\0751\46usg\75AFQjCNFv30c-WBiP6sfBmxXaWBP5YBZg1Q';return true;">www.campaignmonitor.com

On 14 October 2014 08:16, Zaki Agha <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="LbyuT_YorUYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">za...@...> wrote:
Hi
We have several elastic search clusters
Recently we faced an issue in which one of our nodes experienced queueing.
In fact, the queue length was greater than 1000.
Subsequent requests were rejected as the queue was full.

Should we increase the default queue size?

I understand that there are several queue's within elastic search.

  1. Queues in Elastic Search

    1. <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html#modules-threadpool" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-threadpool.html%23modules-threadpool\46sa\75D\46sntz\0751\46usg\75AFQjCNHIfX4aI4qtVACgbOHobhyl-tf1Ww';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-threadpool.html%23modules-threadpool\46sa\75D\46sntz\0751\46usg\75AFQjCNHIfX4aI4qtVACgbOHobhyl-tf1Ww';return true;">Index            - default 200

    2. Bulk              - default 50

    3. Get               - default 1000

    4. Search         - default 1000

    5. Suggest       - default 1000

    6. Percolate     - default 1000

    7. <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html#_literal_fixed_literal" style="font-family:Arial,Helvetica,sans-serif" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-threadpool.html%23_literal_fixed_literal\46sa\75D\46sntz\0751\46usg\75AFQjCNFmvFJXVpMSMYF7M2agXMkE-1Uxbw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-threadpool.html%23_literal_fixed_literal\46sa\75D\46sntz\0751\46usg\75AFQjCNFmvFJXVpMSMYF7M2agXMkE-1Uxbw';return true;">ThreadPool queue_size:  1000




Errors:

  1. Error # 1

[[LApp45][SiyuJOHVRRG1udLiFwM9Yw][es1][inet[/xxx.xxx.xxx.xxx:9300]]], id [84124759]

[2014-07-13 04:13:35,332][WARN ][transport                ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[2014-07-13 04:13:35,332][WARN ][transport                ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[[LApp37][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]], id [80874233]

  1. Error # 2

[2014-07-13 06:28:26,043][WARN ]

[transport                ]

[es2] Received response for a request that has timed out,

sent [55795ms] ago,

timed out [25795ms] ago,

action [discovery/zen/fd/ping],

node

  1. Error # 3

[2014-07-13 06:28:26,049][WARN ][transport                ]

[es2] Received response for a request that has timed out,

sent [56023ms] ago,

timed out [26023ms] ago,

action [discovery/zen/fd/ping],

node [[es3][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]], id [84124758]

  1. Error # 4

There are several errors of this type all for the same index aggregated_user_game_points

[2014-07-13 06:28:26,153][DEBUG][action.search.type       ]

[es2] [aggregated_user_game_points][3], node[8qI5LGo2TxG1S-mQUgEA_w], [P], s

[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3367563e] lastShard [true]

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on                   

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4@71bd1bf

<omitted the rest of the error message>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LbyuT_YorUYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/12e46524-3f8f-4a1e-90d7-5ae4f4c3a191%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.