Limit large number of threads

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Limit large number of threads

Abid Hussain
Hi all,

I know I'm not the first one wondering about the number of threads but I didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is actually (according to what bigdesk says):
* Search 72
* Index 24
* Bulk 24
* Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Limit large number of threads

Lukáš Vlček
Hi,

please note that Bigdesk does not show all internal thread pools in ES now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <[hidden email]> wrote:
Hi all,

I know I'm not the first one wondering about the number of threads but I didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is actually (according to what bigdesk says):
* Search 72
* Index 24
* Bulk 24
* Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO9cvUbbvJBv%3DVnE1cOTRuGQFBEBJ9rYBETwO99NzJn3kKN%2BmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Limit large number of threads

Abid Hussain
Thanks for clarification. 

Still I wonder why such a huge amount of thread is created and if this can lead to issues - especially as it seems to me that they are never released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:
Hi,

please note that Bigdesk does not show all internal thread pools in ES now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Fxr_vI3GYzAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hus...@...> wrote:
Hi all,

I know I'm not the first one wondering about the number of threads but I didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is actually (according to what bigdesk says):
* Search 72
* Index 24
* Bulk 24
* Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="Fxr_vI3GYzAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Limit large number of threads

joergprante@gmail.com
If thread counts go out of bounds, it may be a lockup somewhere. What version of ES do you use?

Jörg

On Fri, Mar 20, 2015 at 2:08 PM, Abid Hussain <[hidden email]> wrote:
Thanks for clarification. 

Still I wonder why such a huge amount of thread is created and if this can lead to issues - especially as it seems to me that they are never released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:
Hi,

please note that Bigdesk does not show all internal thread pools in ES now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <[hidden email]> wrote:
Hi all,

I know I'm not the first one wondering about the number of threads but I didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is actually (according to what bigdesk says):
* Search 72
* Index 24
* Bulk 24
* Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF_OoTYTK7WT1gWhcMkmvy3mqpWtVHZrp3-F8Voqhkahw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Limit large number of threads

Mark Walkom-2
In reply to this post by Abid Hussain
Each segment, each network connection and a whole bunch of other things will add to this.
280 is a very low number and I wouldn't even worry about it.

On 20 March 2015 at 06:08, Abid Hussain <[hidden email]> wrote:
Thanks for clarification. 

Still I wonder why such a huge amount of thread is created and if this can lead to issues - especially as it seems to me that they are never released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:
Hi,

please note that Bigdesk does not show all internal thread pools in ES now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <[hidden email]> wrote:
Hi all,

I know I'm not the first one wondering about the number of threads but I didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is actually (according to what bigdesk says):
* Search 72
* Index 24
* Bulk 24
* Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-uV%3DWf3VAzGuZL6rC_7iQA6-%3DpXrn%2Bn5bT%2BFFuggrCJw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Limit large number of threads

Abid Hussain
In reply to this post by joergprante@gmail.com
We're using 1.4.2 currently. Are there any issues known with this version?

If there is nothing to worry about 400 threads, it's ok for me. I just wonder why they never seem to be released.

Am Freitag, 20. März 2015 16:20:03 UTC+1 schrieb Jörg Prante:
If thread counts go out of bounds, it may be a lockup somewhere. What version of ES do you use?

Jörg

On Fri, Mar 20, 2015 at 2:08 PM, Abid Hussain <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="6FsxnguL-h8J" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hus...@...> wrote:
Thanks for clarification. 

Still I wonder why such a huge amount of thread is created and if this can lead to issues - especially as it seems to me that they are never released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:
Hi,

please note that Bigdesk does not show all internal thread pools in ES now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <[hidden email]> wrote:
Hi all,

I know I'm not the first one wondering about the number of threads but I didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is actually (according to what bigdesk says):
* Search 72
* Index 24
* Bulk 24
* Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="6FsxnguL-h8J" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e13d3eb5-f90d-45e3-ab24-901e1b0c284a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Limit large number of threads

joergprante@gmail.com
In reply to this post by Mark Walkom-2
Hm, I doubt it is ok if a 1.4.0 node has 195 threads in state BLOCKED:

Thread 19374: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Compiled frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=162 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Compiled frame)

Jörg


On Fri, Mar 20, 2015 at 4:30 PM, Mark Walkom <[hidden email]> wrote:
Each segment, each network connection and a whole bunch of other things will add to this.
280 is a very low number and I wouldn't even worry about it.

On 20 March 2015 at 06:08, Abid Hussain <[hidden email]> wrote:
Thanks for clarification. 

Still I wonder why such a huge amount of thread is created and if this can lead to issues - especially as it seems to me that they are never released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:
Hi,

please note that Bigdesk does not show all internal thread pools in ES now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <[hidden email]> wrote:
Hi all,

I know I'm not the first one wondering about the number of threads but I didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is actually (according to what bigdesk says):
* Search 72
* Index 24
* Bulk 24
* Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-uV%3DWf3VAzGuZL6rC_7iQA6-%3DpXrn%2Bn5bT%2BFFuggrCJw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE4t%3DfgaE%3D9SW3dX08U5nUzpXLPi9T0iZsvbEwqfEKdHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Limit large number of threads

joergprante@gmail.com
In reply to this post by Abid Hussain
I think you should check a thread dump created by tools like jstack if you have a high JVM thread count in state BLOCKED. This might be a pointer that something unusual is going on, but I'm not sure.

Jörg


On Fri, Mar 20, 2015 at 4:41 PM, Abid Hussain <[hidden email]> wrote:
We're using 1.4.2 currently. Are there any issues known with this version?

If there is nothing to worry about 400 threads, it's ok for me. I just wonder why they never seem to be released.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFZaDaBTRw_hY9%3DaJmrXCPqfO4i%2B4sOU1Svy1%2BJbHCJNg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Limit large number of threads

Abid Hussain
Thanks Jörg, I did a thread dump: 60 % of ~400 threads are in state WAITING, 35 % are in state RUNNABLE, the rest is in state TIMED_WAITING, none is in state BLOCKED.

So I assume everything is OK - still wondering whats the point of creating hundreds of threads as there are "only" 24 cores available on our machine.

Best regards,

Abid

Am Freitag, 20. März 2015 16:47:28 UTC+1 schrieb Jörg Prante:
I think you should check a thread dump created by tools like jstack if you have a high JVM thread count in state BLOCKED. This might be a pointer that something unusual is going on, but I'm not sure.

Jörg


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e0cef7f2-9f38-4c16-806d-45fbd3dc14a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Limit large number of threads

joergprante@gmail.com
ES uses several threadpools. Some are fixed sized, some are scalable, and the reference is the JVM available core count, i.e. Runtime.getRuntime().availableProcessors(), which can be overridden by a "processors" directive:


I have 32 cores per machine so I'm also observing hundreds of idle threads. I do not use percolators/warmers/snapshot/suggest in my use case. Since they are not active, they steal a lot of my stack memory. Right now I don't care about idle threads but this can be a problem for small memory sized machines.

If you feel safe about reducing thread pool resources, you can decrease them in your config, e.g. -Des.processors=4. But you should monitor your performance, if you see bad numbers, go back to default.

Jörg



On Mon, Mar 23, 2015 at 8:48 AM, Abid Hussain <[hidden email]> wrote:
Thanks Jörg, I did a thread dump: 60 % of ~400 threads are in state WAITING, 35 % are in state RUNNABLE, the rest is in state TIMED_WAITING, none is in state BLOCKED.

So I assume everything is OK - still wondering whats the point of creating hundreds of threads as there are "only" 24 cores available on our machine.

Best regards,

Abid

Am Freitag, 20. März 2015 16:47:28 UTC+1 schrieb Jörg Prante:
I think you should check a thread dump created by tools like jstack if you have a high JVM thread count in state BLOCKED. This might be a pointer that something unusual is going on, but I'm not sure.

Jörg


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e0cef7f2-9f38-4c16-806d-45fbd3dc14a1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF%2B2jrzPH6w%2BHa9v6ixJXOYS8Rdu_%3DMdRc5PVDG6%3DK8-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.