Very high CPU usage of ES in idle state

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Very high CPU usage of ES in idle state

Talha Khan
I have set up a 1 node cluster of ES using ES version 0.19.11 for storing some logs. After inserting about 5000 logs I now see that my CPU usage is rocketing above 100% constanltly even when no/very few search queries are reaching it. ES has become very unresponsive and some of my search queries randomly return the following exception.
 
"error" : "EsRejectedExecutionException[rejected execution of [org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]",
  "status" : 503

By randomly i mean sometimes they succeed but mostly they fail.


Here is my ES config:

index.cache.field.expire: 20m
index.cache.field.type: soft

bootstrap.mlockall: true
threadpool:
   search:
       type: fixed
       size: 20
       queue_size: 1000
   index:
       type: fixed
       size: 16
   bulk:
       type: fixed
       size: 5


I did some digging into the issue. The stats for node_stats, hot_threads api and jstack can be found in this gist

https://gist.github.com/ahmedtalhakhan/5343784

I am also attaching the results of bigdesk. The cluster health is green but looking at these results, it seems that ES is very low on heap memory. But i am unable to understand the constant high CPU usage even in the idle state. How can this issue be solved? 


 Kindly give some context and explanation to any answer.

Thanks
-atk


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

bigdesk_es .jpg (258K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Very high CPU usage of ES in idle state

joergprante@gmail.com
Note, for your workload of around 100 qps, I have the impression you
have selected small memory for heap, around 350MB (?), and your GC is
around 10-20 seconds, but your RAM is total of 2 GB. This indicates you
should consider increasing the heap. GC times are going through the roof
and they give you the high CPU and unresponsiveness.

On a machine of 2 GB RAM there is a higher risk of bootstrap.mlockall:
true to starve your overall system processes because you will quickly
force them into swapping/paging, or you run out of real memory. So
consider removing this option to give the system enough room or upgrade
to a machine with > 4 GB RAM. Another option is adding more nodes of course.

If you stay with 2 GB RAM I would also recommend to disable 64-bit
server JVM with "java -d32 -client" in order to save some bits on the
heap. 64-bit server JVM makes sense on machines where you can use
virtual address space in the area of > 4GB RAM.

Jörg

Am 09.04.13 12:07, schrieb Talha Khan:

> I have set up a 1 node cluster of ES using ES version 0.19.11 for
> storing some logs. After inserting about 5000 logs I now see that my
> CPU usage is rocketing above 100% constanltly even when no/very few
> search queries are reaching it. ES has become very unresponsive and
> some of my search queries randomly return the following exception.
> "error" : "EsRejectedExecutionException[rejected execution of
> [org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]",
>   "status" : 503
>
> By randomly i mean sometimes they succeed but mostly they fail.
>
>
> Here is my ES config:
>
> index.cache.field.expire: 20m
> index.cache.field.type: soft
>
> bootstrap.mlockall: true
> threadpool:
>    search:
>        type: fixed
>        size: 20
>        queue_size: 1000
>    index:
>        type: fixed
>        size: 16
>    bulk:
>        type: fixed
>        size: 5
>
>
> I did some digging into the issue. The stats for node_stats,
> hot_threads api and jstack can be found in this gist
>
> https://gist.github.com/ahmedtalhakhan/5343784
>
> I am also attaching the results of bigdesk. The cluster health is
> green but looking at these results, it seems that ES is very low on
> heap memory. But i am unable to understand the constant high CPU usage
> even in the idle state. How can this issue be solved?
>
>
>  Kindly give some context and explanation to any answer.
>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Very high CPU usage of ES in idle state

Talha Khan
I will try out the recommendations that you suggested. Will post my results.

On Tuesday, 9 April 2013 19:27:04 UTC+5, Jörg Prante wrote:
Note, for your workload of around 100 qps, I have the impression you
have selected small memory for heap, around 350MB (?), and your GC is
around 10-20 seconds, but your RAM is total of 2 GB. This indicates you
should consider increasing the heap. GC times are going through the roof
and they give you the high CPU and unresponsiveness.

On a machine of 2 GB RAM there is a higher risk of bootstrap.mlockall:
true to starve your overall system processes because you will quickly
force them into swapping/paging, or you run out of real memory. So
consider removing this option to give the system enough room or upgrade
to a machine with > 4 GB RAM. Another option is adding more nodes of course.

If you stay with 2 GB RAM I would also recommend to disable 64-bit
server JVM with "java -d32 -client" in order to save some bits on the
heap. 64-bit server JVM makes sense on machines where you can use
virtual address space in the area of > 4GB RAM.

Jörg

Am 09.04.13 12:07, schrieb Talha Khan:

> I have set up a 1 node cluster of ES using ES version 0.19.11 for
> storing some logs. After inserting about 5000 logs I now see that my
> CPU usage is rocketing above 100% constanltly even when no/very few
> search queries are reaching it. ES has become very unresponsive and
> some of my search queries randomly return the following exception.
> "error" : "EsRejectedExecutionException[rejected execution of
> [org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]",
>   "status" : 503
>
> By randomly i mean sometimes they succeed but mostly they fail.
>
>
> Here is my ES config:
>
> index.cache.field.expire: 20m
> index.cache.field.type: soft
>
> bootstrap.mlockall: true
> threadpool:
>    search:
>        type: fixed
>        size: 20
>        queue_size: 1000
>    index:
>        type: fixed
>        size: 16
>    bulk:
>        type: fixed
>        size: 5
>
>
> I did some digging into the issue. The stats for node_stats,
> hot_threads api and jstack can be found in this gist
>
> https://gist.github.com/ahmedtalhakhan/5343784
>
> I am also attaching the results of bigdesk. The cluster health is
> green but looking at these results, it seems that ES is very low on
> heap memory. But i am unable to understand the constant high CPU usage
> even in the idle state. How can this issue be solved?
>
>
>  Kindly give some context and explanation to any answer.
>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.