encountered monitor.jvm warning

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

encountered monitor.jvm warning

Abid Hussain
Hi all,

we today got (for the first time) warning messages which seem to indicate a memory problem:
[2015-03-24 09:08:12,960][WARN ][monitor.jvm              ] [Danger] [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total [5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] [6.9gb]->[3.7gb]/[8.5gb]}
[2015-03-24 09:08:12,960][WARN ][monitor.jvm              ] [Danger] [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total [18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] [6.9gb]->[3.7gb]/[8.5gb]}
[2015-03-24 09:08:15,372][WARN ][monitor.jvm              ] [Danger] [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], total [1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young] [6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old] [3.7gb]->[4.9gb]/[8.5gb]}
[2015-03-24 09:08:18,192][WARN ][monitor.jvm              ] [Danger] [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], total [1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young] [845.4mb]->[1.2mb]/[1.1gb]}{[survivor] [149.7mb]->[149.7mb]/[149.7mb]}{[old] [4.9gb]->[6gb]/[8.5gb]}
[2015-03-24 09:08:21,506][WARN ][monitor.jvm              ] [Danger] [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], total [1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young] [848.6mb]->[2.1mb]/[1.1gb]}{[survivor] [149.7mb]->[149.7mb]/[149.7mb]}{[old] [6gb]->[7.2gb]/[8.5gb]}

We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g, other settings are defaults. From previous posts related to this issue, it is said that field data cache may be a problem.

Requesting /_nodes/stats/indices/fielddata says:
{
   "cluster_name": "my_cluster",
   "nodes": {
      "ILUggMfTSvix8Kc0nfNVAw": {
         "timestamp": 1427188716203,
         "name": "Danger",
         "transport_address": "inet[/192.168.110.91:9300]",
         "host": "xxx",
         "ip": [
            "inet[/192.168.110.91:9300]",
            "NONE"
         ],
         "indices": {
            "fielddata": {
               "memory_size_in_bytes": 64822224,
               "evictions": 0
            }
         }
      }
   }
}

Running top results in:
PID USER      PR  NI  VIRT  RES  SHR S   %CPU %MEM    TIME+  COMMAND
12735 root   20   0 15.8g  10g    0 S     74 13.2   2485:26 java

Any ideas what to do? If possible I would rather avoid increasing ES_HEAP as there isn't that much free memory left on the host.

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: encountered monitor.jvm warning

Jason Wee
what is the new gen setting? how much is the system memory in total?
how many cpu cores in the node? what query do you use?

jason

On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain
<[hidden email]> wrote:

> Hi all,
>
> we today got (for the first time) warning messages which seem to indicate a
> memory problem:
> [2015-03-24 09:08:12,960][WARN ][monitor.jvm              ] [Danger]
> [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total
> [5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young]
> [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
> [6.9gb]->[3.7gb]/[8.5gb]}
> [2015-03-24 09:08:12,960][WARN ][monitor.jvm              ] [Danger]
> [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total
> [18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young]
> [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
> [6.9gb]->[3.7gb]/[8.5gb]}
> [2015-03-24 09:08:15,372][WARN ][monitor.jvm              ] [Danger]
> [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], total
> [1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young]
> [6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old]
> [3.7gb]->[4.9gb]/[8.5gb]}
> [2015-03-24 09:08:18,192][WARN ][monitor.jvm              ] [Danger]
> [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], total
> [1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young]
> [845.4mb]->[1.2mb]/[1.1gb]}{[survivor] [149.7mb]->[149.7mb]/[149.7mb]}{[old]
> [4.9gb]->[6gb]/[8.5gb]}
> [2015-03-24 09:08:21,506][WARN ][monitor.jvm              ] [Danger]
> [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], total
> [1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young]
> [848.6mb]->[2.1mb]/[1.1gb]}{[survivor] [149.7mb]->[149.7mb]/[149.7mb]}{[old]
> [6gb]->[7.2gb]/[8.5gb]}
>
> We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g, other
> settings are defaults. From previous posts related to this issue, it is said
> that field data cache may be a problem.
>
> Requesting /_nodes/stats/indices/fielddata says:
> {
>    "cluster_name": "my_cluster",
>    "nodes": {
>       "ILUggMfTSvix8Kc0nfNVAw": {
>          "timestamp": 1427188716203,
>          "name": "Danger",
>          "transport_address": "inet[/192.168.110.91:9300]",
>          "host": "xxx",
>          "ip": [
>             "inet[/192.168.110.91:9300]",
>             "NONE"
>          ],
>          "indices": {
>             "fielddata": {
>                "memory_size_in_bytes": 64822224,
>                "evictions": 0
>             }
>          }
>       }
>    }
> }
>
> Running top results in:
> PID USER      PR  NI  VIRT  RES  SHR S   %CPU %MEM    TIME+  COMMAND
> 12735 root   20   0 15.8g  10g    0 S     74 13.2   2485:26 java
>
> Any ideas what to do? If possible I would rather avoid increasing ES_HEAP as
> there isn't that much free memory left on the host.
>
> Regards,
>
> Abid
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [hidden email].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itz65SQBZu8NOTXgEo6OGp0LdGbTbzvfhp-NPSR%3DaNVXAw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: encountered monitor.jvm warning

Abid Hussain
All settings except from ES_HEAP (set to 10 GB) are defaults, so I actually am not sure about the new gen setting.

The host has 80 GB memory in total and 24 CPU cores. All ES indices together sum up to ~32 GB where the biggest indices are of size ~8 GB.

We are using queries mostly together with filters and also aggregations.

We "solved" the problem with a restart of the cluster. Are there any recommended diagnostics to be performed when this problem occurs next time?

Regards,

Abid

Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee:
what is the new gen setting? how much is the system memory in total?
how many cpu cores in the node? what query do you use?

jason

On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain
<<a href="javascript:" target="_blank" gdf-obfuscated-mailto="YtC2GQ-i8LwJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hus...@...> wrote:

> Hi all,
>
> we today got (for the first time) warning messages which seem to indicate a
> memory problem:
> [2015-03-24 09:08:12,960][WARN ][monitor.jvm              ] [Danger]
> [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total
> [5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young]
> [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
> [6.9gb]->[3.7gb]/[8.5gb]}
> [2015-03-24 09:08:12,960][WARN ][monitor.jvm              ] [Danger]
> [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total
> [18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young]
> [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
> [6.9gb]->[3.7gb]/[8.5gb]}
> [2015-03-24 09:08:15,372][WARN ][monitor.jvm              ] [Danger]
> [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], total
> [1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young]
> [6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old]
> [3.7gb]->[4.9gb]/[8.5gb]}
> [2015-03-24 09:08:18,192][WARN ][monitor.jvm              ] [Danger]
> [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], total
> [1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young]
> [845.4mb]->[1.2mb]/[1.1gb]}{[survivor] [149.7mb]->[149.7mb]/[149.7mb]}{[old]
> [4.9gb]->[6gb]/[8.5gb]}
> [2015-03-24 09:08:21,506][WARN ][monitor.jvm              ] [Danger]
> [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], total
> [1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young]
> [848.6mb]->[2.1mb]/[1.1gb]}{[survivor] [149.7mb]->[149.7mb]/[149.7mb]}{[old]
> [6gb]->[7.2gb]/[8.5gb]}
>
> We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g, other
> settings are defaults. From previous posts related to this issue, it is said
> that field data cache may be a problem.
>
> Requesting /_nodes/stats/indices/fielddata says:
> {
>    "cluster_name": "my_cluster",
>    "nodes": {
>       "ILUggMfTSvix8Kc0nfNVAw": {
>          "timestamp": 1427188716203,
>          "name": "Danger",
>          "transport_address": "inet[/<a href="http://192.168.110.91:9300" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F192.168.110.91%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNHujj_YzhRDYixTdiJ6ZfHcS36Dgw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F192.168.110.91%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNHujj_YzhRDYixTdiJ6ZfHcS36Dgw';return true;">192.168.110.91:9300]",
>          "host": "xxx",
>          "ip": [
>             "inet[/<a href="http://192.168.110.91:9300" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F192.168.110.91%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNHujj_YzhRDYixTdiJ6ZfHcS36Dgw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F192.168.110.91%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNHujj_YzhRDYixTdiJ6ZfHcS36Dgw';return true;">192.168.110.91:9300]",
>             "NONE"
>          ],
>          "indices": {
>             "fielddata": {
>                "memory_size_in_bytes": 64822224,
>                "evictions": 0
>             }
>          }
>       }
>    }
> }
>
> Running top results in:
> PID USER      PR  NI  VIRT  RES  SHR S   %CPU %MEM    TIME+  COMMAND
> 12735 root   20   0 15.8g  10g    0 S     74 13.2   2485:26 java
>
> Any ideas what to do? If possible I would rather avoid increasing ES_HEAP as
> there isn't that much free memory left on the host.
>
> Regards,
>
> Abid
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="YtC2GQ-i8LwJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
> To view this discussion on the web visit
> <a href="https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com.
> For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: encountered monitor.jvm warning

Jason Wee
Few filters should be fine but aggregations... hm....

You could use dump stack trace and/or heap dump if it happen again and
start to analyze from there. Try as well,
http://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

hth

jason

On Tue, Mar 24, 2015 at 11:59 PM, Abid Hussain
<[hidden email]> wrote:

> All settings except from ES_HEAP (set to 10 GB) are defaults, so I actually
> am not sure about the new gen setting.
>
> The host has 80 GB memory in total and 24 CPU cores. All ES indices together
> sum up to ~32 GB where the biggest indices are of size ~8 GB.
>
> We are using queries mostly together with filters and also aggregations.
>
> We "solved" the problem with a restart of the cluster. Are there any
> recommended diagnostics to be performed when this problem occurs next time?
>
> Regards,
>
> Abid
>
> Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee:
>>
>> what is the new gen setting? how much is the system memory in total?
>> how many cpu cores in the node? what query do you use?
>>
>> jason
>>
>> On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain
>> <[hidden email]> wrote:
>> > Hi all,
>> >
>> > we today got (for the first time) warning messages which seem to
>> > indicate a
>> > memory problem:
>> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm              ] [Danger]
>> > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total
>> > [5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young]
>> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
>> > [6.9gb]->[3.7gb]/[8.5gb]}
>> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm              ] [Danger]
>> > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total
>> > [18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young]
>> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
>> > [6.9gb]->[3.7gb]/[8.5gb]}
>> > [2015-03-24 09:08:15,372][WARN ][monitor.jvm              ] [Danger]
>> > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s],
>> > total
>> > [1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young]
>> > [6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old]
>> > [3.7gb]->[4.9gb]/[8.5gb]}
>> > [2015-03-24 09:08:18,192][WARN ][monitor.jvm              ] [Danger]
>> > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s],
>> > total
>> > [1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young]
>> > [845.4mb]->[1.2mb]/[1.1gb]}{[survivor]
>> > [149.7mb]->[149.7mb]/[149.7mb]}{[old]
>> > [4.9gb]->[6gb]/[8.5gb]}
>> > [2015-03-24 09:08:21,506][WARN ][monitor.jvm              ] [Danger]
>> > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s],
>> > total
>> > [1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young]
>> > [848.6mb]->[2.1mb]/[1.1gb]}{[survivor]
>> > [149.7mb]->[149.7mb]/[149.7mb]}{[old]
>> > [6gb]->[7.2gb]/[8.5gb]}
>> >
>> > We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g,
>> > other
>> > settings are defaults. From previous posts related to this issue, it is
>> > said
>> > that field data cache may be a problem.
>> >
>> > Requesting /_nodes/stats/indices/fielddata says:
>> > {
>> >    "cluster_name": "my_cluster",
>> >    "nodes": {
>> >       "ILUggMfTSvix8Kc0nfNVAw": {
>> >          "timestamp": 1427188716203,
>> >          "name": "Danger",
>> >          "transport_address": "inet[/192.168.110.91:9300]",
>> >          "host": "xxx",
>> >          "ip": [
>> >             "inet[/192.168.110.91:9300]",
>> >             "NONE"
>> >          ],
>> >          "indices": {
>> >             "fielddata": {
>> >                "memory_size_in_bytes": 64822224,
>> >                "evictions": 0
>> >             }
>> >          }
>> >       }
>> >    }
>> > }
>> >
>> > Running top results in:
>> > PID USER      PR  NI  VIRT  RES  SHR S   %CPU %MEM    TIME+  COMMAND
>> > 12735 root   20   0 15.8g  10g    0 S     74 13.2   2485:26 java
>> >
>> > Any ideas what to do? If possible I would rather avoid increasing
>> > ES_HEAP as
>> > there isn't that much free memory left on the host.
>> >
>> > Regards,
>> >
>> > Abid
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "elasticsearch" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to [hidden email].
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [hidden email].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itw3Ezh7fxMKaHAJLcK1kPSPhxMOVNNPZX-w0rpPnVmrqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: encountered monitor.jvm warning

Abid Hussain
Maybe aggregations are a cause for memory problems. According to the docs, we set the fielddata cache property to
indices.fielddata.cache.size: 40%
... hoping this will help to avoid this kind of issues.

Regards,

Abid

Am Mittwoch, 25. März 2015 00:47:09 UTC+1 schrieb Jason Wee:
Few filters should be fine but aggregations... hm....

You could use dump stack trace and/or heap dump if it happen again and
start to analyze from there. Try as well,
<a href="http://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fcluster-nodes-hot-threads.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGFdPHk8rtOyi5j1yOjcQS4V3KkcA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fcluster-nodes-hot-threads.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGFdPHk8rtOyi5j1yOjcQS4V3KkcA';return true;">http://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

hth

jason

On Tue, Mar 24, 2015 at 11:59 PM, Abid Hussain
<<a href="javascript:" target="_blank" gdf-obfuscated-mailto="IEFIbNXL-JcJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hus...@...> wrote:

> All settings except from ES_HEAP (set to 10 GB) are defaults, so I actually
> am not sure about the new gen setting.
>
> The host has 80 GB memory in total and 24 CPU cores. All ES indices together
> sum up to ~32 GB where the biggest indices are of size ~8 GB.
>
> We are using queries mostly together with filters and also aggregations.
>
> We "solved" the problem with a restart of the cluster. Are there any
> recommended diagnostics to be performed when this problem occurs next time?
>
> Regards,
>
> Abid
>
> Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee:
>>
>> what is the new gen setting? how much is the system memory in total?
>> how many cpu cores in the node? what query do you use?
>>
>> jason
>>
>> On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain
>> <[hidden email]> wrote:
>> > Hi all,
>> >
>> > we today got (for the first time) warning messages which seem to
>> > indicate a
>> > memory problem:
>> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm              ] [Danger]
>> > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total
>> > [5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young]
>> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
>> > [6.9gb]->[3.7gb]/[8.5gb]}
>> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm              ] [Danger]
>> > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total
>> > [18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young]
>> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
>> > [6.9gb]->[3.7gb]/[8.5gb]}
>> > [2015-03-24 09:08:15,372][WARN ][monitor.jvm              ] [Danger]
>> > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s],
>> > total
>> > [1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young]
>> > [6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old]
>> > [3.7gb]->[4.9gb]/[8.5gb]}
>> > [2015-03-24 09:08:18,192][WARN ][monitor.jvm              ] [Danger]
>> > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s],
>> > total
>> > [1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young]
>> > [845.4mb]->[1.2mb]/[1.1gb]}{[survivor]
>> > [149.7mb]->[149.7mb]/[149.7mb]}{[old]
>> > [4.9gb]->[6gb]/[8.5gb]}
>> > [2015-03-24 09:08:21,506][WARN ][monitor.jvm              ] [Danger]
>> > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s],
>> > total
>> > [1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young]
>> > [848.6mb]->[2.1mb]/[1.1gb]}{[survivor]
>> > [149.7mb]->[149.7mb]/[149.7mb]}{[old]
>> > [6gb]->[7.2gb]/[8.5gb]}
>> >
>> > We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g,
>> > other
>> > settings are defaults. From previous posts related to this issue, it is
>> > said
>> > that field data cache may be a problem.
>> >
>> > Requesting /_nodes/stats/indices/fielddata says:
>> > {
>> >    "cluster_name": "my_cluster",
>> >    "nodes": {
>> >       "ILUggMfTSvix8Kc0nfNVAw": {
>> >          "timestamp": 1427188716203,
>> >          "name": "Danger",
>> >          "transport_address": "inet[/<a href="http://192.168.110.91:9300" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F192.168.110.91%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNHujj_YzhRDYixTdiJ6ZfHcS36Dgw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F192.168.110.91%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNHujj_YzhRDYixTdiJ6ZfHcS36Dgw';return true;">192.168.110.91:9300]",
>> >          "host": "xxx",
>> >          "ip": [
>> >             "inet[/<a href="http://192.168.110.91:9300" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F192.168.110.91%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNHujj_YzhRDYixTdiJ6ZfHcS36Dgw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F192.168.110.91%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNHujj_YzhRDYixTdiJ6ZfHcS36Dgw';return true;">192.168.110.91:9300]",
>> >             "NONE"
>> >          ],
>> >          "indices": {
>> >             "fielddata": {
>> >                "memory_size_in_bytes": 64822224,
>> >                "evictions": 0
>> >             }
>> >          }
>> >       }
>> >    }
>> > }
>> >
>> > Running top results in:
>> > PID USER      PR  NI  VIRT  RES  SHR S   %CPU %MEM    TIME+  COMMAND
>> > 12735 root   20   0 15.8g  10g    0 S     74 13.2   2485:26 java
>> >
>> > Any ideas what to do? If possible I would rather avoid increasing
>> > ES_HEAP as
>> > there isn't that much free memory left on the host.
>> >
>> > Regards,
>> >
>> > Abid
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "elasticsearch" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to elasticsearc...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > <a href="https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com.
>> > For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="IEFIbNXL-JcJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
> To view this discussion on the web visit
> <a href="https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com.
> For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fbf022ac-1b94-413c-b7db-0bee0ea6dbe8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.