JVM Exception when searching on huge data

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

JVM Exception when searching on huge data

manoj-2
Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]


Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

manoj-2
It gone more weird....

Now the Index status turned Yellow too...

  "cluster_name" : "test",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 277,
  "active_shards" : 461,
  "relocating_shards" : 0,
  "initializing_shards" : 2,
  "unassigned_shards" : 91


On Monday, May 7, 2012 6:28:28 PM UTC+5:30, Manoj wrote:
Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]


Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

Rafał Kuć-3
In reply to this post by manoj-2
Re: JVM Exception when searching on huge data Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of 

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

manoj-2
Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of 

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

Rafał Kuć-3
Re: JVM Exception when searching on huge data Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed. 

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of 

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]
Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

manoj-2
Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
 "size" : "794.7gb",

  "cache" : {
          "field_evictions" : 0,
          "field_size" : "6.4gb",
          "field_size_in_bytes" : 6899525304,
          "filter_count" : 325,
          "filter_evictions" : 0,
          "filter_size" : "55.7mb",
          "filter_size_in_bytes" : 58422192
        },

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed. 

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of 

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]
Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

Rafał Kuć-3
Re: JVM Exception when searching on huge data Hello!

Yes, this may be the reason your cluster went from Green to Yellow state, for example one of your nodes got disconnected from the cluster because of the garbage collection which happened before JVM reported OutOfMemory error. 

I can't say how many entries in the field data cache will be OK for you. You may need to experiment with that. In the stats you pasted, you can see that while doing the stats, your cache was already at 6.4gb for that node. Given that you have ES_MAX_MEM set to 8gb you are probably near the end of the heap limit. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
 "size" : "794.7gb",

  "cache" : {
          "field_evictions" : 0,
          "field_size" : "6.4gb",
          "field_size_in_bytes" : 6899525304,
          "filter_count" : 325,
          "filter_evictions" : 0,
          "filter_size" : "55.7mb",
          "filter_size_in_bytes" : 58422192
        },

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed. 

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of 

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]
Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

manoj-2
Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch
1. /usr/share/es/bin/service/elasticsearch.conf
2. another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max_size and index.cache.field.type?


On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:
Hello!

Yes, this may be the reason your cluster went from Green to Yellow state, for example one of your nodes got disconnected from the cluster because of the garbage collection which happened before JVM reported OutOfMemory error. 

I can't say how many entries in the field data cache will be OK for you. You may need to experiment with that. In the stats you pasted, you can see that while doing the stats, your cache was already at 6.4gb for that node. Given that you have ES_MAX_MEM set to 8gb you are probably near the end of the heap limit. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
 "size" : "794.7gb",

  "cache" : {
          "field_evictions" : 0,
          "field_size" : "6.4gb",
          "field_size_in_bytes" : 6899525304,
          "filter_count" : 325,
          "filter_evictions" : 0,
          "filter_size" : "55.7mb",
          "filter_size_in_bytes" : 58422192
        },

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed. 

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of 

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]
Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

Rafał Kuć-3
Re: JVM Exception when searching on huge data Hello!

You need to check which of those two files is the one that is actually used in your system (maybe one is a link to the other ?). 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch
1. /usr/share/es/bin/service/elasticsearch.conf
2. another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max_size and index.cache.field.type?


On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:
Hello!

Yes, this may be the reason your cluster went from Green to Yellow state, for example one of your nodes got disconnected from the cluster because of the garbage collection which happened before JVM reported OutOfMemory error. 

I can't say how many entries in the field data cache will be OK for you. You may need to experiment with that. In the stats you pasted, you can see that while doing the stats, your cache was already at 6.4gb for that node. Given that you have ES_MAX_MEM set to 8gb you are probably near the end of the heap limit. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
 "size" : "794.7gb",

  "cache" : {
          "field_evictions" : 0,
          "field_size" : "6.4gb",
          "field_size_in_bytes" : 6899525304,
          "filter_count" : 325,
          "filter_evictions" : 0,
          "filter_size" : "55.7mb",
          "filter_size_in_bytes" : 58422192
        },

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed. 

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of 

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]
Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

kimchy
Administrator
In reply to this post by manoj-2
Its better to simply increase the memory used for the JVM, or scale out to more machines in the cluster.

On Mon, May 7, 2012 at 6:45 PM, Manoj <[hidden email]> wrote:
Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch
1. /usr/share/es/bin/service/elasticsearch.conf
2. another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max_size and index.cache.field.type?



On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:
Hello!

Yes, this may be the reason your cluster went from Green to Yellow state, for example one of your nodes got disconnected from the cluster because of the garbage collection which happened before JVM reported OutOfMemory error. 

I can't say how many entries in the field data cache will be OK for you. You may need to experiment with that. In the stats you pasted, you can see that while doing the stats, your cache was already at 6.4gb for that node. Given that you have ES_MAX_MEM set to 8gb you are probably near the end of the heap limit. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
 "size" : "794.7gb",

  "cache" : {
          "field_evictions" : 0,
          "field_size" : "6.4gb",
          "field_size_in_bytes" : 6899525304,
          "filter_count" : 325,
          "filter_evictions" : 0,
          "filter_size" : "55.7mb",
          "filter_size_in_bytes" : 58422192
        },

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed. 

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of 

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

manoj-2
Thanks Kimchy and Rafal....

We increased the JVM memory to 16GB. Also changed the Array type fields in the document to String types.  Achieved String type by using some separating special characters between.  Now, with this new String type data of about 50GB, we again went for faceting.  It still causing the problem...

Also, I would like to confirm, if we are using enough the machine's efficiency by using 2 machines now, with 16GB of JVM for ES. What is the MAX JVM I can give for the current structure?

I am just interested to know the MAX efficiency that could be pulled out of a cluster....

Thanks a lot!


On Wednesday, May 9, 2012 2:34:09 PM UTC+5:30, kimchy wrote:
Its better to simply increase the memory used for the JVM, or scale out to more machines in the cluster.

On Mon, May 7, 2012 at 6:45 PM, Manoj <[hidden email]> wrote:
Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch
1. /usr/share/es/bin/service/elasticsearch.conf
2. another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max_size and index.cache.field.type?



On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:
Hello!

Yes, this may be the reason your cluster went from Green to Yellow state, for example one of your nodes got disconnected from the cluster because of the garbage collection which happened before JVM reported OutOfMemory error. 

I can't say how many entries in the field data cache will be OK for you. You may need to experiment with that. In the stats you pasted, you can see that while doing the stats, your cache was already at 6.4gb for that node. Given that you have ES_MAX_MEM set to 8gb you are probably near the end of the heap limit. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
 "size" : "794.7gb",

  "cache" : {
          "field_evictions" : 0,
          "field_size" : "6.4gb",
          "field_size_in_bytes" : 6899525304,
          "filter_count" : 325,
          "filter_evictions" : 0,
          "filter_size" : "55.7mb",
          "filter_size_in_bytes" : 58422192
        },

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed. 

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of 

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]

Reply | Threaded
Open this post in threaded view
|

Re: JVM Exception when searching on huge data

kimchy
Administrator
Are you asking regarding the maximum amount of memory you can allocate to the elasticsearch java process? It shouldn't be more than the machine memory, leaving space to the OS, but effectively, you probably want to leave some memory as well for things like file system cache.

On Tue, May 15, 2012 at 7:32 AM, Manoj <[hidden email]> wrote:
Thanks Kimchy and Rafal....

We increased the JVM memory to 16GB. Also changed the Array type fields in the document to String types.  Achieved String type by using some separating special characters between.  Now, with this new String type data of about 50GB, we again went for faceting.  It still causing the problem...

Also, I would like to confirm, if we are using enough the machine's efficiency by using 2 machines now, with 16GB of JVM for ES. What is the MAX JVM I can give for the current structure?

I am just interested to know the MAX efficiency that could be pulled out of a cluster....

Thanks a lot!



On Wednesday, May 9, 2012 2:34:09 PM UTC+5:30, kimchy wrote:
Its better to simply increase the memory used for the JVM, or scale out to more machines in the cluster.

On Mon, May 7, 2012 at 6:45 PM, Manoj <[hidden email]> wrote:
Thanks Shay, that really helped...

I could find two configuration file in linux installed Elasticsearch
1. /usr/share/es/bin/service/elasticsearch.conf
2. another is es/config/elasticsearch.yml

Whr should I put this configuration index.cache.field.max_size and index.cache.field.type?



On Monday, May 7, 2012 8:21:34 PM UTC+5:30, Rafał Kuć wrote:
Hello!

Yes, this may be the reason your cluster went from Green to Yellow state, for example one of your nodes got disconnected from the cluster because of the garbage collection which happened before JVM reported OutOfMemory error. 

I can't say how many entries in the field data cache will be OK for you. You may need to experiment with that. In the stats you pasted, you can see that while doing the stats, your cache was already at 6.4gb for that node. Given that you have ES_MAX_MEM set to 8gb you are probably near the end of the heap limit. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks Rafal, you showed me the right way...

Could u also help me in finding answers for the below:

Is this the reason for the index to go Yellow?

How can we decide up on the size to be allotted for
index.cache.field.max_size: 10000

-FYI-
while running the Nodes stats we found the following detail
 "size" : "794.7gb",

  "cache" : {
          "field_evictions" : 0,
          "field_size" : "6.4gb",
          "field_size_in_bytes" : 6899525304,
          "filter_count" : 325,
          "filter_evictions" : 0,
          "filter_size" : "55.7mb",
          "filter_size_in_bytes" : 58422192
        },

Thank You!

On Monday, May 7, 2012 6:54:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

There was a thread recently about field data cache and its memory usage. Basically, when you use faceting or sorting on a given field for the first time, ElasticSearch needs to read data for that field and put it into field data cache. That's why you didn't encounter that problem with small amount of data and you started seeing it after the amount of data changed. 

What you can do is limit your field data cache size, set it expiration time or change its type from resident to soft. I suggest to read the following thread: 
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/DLcErfeivQk , because it's more or less about the same situation you are currently experiencing. 

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Thanks for the fast reply Rafal .....

We are using Faceting on that field

Could this JVM cause the index to go Yellow?

On Monday, May 7, 2012 6:37:31 PM UTC+5:30, Rafał Kuć wrote:
Hello!

That error tells you that you don't have enough memory to load field data into field data cache. Do you use faceting or sorting ?

-- 
Regards,
 Rafał Kuć
 Sematext :: 
http://sematext.com/ :: Solr - Lucene - Nutch


Hi Shay,

We were running an App on Elasticsearch, which didn't have Nested or Array Types.  As an addition of data, we introduced Nested and Array Types in to the existing index.  This new fields carry even an array size of 100. When I try to query now after indexing data, its bringing the below JVM memory. Already we have an JVM of 

set.default.ES_MIN_MEM=2048
set.default.ES_MAX_MEM=8192

My App was working fine in local when with smaller data(consists of nested and array index too). Today I started to run the app with a huge data. And when on searching this huge index the following exception is occurring...


[2012-05-07 00:47:50,403][WARN ][index.cache.field.data.resident] [Cloak] [2012-05-03] loading field [field.field_type_and_value] caused out of memory failure
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61)
        at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
        at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
        at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
        at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:122)
        at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
        at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:75)
        at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:194)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
[2012-05-07 02:00:20,400][INFO ][monitor.jvm              ] [Cloak] [gc][ParNew][62065][8692] duration [717ms], collections [1]/[7.5s], total [717ms]/[2.1m], memory [7.8gb]->[7.6gb]/[7.9gb]

[2012-05-07 08:37:03,087][WARN ][transport                ] [Cloak] Received response for a request that has timed out, sent [31974ms] ago, timed out [1973ms] ago, action [discovery/zen/fd/ping], node [[Astron][tlPRmwDxQ0ma7PJJPD2oXw][inet[/172.16.144.203:9300]]], id [152791]