ES load test ended up with out of memory error after enabling the clustering

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

ES load test ended up with out of memory error after enabling the clustering

Manjula Piyumal
Hi all,

I am trying to run load test with ES to identify system requirements and optimum configurations with respect to my load. I have 10 data publishing tasks and 100 data consuming tasks in my load test.
Data publisher : Each publisher publishes data in every minute and it publishes 1700 records as a batch using java bulk API.
Data consumer : Each consumer runs in every minute and run a query with randomly selected aggregation type(average, minimum or maximum) for a selected data for last hour.
example query that consumer run in every minute :

SearchResponse searchResponse = client.prepareSearch("myIndex").setTypes("myRecordType")
.setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchQuery("filed1", "value1"),FilterBuilders.rangeFilter("field2").from("value2").to("value3")))
.addAggregation(AggregationBuilders.avg("AVG_NAME").field("field3")).execute().actionGet();

I have run above test case in my local machine without ES clustering and it was run around 4 hours without any errors. Memory consumption of ES was under 2GB. After that I have run same test case in three node ES cluster(EC2 instances) and ES has ended up with out of memory error after around 5 minutes in that case. My all three instances have following same hardware configurations,

8GB RAM
80GB SSD hard disk
4 core CPU

Instance 1
Elasticsearch server (4GB heap)
10 data publishers which will publish data to the local ES server

Instance 2
Elasticsearch server (8GB heap)
10 consumers which will query data from the local ES server

Instance 3
Elasticsearch server (4GB heap)

I'm using ES 1.5.1 version with jdk 1.8.0_40.

My ES cluster have following custom configurations (all other configurations are default configurations)

bootstrap.mlockall: true
indices.fielddata.cache.size: "30%"
indices.cache.filter.size: "30%
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host1:9300","host2:9300","host3:9300"]

I believe I have missed something here regarding ES clustering configuration. Please help me to identify what I have missed here. I want to reduce the memory utilization as much as possible, that's why I have given only 4GB heap to ES. If there is a way to reduce the memory consumption by reducing read consistency level that option is also OK for me. I have increased the refresh interval for my index, but still no luck :(

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f8ebc90-dd37-4da7-97bd-3c0d1c00165c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ES load test ended up with out of memory error after enabling the clustering

joergprante@gmail.com
Did you assign different heap sizes? Please use same heap size for all data nodes. Do not limit cache to 30%, this is very small. Let ES use the default settings.

Jörg

On Thu, Apr 16, 2015 at 5:43 PM, Manjula Piyumal <[hidden email]> wrote:
Hi all,

I am trying to run load test with ES to identify system requirements and optimum configurations with respect to my load. I have 10 data publishing tasks and 100 data consuming tasks in my load test.
Data publisher : Each publisher publishes data in every minute and it publishes 1700 records as a batch using java bulk API.
Data consumer : Each consumer runs in every minute and run a query with randomly selected aggregation type(average, minimum or maximum) for a selected data for last hour.
example query that consumer run in every minute :

SearchResponse searchResponse = client.prepareSearch("myIndex").setTypes("myRecordType")
.setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchQuery("filed1", "value1"),FilterBuilders.rangeFilter("field2").from("value2").to("value3")))
.addAggregation(AggregationBuilders.avg("AVG_NAME").field("field3")).execute().actionGet();

I have run above test case in my local machine without ES clustering and it was run around 4 hours without any errors. Memory consumption of ES was under 2GB. After that I have run same test case in three node ES cluster(EC2 instances) and ES has ended up with out of memory error after around 5 minutes in that case. My all three instances have following same hardware configurations,

8GB RAM
80GB SSD hard disk
4 core CPU

Instance 1
Elasticsearch server (4GB heap)
10 data publishers which will publish data to the local ES server

Instance 2
Elasticsearch server (8GB heap)
10 consumers which will query data from the local ES server

Instance 3
Elasticsearch server (4GB heap)

I'm using ES 1.5.1 version with jdk 1.8.0_40.

My ES cluster have following custom configurations (all other configurations are default configurations)

bootstrap.mlockall: true
indices.fielddata.cache.size: "30%"
indices.cache.filter.size: "30%
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host1:9300","host2:9300","host3:9300"]

I believe I have missed something here regarding ES clustering configuration. Please help me to identify what I have missed here. I want to reduce the memory utilization as much as possible, that's why I have given only 4GB heap to ES. If there is a way to reduce the memory consumption by reducing read consistency level that option is also OK for me. I have increased the refresh interval for my index, but still no luck :(

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f8ebc90-dd37-4da7-97bd-3c0d1c00165c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGuT%3DEo74poKcyVhfgiW5W82AFmMSX9ouUEJDAPqhajpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ES load test ended up with out of memory error after enabling the clustering

Manjula Piyumal
Hi Jörg,

Sorry, my bad. It's a typo, I have used 4GB heap for all three ES servers. I have tried without limiting the cache size as the first attempt. But it also got the out of memory error. Am I not missing any other configuration? Or is this load is too much for 4GB heap?

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ES load test ended up with out of memory error after enabling the clustering

joergprante@gmail.com
I have thousands of concurrent indexing/queries running per second on non-virtualized servers.

4G heap is ok, it is more than enough, there should be other reasons for OOM I am sure.

Maybe Bigdesk can help for monitoring heap and cache eviction rates.

I think I can not help any more - no experience with EC2.

Jörg

On Thu, Apr 16, 2015 at 8:05 PM, Manjula Piyumal <[hidden email]> wrote:
Hi Jörg,

Sorry, my bad. It's a typo, I have used 4GB heap for all three ES servers. I have tried without limiting the cache size as the first attempt. But it also got the out of memory error. Am I not missing any other configuration? Or is this load is too much for 4GB heap?

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ES load test ended up with out of memory error after enabling the clustering

Manjula Piyumal
Hi Jörg,

Thanks a lot for your help. May I know what is the size of one record that you are publishing to ES. And did you do any configuration changes to the default configurations?

Thanks
Manjula

On Fri, Apr 17, 2015 at 2:51 AM, [hidden email] <[hidden email]> wrote:
I have thousands of concurrent indexing/queries running per second on non-virtualized servers.

4G heap is ok, it is more than enough, there should be other reasons for OOM I am sure.

Maybe Bigdesk can help for monitoring heap and cache eviction rates.

I think I can not help any more - no experience with EC2.

Jörg

On Thu, Apr 16, 2015 at 8:05 PM, Manjula Piyumal <[hidden email]> wrote:
Hi Jörg,

Sorry, my bad. It's a typo, I have used 4GB heap for all three ES servers. I have tried without limiting the cache size as the first attempt. But it also got the out of memory error. Am I not missing any other configuration? Or is this load is too much for 4GB heap?

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/ci_YFAPLYX4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFLAp0Wnsw8ujLV0Bz8vi7tswwZkcSb9HHN1LR17JnNV_5B%2BVA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ES load test ended up with out of memory error after enabling the clustering

Manjula Piyumal
Hi Jörg,

You are correct, there was a problem in my query client which produced the query load than expected. Thanks again for your help and sorry for the inconvenience.

Thanks
Manjula

On Fri, Apr 17, 2015 at 9:18 AM, Manjula Piyumal <[hidden email]> wrote:
Hi Jörg,

Thanks a lot for your help. May I know what is the size of one record that you are publishing to ES. And did you do any configuration changes to the default configurations?

Thanks
Manjula

On Fri, Apr 17, 2015 at 2:51 AM, [hidden email] <[hidden email]> wrote:
I have thousands of concurrent indexing/queries running per second on non-virtualized servers.

4G heap is ok, it is more than enough, there should be other reasons for OOM I am sure.

Maybe Bigdesk can help for monitoring heap and cache eviction rates.

I think I can not help any more - no experience with EC2.

Jörg

On Thu, Apr 16, 2015 at 8:05 PM, Manjula Piyumal <[hidden email]> wrote:
Hi Jörg,

Sorry, my bad. It's a typo, I have used 4GB heap for all three ES servers. I have tried without limiting the cache size as the first attempt. But it also got the out of memory error. Am I not missing any other configuration? Or is this load is too much for 4GB heap?

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/ci_YFAPLYX4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,



--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFLAp0Vo5iQZV%3D7cBGp3WisLuO-2OLM18-OGDyJ9QA4jCxByfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ES load test ended up with out of memory error after enabling the clustering

joergprante@gmail.com
Glad you solved it. You're welcome.

Best,

Jörg

On Sat, Apr 25, 2015 at 8:12 PM, Manjula Piyumal <[hidden email]> wrote:
Hi Jörg,

You are correct, there was a problem in my query client which produced the query load than expected. Thanks again for your help and sorry for the inconvenience.

Thanks
Manjula

On Fri, Apr 17, 2015 at 9:18 AM, Manjula Piyumal <[hidden email]> wrote:
Hi Jörg,

Thanks a lot for your help. May I know what is the size of one record that you are publishing to ES. And did you do any configuration changes to the default configurations?

Thanks
Manjula

On Fri, Apr 17, 2015 at 2:51 AM, [hidden email] <[hidden email]> wrote:
I have thousands of concurrent indexing/queries running per second on non-virtualized servers.

4G heap is ok, it is more than enough, there should be other reasons for OOM I am sure.

Maybe Bigdesk can help for monitoring heap and cache eviction rates.

I think I can not help any more - no experience with EC2.

Jörg

On Thu, Apr 16, 2015 at 8:05 PM, Manjula Piyumal <[hidden email]> wrote:
Hi Jörg,

Sorry, my bad. It's a typo, I have used 4GB heap for all three ES servers. I have tried without limiting the cache size as the first attempt. But it also got the out of memory error. Am I not missing any other configuration? Or is this load is too much for 4GB heap?

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/ci_YFAPLYX4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,



--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFLAp0Vo5iQZV%3D7cBGp3WisLuO-2OLM18-OGDyJ9QA4jCxByfA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHDJgqGyUOVA6T14ryfsh-Mmj5PoyzcXqWSSYS7zubsAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.