ES runs out of memory

[vachan_da]

Hi everyone,
We've a 3 node ES cluster which powers over website search. The nodes suddenly fail because of Out of memory exception. We are using ec2 instances running on Ubuntu 14.04Lts with open-jdk-7 and 8g of RAM.
Our ES configuration is :

bootstrap.mlockall: true
indices.fielddata.cache.size: 25%
index.number_of_shards: 6
index.auto_expand_replicas: 0-all

This gives our ES 4g of heap space. My question how do I overcome the "Out of memory exception", apart from provisioning more memory?

[Tri H Nguyen]

Occasionally, the error message is misleading so I suggest you check the following

- if these are VMs then check the number of processors allocated per VM vs the number of ES instances per VM if you are running more than one instance of ES per VM

- the settings in /etc/security/limits.conf for the number of file descriptors, number of processes

- it depends on the traffic to your cluster, in my experience, when I split the responsibilities to the master node and client node, the cluster is behaving better and the error is gone. For example, if your master node is also acting as a client node (if http.enabled is true) and maybe data node in this case since you have 3-node. Then you need to turn http off in master node and data node(s), add a client node with http on to minimize the amount of work the master node has to perform. You can index data directly with data node(s), for searches and aggregation, use the client node.

To learn more about these, check the documentation for master node, client node and data node.

Hope this helps.