How to decrease memory footprint of ElasticSearch

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to decrease memory footprint of ElasticSearch

Axsuul
Is it me or does ElasticSearch require a hefty memory footprint even at its bare minimum? Currently, I have it at 

-Xms32 -Xmx32

and it shows that it's using 380m RES. I tried changing it to 16 but it wouldn't even run. I'm trying to use it for a small project of mine on a low-memory VPS. Can anyone recommend how to decrease the memory to around something reasonable like 100m.

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: How to decrease memory footprint of ElasticSearch

Artem Grinblat
Those resident bits might be the memory-mapped indexes. Try configuring ElasticSearch to use niofs explicitly. Use soft or weak caches. Install bigdesk plugin, maybe it will help with figuring out the memory usage. Keep in mind that JVM needs a lot of RAM for the JIT compiler and the compiled machine code, try running ElasticSearch with -Xint.

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: How to decrease memory footprint of ElasticSearch

joergprante@gmail.com
Hi,

the total process space size can't be limited by -Xms or -Xmx, as they control only the heap space.

There is no way to prevent a JVM to stay within a process limit, as it must load objects into the process space. With NIO, the NIO buffers outside of the heap are accounted for process space.

To get the smallest footprint, try the following

- disable all plugins
- create a $HOME/.elasticsearch.in.sh file with your custom JVM params
- use -d32 for 32-bit JVM (disabling niofs and using simplefs is a natural consequence of 32bit)
- use -Xms64m -Xmx64m -Xss192k (128k for JDK 6), that is the lowest JVM start parameters I know that are safe
- disable all other extra JVM settings in elasticsearch.in.sh (comment them out)
- tune settings in config/elasticsearch.yml: set shards = 1, set replica = 0, set all thread pools to max size 1, set segment tuning to lowest values (you have to experiment, around 1m for segment size should work)
- tune your API usage, do not use caches for facets or for filters, do not use warming
- check memory footprint with the cluster state API

There is a price you pay, performance will be low and the maximum data volume too, and the risk of OOM is high.

There are many JVMs out there beside the standard JVM of Oracle which are targeted at small embedded systems. But, unfortunately, without hacking the ES source (and the dependencies), it won't be possible to get ES running on other JVMs.

Jörg

On Monday, November 5, 2012 12:51:24 AM UTC+1, Artem Grinblat wrote:
Those resident bits might be the memory-mapped indexes. Try configuring ElasticSearch to use niofs explicitly. Use soft or weak caches. Install bigdesk plugin, maybe it will help with figuring out the memory usage. Keep in mind that JVM needs a lot of RAM for the JIT compiler and the compiled machine code, try running ElasticSearch with -Xint.

--