Server configuration suggestions

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Server configuration suggestions

Jerome Kleinen
Hi all,

At work we are using elasticsearch to index the logs from network devices like firewalls, proxy servers, dns servers etc. I was not involved in the initial setup of the system however after issues with performance and dropping logs I have made it my goal to get the configuration right. I believe the root cause of our issues is the poor elasticsearch configuration and I would like to ask the input from you guys to see how we should configure everything.

Our server has 24 cores and 128 gb of ram. In terms of storage we unfortunately have to make due with 10 network drives mapped on the system. I imagine we can alter the number of mapped drives by simple readjusting their sizes.

Our current configuration is a single ES instance with about 60gb of ram assigned to it.

I want to set up a cluster of ES instances on the same machine. However, this is where I would to get input from you guys. Considering the specs mentioned above, what should we opt for? I am looking for recommendations regarding the number of instances, the amount of memory to assign to each of them, the disk layout and perhaps configuration settings.

Regarding the amount of instances and memory, I am considering running 4x ES with 16GB each.

Regarding the storage, I am unsure whether it is even beneficial to divide it into x pieces per ES considering we  have to work with network storage. If it is, we can probably divide each mapped drive's size in half, and assign 5 mapped drives to each node.

Regarding settings, I would like to disable replication as I feel it does not make a lot of sense in this setup. Furthermore our ES system is not a true production server with high availability requirements and the logs are kept on another system for compliance reasons anyway.

Any input would be highly appreciated!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a2f3e746-f299-43e2-bec7-1f826b434c17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Server configuration suggestions

Mark Walkom-2
Multiple instances per physical makes some sense here.

Start with 4 x 16GB heap instances (32GB system), use ES "striping" (ie multiple path.data) to write to N drives per instance (probably 2, or 3 if you can get more). Use the os.processors setting to lock 5 cores per instance, leaving some for the OS to manage things with.

Disabling replication is your call here, most wouldn't recommend it, but given you're also only running on a single physical you're not really redundant anyway. However you can potentially utilise http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-shadow-replicas.html

On 4 May 2015 at 00:30, Jerome Kleinen <[hidden email]> wrote:
Hi all,

At work we are using elasticsearch to index the logs from network devices like firewalls, proxy servers, dns servers etc. I was not involved in the initial setup of the system however after issues with performance and dropping logs I have made it my goal to get the configuration right. I believe the root cause of our issues is the poor elasticsearch configuration and I would like to ask the input from you guys to see how we should configure everything.

Our server has 24 cores and 128 gb of ram. In terms of storage we unfortunately have to make due with 10 network drives mapped on the system. I imagine we can alter the number of mapped drives by simple readjusting their sizes.

Our current configuration is a single ES instance with about 60gb of ram assigned to it.

I want to set up a cluster of ES instances on the same machine. However, this is where I would to get input from you guys. Considering the specs mentioned above, what should we opt for? I am looking for recommendations regarding the number of instances, the amount of memory to assign to each of them, the disk layout and perhaps configuration settings.

Regarding the amount of instances and memory, I am considering running 4x ES with 16GB each.

Regarding the storage, I am unsure whether it is even beneficial to divide it into x pieces per ES considering we  have to work with network storage. If it is, we can probably divide each mapped drive's size in half, and assign 5 mapped drives to each node.

Regarding settings, I would like to disable replication as I feel it does not make a lot of sense in this setup. Furthermore our ES system is not a true production server with high availability requirements and the logs are kept on another system for compliance reasons anyway.

Any input would be highly appreciated!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a2f3e746-f299-43e2-bec7-1f826b434c17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9SgwSQKkKRjBS_Xgka_3UqOH-Z-__%2Bw-HGLx2V75WgfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.