Zen Discovery settings in EC2 and escaping from Java API
Before asking my questions, I want to describe my situation.
I deployed elasticsearch in the production today but within 3 hours, I had to turn off all the clients because elasticsearch cluster got crashed.
Now, I am using elasticsearch for realtime log analysis. It should handle multi billion rows in a day and I deployed elasitcsearch to more than 30 instances in EC2.
I changed several settings including number of shards and replication but most settings were default.
First, its performance was really good but when the clients started to be throttling, whole elasticsearch cluster was gone to unhealthy. I checked the log and there were lots of fault detection but every instance was alive.
So my plan is to change the zen discovery setting as the following:
My second question is, when I restart servers, do I have to restart the client too? Even though I restarted servers, the clients couldn't get connected. If I want the clients totally independent from servers, I have to use REST API, what's the best way to generate JSON string request? Is there any method from Java API to JSON string such as ToXContent interface?