This is kind of a newbie question but I was unable to find any definitive answers.
Some of this is deployment specific but I think probably the pattern we use is fairly general.
We deploy our clusters with:
1) Master nodes, typically three per cluster with two required masters.
2) Data nodes as usual.
3) Query nodes which our what our ES client applications run against.
We have had a little of debate and confusion about what order to do things in regards to full cluster shutdowns, rolling restarts (mostly for upgrades) and cluster startups.
My understanding is that we should do things as below.
The debate we have had a little bit is in regards to the ordering in the 'rolling restart' case.
- also I just now realized I should probably just look at the source for the API cluster-wide shutdown?
#1. Shutdown query nodes to stop clients.
#2. Shutdown data nodes
#3. Shutdown masters last.
#1. Update masters first so they are forwards/backwards compliant
#2. Update data nodes
#3. Update query nodes
#1. Start up masters - nothing works without these
#2. Start up data nodes
#3. Start up query nodes and begin accepting clients.
Thanks for any advice or definitive answers,
- Michael deMan