Add / Remove nodes in cluster, good practice question

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Add / Remove nodes in cluster, good practice question

Pierre-Vincent Ledoux
Hi,

We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M documents each, replicated on both nodes.

Those 2 nodes are on VPS that are stored on the same physical host. As it represents a single point of failure, we have decided to start a new VPS on a different host.

What is the correct procedure to add the new node to the cluster, get the indexes replicated in it, and then removed one of the older node?

We don't use mutlicast, so I imagine that I can add the node the the unicast list in the config file, but how I can be sure that it will not fail all my cluster when I will restart elasticsearch?

Those nodes are online in production so it's a bit touchy for us to take any risk with it.


Cheers,

Pv


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Add / Remove nodes in cluster, good practice question

Mark Walkom
If this is production you really want an odd number of nodes to reduce potential split brain issues.

However in your case, just add the new node to the cluster, let it replicate across, then shutdown the node you no longer want. Any impact will be minimal.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email]
web: www.campaignmonitor.com


On 24 July 2014 00:21, Pierre-Vincent Ledoux <[hidden email]> wrote:
Hi,

We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M documents each, replicated on both nodes.

Those 2 nodes are on VPS that are stored on the same physical host. As it represents a single point of failure, we have decided to start a new VPS on a different host.

What is the correct procedure to add the new node to the cluster, get the indexes replicated in it, and then removed one of the older node?

We don't use mutlicast, so I imagine that I can add the node the the unicast list in the config file, but how I can be sure that it will not fail all my cluster when I will restart elasticsearch?

Those nodes are online in production so it's a bit touchy for us to take any risk with it.


Cheers,

Pv


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ak4b-%2BUOw-1KD0s8Pyvjdv%2BrEmbYXUeXsittqqPfKv%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Add / Remove nodes in cluster, good practice question

Nikolas Everett
In reply to this post by Pierre-Vincent Ledoux

1.  Fix the unicast discovery list to contain the two nodes you'll end up with. Make sure the cluster name matches.
2. Start the new node.
3. Watch it join the cluster. If it doesn't then check step 1 and repeat 2 until it works.
4. Use the allocation filtering API to ban all shards from the node you will be removing.
5. Wait until there are no shards on that node. You can check using the _cat/shards API.
6. Shut down the node.

You might want to think about getting three nodes and configuring minimum master nodes but I'm sure you've heard that before.

Nik

On Jul 23, 2014 10:21 AM, "Pierre-Vincent Ledoux" <[hidden email]> wrote:
Hi,

We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M documents each, replicated on both nodes.

Those 2 nodes are on VPS that are stored on the same physical host. As it represents a single point of failure, we have decided to start a new VPS on a different host.

What is the correct procedure to add the new node to the cluster, get the indexes replicated in it, and then removed one of the older node?

We don't use mutlicast, so I imagine that I can add the node the the unicast list in the config file, but how I can be sure that it will not fail all my cluster when I will restart elasticsearch?

Those nodes are online in production so it's a bit touchy for us to take any risk with it.


Cheers,

Pv


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0L2oP0KzeVjZ_5%2BWoeAC%3D4%2Bb_FVkhdVhn_UtC6vmLMxg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Add / Remove nodes in cluster, good practice question

Pierre-Vincent Ledoux
In reply to this post by Mark Walkom
Hi Mark,

thanks! Worked like a charm!


Best regards,

Pv

On 24 Jul 2014, at 00:58, Mark Walkom <[hidden email]> wrote:

If this is production you really want an odd number of nodes to reduce potential split brain issues.

However in your case, just add the new node to the cluster, let it replicate across, then shutdown the node you no longer want. Any impact will be minimal.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email]
web: www.campaignmonitor.com


On 24 July 2014 00:21, Pierre-Vincent Ledoux <[hidden email]> wrote:
Hi,

We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M documents each, replicated on both nodes.

Those 2 nodes are on VPS that are stored on the same physical host. As it represents a single point of failure, we have decided to start a new VPS on a different host.

What is the correct procedure to add the new node to the cluster, get the indexes replicated in it, and then removed one of the older node?

We don't use mutlicast, so I imagine that I can add the node the the unicast list in the config file, but how I can be sure that it will not fail all my cluster when I will restart elasticsearch?

Those nodes are online in production so it's a bit touchy for us to take any risk with it.


Cheers,

Pv



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/7VWJwyNUqvA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ak4b-%2BUOw-1KD0s8Pyvjdv%2BrEmbYXUeXsittqqPfKv%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


signature.asc (506 bytes) Download Attachment