What is the proper way to restart node without clients noticing?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

What is the proper way to restart node without clients noticing?

Maciej Dziardziel
Hi

I have a cluster of few es nodes. Due to changes in java scripts, or maintenance, sometimes i need to restart them.
What is the proper way of doing it without clients noticing?

I was thinking about setting up additional nodes with node.data set to false. Clients would connect to those non-data nodes.
Now i would only need a way to say to es cluster "i'm going  to shut down this node in 10 minutes, so let it finish processing its requests,
and redirect new ones to other nodes". Is it possibe?

-- 
Maciej Dziardziel ([hidden email])

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: What is the proper way to restart node without clients noticing?

Ivan Brusic
If your replication values and quorum settings are correct, you can disable allocation and simply restart the nodes. Assuming the shards are correctly distributed, the other nodes will contain all the search data. It helps to disable indexing as well and flush the transaction logs so that you can return the the original state after the nodes rejoin and allocation is reenabled. 

Cheers,

Ivan


On Tue, Jun 18, 2013 at 6:06 AM, Maciej Dziardziel <[hidden email]> wrote:
Hi

I have a cluster of few es nodes. Due to changes in java scripts, or maintenance, sometimes i need to restart them.
What is the proper way of doing it without clients noticing?

I was thinking about setting up additional nodes with node.data set to false. Clients would connect to those non-data nodes.
Now i would only need a way to say to es cluster "i'm going  to shut down this node in 10 minutes, so let it finish processing its requests,
and redirect new ones to other nodes". Is it possibe?

-- 
Maciej Dziardziel ([hidden email])

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: What is the proper way to restart node without clients noticing?

Maciej Dziardziel
Hi

Thanks for your answer. Yes, other nodes will contain data and new requests can be handled by them. The problem is that:

1) Every request from a client connected to the node i am going to restart will fail when i do that.
2) It will take time and some failed requests for a client to realize that a node is down and it should speak to others.

That's the problems i was trying to avoid.

Since i can't find anything useful in ES documentation, i worked around it with a load balancer that allows me to gracefully remove nodes
(without breaking open connections). It would be handy to have such feature in ES by default though.

On Wednesday, June 19, 2013 3:14:00 PM UTC+1, Ivan Brusic wrote:
If your replication values and quorum settings are correct, you can disable allocation and simply restart the nodes. Assuming the shards are correctly distributed, the other nodes will contain all the search data. It helps to disable indexing as well and flush the transaction logs so that you can return the the original state after the nodes rejoin and allocation is reenabled. 

Cheers,

Ivan


On Tue, Jun 18, 2013 at 6:06 AM, Maciej Dziardziel <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="jnnBSNL_NocJ">fie...@...> wrote:
Hi

I have a cluster of few es nodes. Due to changes in java scripts, or maintenance, sometimes i need to restart them.
What is the proper way of doing it without clients noticing?

I was thinking about setting up additional nodes with node.data set to false. Clients would connect to those non-data nodes.
Now i would only need a way to say to es cluster "i'm going  to shut down this node in 10 minutes, so let it finish processing its requests,
and redirect new ones to other nodes". Is it possibe?

-- 
Maciej Dziardziel (<a href="javascript:" target="_blank" gdf-obfuscated-mailto="jnnBSNL_NocJ">fie...@...)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="jnnBSNL_NocJ">elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: What is the proper way to restart node without clients noticing?

Ivan Brusic
Are you using the TransportClient or the REST client? The TransportClient should handle most of these issues. Can't really help if you are using HTTP.

Cheers,

Ivan


On Wed, Jun 19, 2013 at 3:08 PM, Maciej Dziardziel <[hidden email]> wrote:
Hi

Thanks for your answer. Yes, other nodes will contain data and new requests can be handled by them. The problem is that:

1) Every request from a client connected to the node i am going to restart will fail when i do that.
2) It will take time and some failed requests for a client to realize that a node is down and it should speak to others.

That's the problems i was trying to avoid.

Since i can't find anything useful in ES documentation, i worked around it with a load balancer that allows me to gracefully remove nodes
(without breaking open connections). It would be handy to have such feature in ES by default though.


On Wednesday, June 19, 2013 3:14:00 PM UTC+1, Ivan Brusic wrote:
If your replication values and quorum settings are correct, you can disable allocation and simply restart the nodes. Assuming the shards are correctly distributed, the other nodes will contain all the search data. It helps to disable indexing as well and flush the transaction logs so that you can return the the original state after the nodes rejoin and allocation is reenabled. 

Cheers,

Ivan


On Tue, Jun 18, 2013 at 6:06 AM, Maciej Dziardziel <[hidden email]> wrote:
Hi

I have a cluster of few es nodes. Due to changes in java scripts, or maintenance, sometimes i need to restart them.
What is the proper way of doing it without clients noticing?

I was thinking about setting up additional nodes with node.data set to false. Clients would connect to those non-data nodes.
Now i would only need a way to say to es cluster "i'm going  to shut down this node in 10 minutes, so let it finish processing its requests,
and redirect new ones to other nodes". Is it possibe?

-- 
Maciej Dziardziel ([hidden email])

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: What is the proper way to restart node without clients noticing?

sujoysett
In reply to this post by Maciej Dziardziel
Hi,

Once used https://github.com/exalead/commons-httpclient-failover with elasticsearch multiple nodes in a cluster.
That was long back though, not sure whether its still relevant or not.

- Sujoy.


On Thursday, June 20, 2013 3:38:02 AM UTC+5:30, Maciej Dziardziel wrote:
Hi

Thanks for your answer. Yes, other nodes will contain data and new requests can be handled by them. The problem is that:

1) Every request from a client connected to the node i am going to restart will fail when i do that.
2) It will take time and some failed requests for a client to realize that a node is down and it should speak to others.

That's the problems i was trying to avoid.

Since i can't find anything useful in ES documentation, i worked around it with a load balancer that allows me to gracefully remove nodes
(without breaking open connections). It would be handy to have such feature in ES by default though.

On Wednesday, June 19, 2013 3:14:00 PM UTC+1, Ivan Brusic wrote:
If your replication values and quorum settings are correct, you can disable allocation and simply restart the nodes. Assuming the shards are correctly distributed, the other nodes will contain all the search data. It helps to disable indexing as well and flush the transaction logs so that you can return the the original state after the nodes rejoin and allocation is reenabled. 

Cheers,

Ivan


On Tue, Jun 18, 2013 at 6:06 AM, Maciej Dziardziel <[hidden email]> wrote:
Hi

I have a cluster of few es nodes. Due to changes in java scripts, or maintenance, sometimes i need to restart them.
What is the proper way of doing it without clients noticing?

I was thinking about setting up additional nodes with node.data set to false. Clients would connect to those non-data nodes.
Now i would only need a way to say to es cluster "i'm going  to shut down this node in 10 minutes, so let it finish processing its requests,
and redirect new ones to other nodes". Is it possibe?

-- 
Maciej Dziardziel ([hidden email])

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.