clarity for shard allocation disable/enable during upgrade

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

clarity for shard allocation disable/enable during upgrade

bitsofinfo.g
I have 8 data nodes and 6 coordinator nodes in an active cluster running 1.2.1

I want to upgrade to 1.3.1

When reading http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html the upgrade docs am I correct to assume:

a) disable shard allocation before doing anything

b) proceed to upgrade each node to 1.3.1

c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.

My question is that at some point during the upgrade of all the data nodes, the shards on them will be "unassigned" and the cluster will not function... correct?

So in other words running some nodes as 1.2.1 and others as 1.3.1 with shard allocation *enabled* is NOT advised and in general cluster un-availability is expected due to shards being in an unassigned state as each data node is upgraded.

At least this is the behavior I see today, (not during an upgrade) when I disable allocation and restart a node, those shards are unassigned until I re-enable allocation


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: clarity for shard allocation disable/enable during upgrade

Mark Walkom
When you disable reallocation and then reboot a node, it should simply re-enable the shards locally rather than trying to do so on other nodes.

However as you have noticed this doesn't actually occur and I have raised a github issue to get this investigated, so feel free to comment on it as well - https://github.com/elasticsearch/elasticsearch/issues/7117

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email]
web: www.campaignmonitor.com


On 11 August 2014 23:31, <[hidden email]> wrote:
I have 8 data nodes and 6 coordinator nodes in an active cluster running 1.2.1

I want to upgrade to 1.3.1


a) disable shard allocation before doing anything

b) proceed to upgrade each node to 1.3.1

c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.

My question is that at some point during the upgrade of all the data nodes, the shards on them will be "unassigned" and the cluster will not function... correct?

So in other words running some nodes as 1.2.1 and others as 1.3.1 with shard allocation *enabled* is NOT advised and in general cluster un-availability is expected due to shards being in an unassigned state as each data node is upgraded.

At least this is the behavior I see today, (not during an upgrade) when I disable allocation and restart a node, those shards are unassigned until I re-enable allocation


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Zz_WEenoivSnn8q0jgYe_MTiCs%2BOaueAxCtNrZR526jw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: clarity for shard allocation disable/enable during upgrade

Clinton Gormley-2
In reply to this post by bitsofinfo.g


On Monday, 11 August 2014 15:31:28 UTC+2, [hidden email] wrote:
I have 8 data nodes and 6 coordinator nodes in an active cluster running 1.2.1

I want to upgrade to 1.3.1

When reading <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-upgrade.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHKPY0AkYGbRbNuB8V1s7CRSmDwaA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-upgrade.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHKPY0AkYGbRbNuB8V1s7CRSmDwaA';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html the upgrade docs am I correct to assume:

a) disable shard allocation before doing anything

b) proceed to upgrade each node to 1.3.1

c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.

My question is that at some point during the upgrade of all the data nodes, the shards on them will be "unassigned" and the cluster will not function... correct?

So in other words running some nodes as 1.2.1 and others as 1.3.1 with shard allocation *enabled* is NOT advised and in general cluster un-availability is expected due to shards being in an unassigned state as each data node is upgraded.

At least this is the behavior I see today, (not during an upgrade) when I disable allocation and restart a node, those shards are unassigned until I re-enable allocation


No, the procedure outlined above is not correct and would indeed result in unassigned shards, as you suspect.  Instead, you should:

1. Disable allocation
2. Upgrade ONE node
3. Reenable allocation
4. Wait for green
5. Repeat

Even when following the above process, you will likely end up with shards being copied over from one node to another (once allocation has been reenabled).  After restart, a replica will only reuse the segments that are exactly the same as those in the primary.  However, because primaries and replicas refresh, flush, and merge at different times, shards diverge from each other over time. The longer it has been since a replica was copied over from the primary, the fewer identical segments they will have in common.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/11f7f772-9d86-4a84-a523-a047f4bb88b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: clarity for shard allocation disable/enable during upgrade

bitsofinfo.g
Mark - isn't the shard allocation all/none a cluster wide setting? Hence why it does that on all nodes?

Clinton - What you said makes sense, however if that procedure is incorrect then the official upgrade page on the elasticsearch site should be changed, as it states

"When the process is complete on all nodes, you can re-enable shard reallocation"


On Tuesday, August 12, 2014 4:04:28 AM UTC-4, Clinton Gormley wrote:


On Monday, 11 August 2014 15:31:28 UTC+2, [hidden email] wrote:
I have 8 data nodes and 6 coordinator nodes in an active cluster running 1.2.1

I want to upgrade to 1.3.1

When reading <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-upgrade.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHKPY0AkYGbRbNuB8V1s7CRSmDwaA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-upgrade.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHKPY0AkYGbRbNuB8V1s7CRSmDwaA';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html the upgrade docs am I correct to assume:

a) disable shard allocation before doing anything

b) proceed to upgrade each node to 1.3.1

c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.

My question is that at some point during the upgrade of all the data nodes, the shards on them will be "unassigned" and the cluster will not function... correct?

So in other words running some nodes as 1.2.1 and others as 1.3.1 with shard allocation *enabled* is NOT advised and in general cluster un-availability is expected due to shards being in an unassigned state as each data node is upgraded.

At least this is the behavior I see today, (not during an upgrade) when I disable allocation and restart a node, those shards are unassigned until I re-enable allocation


No, the procedure outlined above is not correct and would indeed result in unassigned shards, as you suspect.  Instead, you should:

1. Disable allocation
2. Upgrade ONE node
3. Reenable allocation
4. Wait for green
5. Repeat

Even when following the above process, you will likely end up with shards being copied over from one node to another (once allocation has been reenabled).  After restart, a replica will only reuse the segments that are exactly the same as those in the primary.  However, because primaries and replicas refresh, flush, and merge at different times, shards diverge from each other over time. The longer it has been since a replica was copied over from the primary, the fewer identical segments they will have in common.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/095cf279-5c51-49bc-8fb3-536e3160a9ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: clarity for shard allocation disable/enable during upgrade

bitsofinfo.g
In reply to this post by Clinton Gormley-2
Also, Clinton, per the upgrade page it states the below, so what you are saying is that re-enabling allocation after each node is restarted (going from 1.2.1 to 1.31) that the below *will not* apply (incompatibility) because shards would be going from 1.2.1 to 1.3.1 vs the reverse Correct?

"Running multiple versions of Elasticsearch in the same cluster for any length of time beyond that required for an upgrade is not supported, as shard replication from the more recent version to the previous versions will not work."

On Tuesday, August 12, 2014 4:04:28 AM UTC-4, Clinton Gormley wrote:


On Monday, 11 August 2014 15:31:28 UTC+2, [hidden email] wrote:
I have 8 data nodes and 6 coordinator nodes in an active cluster running 1.2.1

I want to upgrade to 1.3.1

When reading <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-upgrade.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHKPY0AkYGbRbNuB8V1s7CRSmDwaA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-upgrade.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHKPY0AkYGbRbNuB8V1s7CRSmDwaA';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html the upgrade docs am I correct to assume:

a) disable shard allocation before doing anything

b) proceed to upgrade each node to 1.3.1

c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.

My question is that at some point during the upgrade of all the data nodes, the shards on them will be "unassigned" and the cluster will not function... correct?

So in other words running some nodes as 1.2.1 and others as 1.3.1 with shard allocation *enabled* is NOT advised and in general cluster un-availability is expected due to shards being in an unassigned state as each data node is upgraded.

At least this is the behavior I see today, (not during an upgrade) when I disable allocation and restart a node, those shards are unassigned until I re-enable allocation


No, the procedure outlined above is not correct and would indeed result in unassigned shards, as you suspect.  Instead, you should:

1. Disable allocation
2. Upgrade ONE node
3. Reenable allocation
4. Wait for green
5. Repeat

Even when following the above process, you will likely end up with shards being copied over from one node to another (once allocation has been reenabled).  After restart, a replica will only reuse the segments that are exactly the same as those in the primary.  However, because primaries and replicas refresh, flush, and merge at different times, shards diverge from each other over time. The longer it has been since a replica was copied over from the primary, the fewer identical segments they will have in common.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0dc5d597-3658-447f-94e0-b4b357fa4b08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.