1.4.0 data node can't join existing 1.3.4 cluster

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

1.4.0 data node can't join existing 1.3.4 cluster

Eric Jain
(using elasticsearch-cloud-aws 2.4)

This should work, right? Or do I need to upgrade the cluster to 1.3.5 first?

The connection fails after a few errors like:

2014-11-13 07:18:22,498 [WARN] org.elasticsearch.discovery.zen.ping.unicast - [Porcupine] failed to send ping to [[#cloud-i-b743e456-0][530-1d][inet[/10.186.145.210:9300]]]
org.elasticsearch.transport.RemoteTransportException: [Nomad][inet[/10.186.145.210:9300]][internal:discovery/zen/unicast]
Caused by: org.elasticsearch.transport.ActionNotFoundTransportException: No handler for action [internal:discovery/zen/unicast]
        at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210) ~[org.elasticsearch.elasticsearch-1.4.0.jar:na]
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111) ~[org.elasticsearch.elasticsearch-1.4.0.jar:na]
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[org.elasticsearch.elasticsearch-1.4.0.jar:na]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/efd3232c-b22f-4f2b-94c4-c942b75e81bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Todd Kamin
I'm seeing something very similar, also from 1.3.4 to 1.4.0, also using the elasticsearch-cloud-aws plugin 2.4.

[2014-11-13 16:02:26,055][WARN ][discovery.zen.ping.unicast] [White Fang] failed to send ping to [[#cloud-i-03f79bcb-0][localhost][inet[/10.0.0.76:9300]]]
org.elasticsearch.transport.RemoteTransportException: [Kraven the Hunter][inet[/10.0.0.76:9300]][internal:discovery/zen/unicast]
Caused by: org.elasticsearch.transport.ActionNotFoundTransportException: No handler for action [internal:discovery/zen/unicast]
        at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

joergprante@gmail.com
Do not mix 1.3 with 1.4 nodes, it does not work.

Jörg

On Thu, Nov 13, 2014 at 5:16 PM, Todd Kamin <[hidden email]> wrote:
I'm seeing something very similar, also from 1.3.4 to 1.4.0, also using the elasticsearch-cloud-aws plugin 2.4.

[2014-11-13 16:02:26,055][WARN ][discovery.zen.ping.unicast] [White Fang] failed to send ping to [[#cloud-i-03f79bcb-0][localhost][inet[/10.0.0.76:9300]]]
org.elasticsearch.transport.RemoteTransportException: [Kraven the Hunter][inet[/10.0.0.76:9300]][internal:discovery/zen/unicast]
Caused by: org.elasticsearch.transport.ActionNotFoundTransportException: No handler for action [internal:discovery/zen/unicast]
        at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGOOeMcuZqw7xADCuTc-MTQkuqQ0QtnBR%2Bg9TWBS6DOng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Ivan Brusic
Rolling upgrades should be supported:


How else can you perform a rolling upgrade without having a mixed cluster?

-- 
Ivan

On Thu, Nov 13, 2014 at 1:05 PM, [hidden email] <[hidden email]> wrote:
Do not mix 1.3 with 1.4 nodes, it does not work.

Jörg

On Thu, Nov 13, 2014 at 5:16 PM, Todd Kamin <[hidden email]> wrote:
I'm seeing something very similar, also from 1.3.4 to 1.4.0, also using the elasticsearch-cloud-aws plugin 2.4.

[2014-11-13 16:02:26,055][WARN ][discovery.zen.ping.unicast] [White Fang] failed to send ping to [[#cloud-i-03f79bcb-0][localhost][inet[/10.0.0.76:9300]]]
org.elasticsearch.transport.RemoteTransportException: [Kraven the Hunter][inet[/10.0.0.76:9300]][internal:discovery/zen/unicast]
Caused by: org.elasticsearch.transport.ActionNotFoundTransportException: No handler for action [internal:discovery/zen/unicast]
        at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGOOeMcuZqw7xADCuTc-MTQkuqQ0QtnBR%2Bg9TWBS6DOng%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDrfhTAc6FhPpU-ZcHtW4Qp_9qB9G6z%2BSXUR7fqtB52_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

joergprante@gmail.com
I discovered a peculiarity, well documented in the source 

https://github.com/elasticsearch/elasticsearch/commit/e5de47d928582694c7729d199390086983779e6e

    /**
     * when pinging the initial configured target hosts, we do not know their version. We therefore use
     * the lowest possible version (i.e., 1.0.0) for serializing information on the wire. As of 1.4, we needed to extend
     * the information sent in a ping, to prefer nodes which have previously joined the cluster during master election.
     * This information is only needed if all the cluster is on version 1.4 or up. To bypass this issue we introduce
     * a second action name which is guaranteed to exist only on nodes from version 1.4.0 and up. Using this action,
     * we can safely use 1.4.0 as a serialization format. If this fails with a {@link ActionNotFoundTransportException}
     * we know we speak to a node with <1.4 version, and fall back to use {@link #ACTION_NAME}.
     */
    public static final String ACTION_NAME_GTE_1_4 = "internal:discovery/zen/unicast_gte_1_4";

Not sure if everything works well...

Jörg


On Thu, Nov 13, 2014 at 10:17 PM, Ivan Brusic <[hidden email]> wrote:
Rolling upgrades should be supported:


How else can you perform a rolling upgrade without having a mixed cluster?

-- 
Ivan

On Thu, Nov 13, 2014 at 1:05 PM, [hidden email] <[hidden email]> wrote:
Do not mix 1.3 with 1.4 nodes, it does not work.

Jörg

On Thu, Nov 13, 2014 at 5:16 PM, Todd Kamin <[hidden email]> wrote:
I'm seeing something very similar, also from 1.3.4 to 1.4.0, also using the elasticsearch-cloud-aws plugin 2.4.

[2014-11-13 16:02:26,055][WARN ][discovery.zen.ping.unicast] [White Fang] failed to send ping to [[#cloud-i-03f79bcb-0][localhost][inet[/10.0.0.76:9300]]]
org.elasticsearch.transport.RemoteTransportException: [Kraven the Hunter][inet[/10.0.0.76:9300]][internal:discovery/zen/unicast]
Caused by: org.elasticsearch.transport.ActionNotFoundTransportException: No handler for action [internal:discovery/zen/unicast]
        at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGOOeMcuZqw7xADCuTc-MTQkuqQ0QtnBR%2Bg9TWBS6DOng%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDrfhTAc6FhPpU-ZcHtW4Qp_9qB9G6z%2BSXUR7fqtB52_A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFN06R7-sYGcsaUtg_VagcjSHikbmPw%3DUSz-H29nvrmWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Eric Jain
In reply to this post by joergprante@gmail.com
On Thu, Nov 13, 2014 at 10:05 AM, [hidden email]
<[hidden email]> wrote:
> Do not mix 1.3 with 1.4 nodes, it does not work.

If that is so, that seems like something the release notes should mention?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

joergprante@gmail.com
I think this is only related to unicast. But, nevertheless, it *should* work... not sure if this is a bug or a feature....

Jörg

On Fri, Nov 14, 2014 at 12:58 AM, Eric Jain <[hidden email]> wrote:
On Thu, Nov 13, 2014 at 10:05 AM, [hidden email]
<[hidden email]> wrote:
> Do not mix 1.3 with 1.4 nodes, it does not work.

If that is so, that seems like something the release notes should mention?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE7GvQiiz2G-PjNWXNKg5NWHE%2BUfwE0bg_mHekw6e5WZw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Mads Martin Jørgensen
I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 cluster with cloud-aws plugin version 2.4.0. Is there a workaround to use during upgrade, since I assume it's not a problem when they're all upgraded to 1.4.0.


On Friday, November 14, 2014 11:33:45 AM UTC+1, Jörg Prante wrote:
I think this is only related to unicast. But, nevertheless, it *should* work... not sure if this is a bug or a feature....

Jörg

On Fri, Nov 14, 2014 at 12:58 AM, Eric Jain <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="rcOgUQKzG3wJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">eric...@...> wrote:
On Thu, Nov 13, 2014 at 10:05 AM, <a href="javascript:" target="_blank" gdf-obfuscated-mailto="rcOgUQKzG3wJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">joerg...@...
<<a href="javascript:" target="_blank" gdf-obfuscated-mailto="rcOgUQKzG3wJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">joerg...@...> wrote:
> Do not mix 1.3 with 1.4 nodes, it does not work.

If that is so, that seems like something the release notes should mention?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="rcOgUQKzG3wJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com';return true;">https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/23b1428c-1e55-498e-ac8f-499adc8528e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Mads Martin Jørgensen
1.4.0 trying to join a 1.3.5 cluster with cloud-aws also fails.

On Friday, November 14, 2014 12:41:08 PM UTC+1, [hidden email] wrote:
I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 cluster with cloud-aws plugin version 2.4.0. Is there a workaround to use during upgrade, since I assume it's not a problem when they're all upgraded to 1.4.0.


On Friday, November 14, 2014 11:33:45 AM UTC+1, Jörg Prante wrote:
I think this is only related to unicast. But, nevertheless, it *should* work... not sure if this is a bug or a feature....

Jörg

On Fri, Nov 14, 2014 at 12:58 AM, Eric Jain <[hidden email]> wrote:
On Thu, Nov 13, 2014 at 10:05 AM, [hidden email]
<[hidden email]> wrote:
> Do not mix 1.3 with 1.4 nodes, it does not work.

If that is so, that seems like something the release notes should mention?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com';return true;">https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/967ce7d9-3f53-4478-868b-85443a904620%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Eric Jain
In reply to this post by Mads Martin Jørgensen
On Fri, Nov 14, 2014 at 3:41 AM,  <[hidden email]> wrote:
> I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 cluster
> with cloud-aws plugin version 2.4.0. Is there a workaround to use during
> upgrade, since I assume it's not a problem when they're all upgraded to
> 1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at zenobase.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BJ%2BfnBnus%3DOX6itdwcaB9%2Bh_KDMwDYRBWU-4fWL0CohJA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Matthew Barrington
We are running a 1.3.4 cluster using the AWS plugin and I noticed the same error when I tried to upgrade a single node.

Since I was trying this on my test cluster first I decided to see what would happen if I upgraded a 2nd node. Would it split into 2 clusters, have the same issue, etc.

What I discovered was that when 2 nodes were upgraded to 1.4 they joined the cluster correctly and everything looks to be working.

SO the problem seems to be for the initial node to join, but when you try with two everything works out.

On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:
On Fri, Nov 14, 2014 at 3:41 AM,  <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="B2twBEuQSQMJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">madsm...@...> wrote:
> I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 cluster
> with cloud-aws plugin version 2.4.0. Is there a workaround to use during
> upgrade, since I assume it's not a problem when they're all upgraded to
> 1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at <a href="http://zenobase.com" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fzenobase.com\46sa\75D\46sntz\0751\46usg\75AFQjCNHCBe1cWq-vqhsGWvNuEOwUr_8TIg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fzenobase.com\46sa\75D\46sntz\0751\46usg\75AFQjCNHCBe1cWq-vqhsGWvNuEOwUr_8TIg';return true;">zenobase.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f618a79d-0a4f-4f97-8b89-ab6ccb9d1cbe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Matthew Barrington
I stand corrected, this did not work on our main cluster.

On Monday, 17 November 2014 11:13:22 UTC, Matthew Barrington wrote:
We are running a 1.3.4 cluster using the AWS plugin and I noticed the same error when I tried to upgrade a single node.

Since I was trying this on my test cluster first I decided to see what would happen if I upgraded a 2nd node. Would it split into 2 clusters, have the same issue, etc.

What I discovered was that when 2 nodes were upgraded to 1.4 they joined the cluster correctly and everything looks to be working.

SO the problem seems to be for the initial node to join, but when you try with two everything works out.

On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:
On Fri, Nov 14, 2014 at 3:41 AM,  <[hidden email]> wrote:
> I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 cluster
> with cloud-aws plugin version 2.4.0. Is there a workaround to use during
> upgrade, since I assume it's not a problem when they're all upgraded to
> 1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at <a href="http://zenobase.com" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fzenobase.com\46sa\75D\46sntz\0751\46usg\75AFQjCNHCBe1cWq-vqhsGWvNuEOwUr_8TIg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fzenobase.com\46sa\75D\46sntz\0751\46usg\75AFQjCNHCBe1cWq-vqhsGWvNuEOwUr_8TIg';return true;">zenobase.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fc79c5e7-ddd6-4f52-9641-1bd01df3b866%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

1.4.0 data node can't join existing 1.3.4 cluster

Christian Hedegaard
In reply to this post by Eric Jain

I found this thread while trying to research the same issue and it looks like there is currently no resolution. We like to keep up on our elasticsearch upgrades as often as possible and do rolling upgrades to keep our clusters up. When testing I’m having the same issue, I cannot add a 1.4.0 box to the existing 1.3.4 cluster.

 

Is there a fix for this anticipated?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5CF8216AA982AF47A8E6DEACA629D22B4EBEEF75%40s-us-ex-6.US.R5S.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Boaz Leskes
Hi Christian, 

I'm not sure what thread you refer to exactly, but this shouldn't happen. Can you describe the problem you have some more? Anything in the nodes? (both the 1.4 node and the master)

Cheers,
Boaz

On Wednesday, November 19, 2014 2:39:57 AM UTC+1, Christian Hedegaard wrote:

I found this thread while trying to research the same issue and it looks like there is currently no resolution. We like to keep up on our elasticsearch upgrades as often as possible and do rolling upgrades to keep our clusters up. When testing I’m having the same issue, I cannot add a 1.4.0 box to the existing 1.3.4 cluster.

 

Is there a fix for this anticipated?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/16f33a69-fb07-4db1-9c05-b9031c867a63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: 1.4.0 data node can't join existing 1.3.4 cluster

Christian Hedegaard

From the archives: https://groups.google.com/forum/#!searchin/elasticsearch/1.4$20data$20node/elasticsearch/8pUwFld88tI/sBH7bB7rYzsJ

 

Same subject as mine. Started on the 13th.

 

Anyways, I’m having the exact same issue. I’ve got a cluster on 1.3.4 (well, now I’ve upgraded it to 1.3.5). When I provision a new node with 1.4 and the cloud-aws plugin with the proper version (2.4.0), the new node will not come up and join the cluster and so a rolling upgrade does not appear possible.

 

 

From: Boaz Leskes [mailto:[hidden email]]
Sent: Wednesday, November 19, 2014 2:27 PM
To: [hidden email]
Cc: Christian Hedegaard
Subject: Re: 1.4.0 data node can't join existing 1.3.4 cluster

 

Hi Christian, 

 

I'm not sure what thread you refer to exactly, but this shouldn't happen. Can you describe the problem you have some more? Anything in the nodes? (both the 1.4 node and the master)

 

Cheers,

Boaz


On Wednesday, November 19, 2014 2:39:57 AM UTC+1, Christian Hedegaard wrote:

I found this thread while trying to research the same issue and it looks like there is currently no resolution. We like to keep up on our elasticsearch upgrades as often as possible and do rolling upgrades to keep our clusters up. When testing I’m having the same issue, I cannot add a 1.4.0 box to the existing 1.3.4 cluster.

 

Is there a fix for this anticipated?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5CF8216AA982AF47A8E6DEACA629D22B4EBF0DA6%40s-us-ex-6.US.R5S.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Daniel Schonfeld
We use GCE and the GCE plugin, on initial load it seemed like what you are describing.  Only when I examined the logs carefully i noticed that its more strict in reading the yaml and in our case the zone used to be

cloud:
  gce:
      zone: ["us-central1-a","us-central1-f"]

And with 1.4 it should now be:

cloud:
  gce:
    zone: us-central1-a,us-central1-f

I haven't looked into the AWS plugin, but perhaps it the same kind of strict yaml configuration read.

Good luck!

On Wednesday, November 19, 2014 5:43:49 PM UTC-5, Christian Hedegaard wrote:

From the archives: <a href="https://groups.google.com/forum/#!searchin/elasticsearch/1.4$20data$20node/elasticsearch/8pUwFld88tI/sBH7bB7rYzsJ" target="_blank" onmousedown="this.href='https://groups.google.com/forum/#!searchin/elasticsearch/1.4$20data$20node/elasticsearch/8pUwFld88tI/sBH7bB7rYzsJ';return true;" onclick="this.href='https://groups.google.com/forum/#!searchin/elasticsearch/1.4$20data$20node/elasticsearch/8pUwFld88tI/sBH7bB7rYzsJ';return true;"> https://groups.google.com/forum/#!searchin/elasticsearch/1.4$20data$20node/elasticsearch/8pUwFld88tI/sBH7bB7rYzsJ

 

Same subject as mine. Started on the 13th.

 

Anyways, I’m having the exact same issue. I’ve got a cluster on 1.3.4 (well, now I’ve upgraded it to 1.3.5). When I provision a new node with 1.4 and the cloud-aws plugin with the proper version (2.4.0), the new node will not come up and join the cluster and so a rolling upgrade does not appear possible.

 

 

From: Boaz Leskes [mailto:<a href="javascript:" target="_blank" gdf-obfuscated-mailto="DELG_X3be0UJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">b.le...@...]
Sent: Wednesday, November 19, 2014 2:27 PM
To: <a href="javascript:" target="_blank" gdf-obfuscated-mailto="DELG_X3be0UJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasti...@...
Cc: Christian Hedegaard
Subject: Re: 1.4.0 data node can't join existing 1.3.4 cluster

 

Hi Christian, 

 

I'm not sure what thread you refer to exactly, but this shouldn't happen. Can you describe the problem you have some more? Anything in the nodes? (both the 1.4 node and the master)

 

Cheers,

Boaz


On Wednesday, November 19, 2014 2:39:57 AM UTC+1, Christian Hedegaard wrote:

I found this thread while trying to research the same issue and it looks like there is currently no resolution. We like to keep up on our elasticsearch upgrades as often as possible and do rolling upgrades to keep our clusters up. When testing I’m having the same issue, I cannot add a 1.4.0 box to the existing 1.3.4 cluster.

 

Is there a fix for this anticipated?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/294e7c85-10a0-4975-a3ef-12fd106cd69c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: 1.4.0 data node can't join existing 1.3.4 cluster

Christian Hedegaard
In reply to this post by Boaz Leskes

FYI, I have found a solution that works (at least for me).

 

I’ve got a small cluster for testing, only 4 v1.3.5 nodes. What I’ve done is bring up 4X new v1.4.0 nodes as data-only machines. In the yaml I added a line to point the nodes via unicast explicitly to the current master:

discovery.zen.ping.unicast.hosts: ["10.210.9.224:9300"]

 

When I restarted elasticsearch with that setting, with cloud-aws installed and configured on version 2.4.0, the new nodes found the cluster and properly joined it.

 

I will now start nuking the old v1.3.5 nodes to migrate the data off of them. Before the final 1.3.5 node is nuked, I will change the config on one of the v1.4.0 nodes to allow it as master and restart it.

 

I’m not sure if the master stuff is needed or not, but I was very afraid of a split-brain problem. I have another 4-node testing cluster that I will be able to try this upgrade again with in a more controlled manner.

 

I’m NOT looking forward to upgrading our current production cluster this way (15 data-only nodes, 3 master-only nodes).

 

So it would appear that the problem is somewhere in the unicast discovery code. The question is who’s to blame? Elasticsearch or the cloud-aws plugin?

 

 

 

From: Boaz Leskes [mailto:[hidden email]]
Sent: Wednesday, November 19, 2014 2:27 PM
To: [hidden email]
Cc: Christian Hedegaard
Subject: Re: 1.4.0 data node can't join existing 1.3.4 cluster

 

Hi Christian, 

 

I'm not sure what thread you refer to exactly, but this shouldn't happen. Can you describe the problem you have some more? Anything in the nodes? (both the 1.4 node and the master)

 

Cheers,

Boaz


On Wednesday, November 19, 2014 2:39:57 AM UTC+1, Christian Hedegaard wrote:

I found this thread while trying to research the same issue and it looks like there is currently no resolution. We like to keep up on our elasticsearch upgrades as often as possible and do rolling upgrades to keep our clusters up. When testing I’m having the same issue, I cannot add a 1.4.0 box to the existing 1.3.4 cluster.

 

Is there a fix for this anticipated?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5CF8216AA982AF47A8E6DEACA629D22B4EBF409B%40s-us-ex-6.US.R5S.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Mads Martin Jørgensen
In reply to this post by Matthew Barrington
Is there any of the elasticsearch team members that can hint to whether or not this is something that will be fixed in 1.4.1? Then we'll simply wait for it instead of doing different hacks to upgrade.

On Monday, November 17, 2014 12:35:03 PM UTC+1, Matthew Barrington wrote:
I stand corrected, this did not work on our main cluster.

On Monday, 17 November 2014 11:13:22 UTC, Matthew Barrington wrote:
We are running a 1.3.4 cluster using the AWS plugin and I noticed the same error when I tried to upgrade a single node.

Since I was trying this on my test cluster first I decided to see what would happen if I upgraded a 2nd node. Would it split into 2 clusters, have the same issue, etc.

What I discovered was that when 2 nodes were upgraded to 1.4 they joined the cluster correctly and everything looks to be working.

SO the problem seems to be for the initial node to join, but when you try with two everything works out.

On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:
On Fri, Nov 14, 2014 at 3:41 AM,  <[hidden email]> wrote:
> I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 cluster
> with cloud-aws plugin version 2.4.0. Is there a workaround to use during
> upgrade, since I assume it's not a problem when they're all upgraded to
> 1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at <a href="http://zenobase.com" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fzenobase.com\46sa\75D\46sntz\0751\46usg\75AFQjCNHCBe1cWq-vqhsGWvNuEOwUr_8TIg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fzenobase.com\46sa\75D\46sntz\0751\46usg\75AFQjCNHCBe1cWq-vqhsGWvNuEOwUr_8TIg';return true;">zenobase.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/312dcdc1-d826-4cb9-b480-620232634ea7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Mark Walkom-2
It's being looked at, but I don't know much beyond that at the moment sorry.

On 21 November 2014 20:02, <[hidden email]> wrote:
Is there any of the elasticsearch team members that can hint to whether or not this is something that will be fixed in 1.4.1? Then we'll simply wait for it instead of doing different hacks to upgrade.

On Monday, November 17, 2014 12:35:03 PM UTC+1, Matthew Barrington wrote:
I stand corrected, this did not work on our main cluster.

On Monday, 17 November 2014 11:13:22 UTC, Matthew Barrington wrote:
We are running a 1.3.4 cluster using the AWS plugin and I noticed the same error when I tried to upgrade a single node.

Since I was trying this on my test cluster first I decided to see what would happen if I upgraded a 2nd node. Would it split into 2 clusters, have the same issue, etc.

What I discovered was that when 2 nodes were upgraded to 1.4 they joined the cluster correctly and everything looks to be working.

SO the problem seems to be for the initial node to join, but when you try with two everything works out.

On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:
On Fri, Nov 14, 2014 at 3:41 AM,  <[hidden email]> wrote:
> I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 cluster
> with cloud-aws plugin version 2.4.0. Is there a workaround to use during
> upgrade, since I assume it's not a problem when they're all upgraded to
> 1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at zenobase.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/312dcdc1-d826-4cb9-b480-620232634ea7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZn-ryDDoQps-smzUPkJd5ru9EHfKuAGRReU2-J-C35kvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.4.0 data node can't join existing 1.3.4 cluster

Mads Martin Jørgensen
Cool :)

Usually this means a fix will emerge. Thanks!

On Friday, November 21, 2014 10:07:03 AM UTC+1, Mark Walkom wrote:
It's being looked at, but I don't know much beyond that at the moment sorry.

On 21 November 2014 20:02, <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="0UOoZyuu9eoJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">madsm...@...> wrote:
Is there any of the elasticsearch team members that can hint to whether or not this is something that will be fixed in 1.4.1? Then we'll simply wait for it instead of doing different hacks to upgrade.

On Monday, November 17, 2014 12:35:03 PM UTC+1, Matthew Barrington wrote:
I stand corrected, this did not work on our main cluster.

On Monday, 17 November 2014 11:13:22 UTC, Matthew Barrington wrote:
We are running a 1.3.4 cluster using the AWS plugin and I noticed the same error when I tried to upgrade a single node.

Since I was trying this on my test cluster first I decided to see what would happen if I upgraded a 2nd node. Would it split into 2 clusters, have the same issue, etc.

What I discovered was that when 2 nodes were upgraded to 1.4 they joined the cluster correctly and everything looks to be working.

SO the problem seems to be for the initial node to join, but when you try with two everything works out.

On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:
On Fri, Nov 14, 2014 at 3:41 AM,  <[hidden email]> wrote:
> I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 cluster
> with cloud-aws plugin version 2.4.0. Is there a workaround to use during
> upgrade, since I assume it's not a problem when they're all upgraded to
> 1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at <a href="http://zenobase.com" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fzenobase.com\46sa\75D\46sntz\0751\46usg\75AFQjCNHCBe1cWq-vqhsGWvNuEOwUr_8TIg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fzenobase.com\46sa\75D\46sntz\0751\46usg\75AFQjCNHCBe1cWq-vqhsGWvNuEOwUr_8TIg';return true;">zenobase.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="0UOoZyuu9eoJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/312dcdc1-d826-4cb9-b480-620232634ea7%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/312dcdc1-d826-4cb9-b480-620232634ea7%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/312dcdc1-d826-4cb9-b480-620232634ea7%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/312dcdc1-d826-4cb9-b480-620232634ea7%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8decfa8a-7583-41ad-ba0f-f7982e49b73d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
12