Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Ben Berg
Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I upgraded the first node,removed and reinstalled latest versions of plugins - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the errors below in logs on upgraded node and node does not join cluster. If i downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin version 1.3.0.4 the node properly joins the cluster.
Errors from logs:
[2015-01-06 21:29:39,970][INFO ][node                     ] [bi-es1] version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
[2015-01-06 21:29:39,971][INFO ][node                     ] [bi-es1] initializing ...
[2015-01-06 21:29:40,009][INFO ][plugins                  ] [bi-es1] loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, paramedic, HQ, whatson]
[2015-01-06 21:29:45,587][INFO ][node                     ] [bi-es1] initialized
[2015-01-06 21:29:45,588][INFO ][node                     ] [bi-es1] starting ...
[2015-01-06 21:29:45,960][INFO ][transport                ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.83.231:9300]}
[2015-01-06 21:29:45,982][INFO ][discovery                ] [bi-es1] bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
[2015-01-06 21:29:51,906][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28805963] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:51,915][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:29:57,028][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28807127] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:57,036][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:02,245][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28808254] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:02,252][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:07,576][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28809377] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:07,583][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:12,689][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28810050] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:12,696][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:15,989][WARN ][discovery                ] [bi-es1] waited for 30s and no initial state was set by the discovery
[2015-01-06 21:30:16,002][INFO ][http                     ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.83.231:9200]}
[2015-01-06 21:30:16,003][INFO ][node                     ] [bi-es1] started
[2015-01-06 21:30:18,277][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28811164] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:18,292][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results in same errors. I have tried closing the _river index and removing all defined rivers (wasn't hopeful that would work but gave it a shot). 

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem good except still on 1.3.2. Would prefer not to do a full cluster shutdown to upgrade all nodes and jdbc plugin but that seems maybe it would solve this?

I have had issues in past with upgrades/jdbc river plugin (really love the plugin though) so my brain is focusing on that and might not be the issue? I used same process on a test cluster that I just realized did not have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Radu Gheorghe-2
Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and do a rolling restart (sticking to 1.3.2), then do the upgrade with another rolling restart, then install the plugin back again with yet another rolling restart?

I would understand if you said "no way I do 3 restarts!" :) But maybe this will help in future: https://github.com/jprante/elasticsearch-plugin-deploy

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg <[hidden email]> wrote:
Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I upgraded the first node,removed and reinstalled latest versions of plugins - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the errors below in logs on upgraded node and node does not join cluster. If i downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin version 1.3.0.4 the node properly joins the cluster.
Errors from logs:
[2015-01-06 21:29:39,970][INFO ][node                     ] [bi-es1] version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
[2015-01-06 21:29:39,971][INFO ][node                     ] [bi-es1] initializing ...
[2015-01-06 21:29:40,009][INFO ][plugins                  ] [bi-es1] loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, paramedic, HQ, whatson]
[2015-01-06 21:29:45,587][INFO ][node                     ] [bi-es1] initialized
[2015-01-06 21:29:45,588][INFO ][node                     ] [bi-es1] starting ...
[2015-01-06 21:29:45,960][INFO ][transport                ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.83.231:9300]}
[2015-01-06 21:29:45,982][INFO ][discovery                ] [bi-es1] bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
[2015-01-06 21:29:51,906][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28805963] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:51,915][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:29:57,028][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28807127] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:57,036][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:02,245][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28808254] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:02,252][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:07,576][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28809377] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:07,583][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:12,689][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28810050] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:12,696][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:15,989][WARN ][discovery                ] [bi-es1] waited for 30s and no initial state was set by the discovery
[2015-01-06 21:30:16,002][INFO ][http                     ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.83.231:9200]}
[2015-01-06 21:30:16,003][INFO ][node                     ] [bi-es1] started
[2015-01-06 21:30:18,277][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28811164] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:18,292][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results in same errors. I have tried closing the _river index and removing all defined rivers (wasn't hopeful that would work but gave it a shot). 

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem good except still on 1.3.2. Would prefer not to do a full cluster shutdown to upgrade all nodes and jdbc plugin but that seems maybe it would solve this?

I have had issues in past with upgrades/jdbc river plugin (really love the plugin though) so my brain is focusing on that and might not be the issue? I used same process on a test cluster that I just realized did not have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Ben Berg
Thanks for the reply!

That is a good idea to try, but another problem with this is that when i uninstall the plugins and keep at version 1.3.2 the node will not join cluster either.
We upgraded to java 8 right before starting this upgrade and thinking it may have something to do with that - also noticed that somehow JAVA_HOME was not defined - working on setting that up across the cluster and will attempt to remove plugin while on 1.3.2 and get node to join cluster - if that does not work I will downgrade to java 7 and try again.
Thanks again for reply, insights and huge thanks for link to the plugin repo - looks awesome and will probably try it after sorting this out. Will update with findings later today hopefully.
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b5dfdbb1-507e-4032-9ec2-40d91266432b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Ben Berg
Ended up finding out that the jdbc plugin requires a full cluster shutdown for restart - https://github.com/jprante/elasticsearch-river-jdbc/issues/433
Going to be doing that in maintenance window tomorrow and will verify that worked.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/60577717-944f-4b13-b66d-124cfb2e7b39%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

joergprante@gmail.com
In reply to this post by Radu Gheorghe-2
Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it can only restart services of a plugin, i.e. lifecycle components with carefully implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions, parsers, functions etc. as modules by the onModule...() mechanism, so they get "baked" into the ES node as immutable objects before the node is going up. This means, the deploy plugin can not change or undo this behavior. (Maybe some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web server that does not use baked-in ES modules, just a restartable service. It is really delightful to just use a single curl PUT command for rapid prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:


Jörg



On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe <[hidden email]> wrote:
Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and do a rolling restart (sticking to 1.3.2), then do the upgrade with another rolling restart, then install the plugin back again with yet another rolling restart?

I would understand if you said "no way I do 3 restarts!" :) But maybe this will help in future: https://github.com/jprante/elasticsearch-plugin-deploy

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg <[hidden email]> wrote:
Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I upgraded the first node,removed and reinstalled latest versions of plugins - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the errors below in logs on upgraded node and node does not join cluster. If i downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin version 1.3.0.4 the node properly joins the cluster.
Errors from logs:
[2015-01-06 21:29:39,970][INFO ][node                     ] [bi-es1] version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
[2015-01-06 21:29:39,971][INFO ][node                     ] [bi-es1] initializing ...
[2015-01-06 21:29:40,009][INFO ][plugins                  ] [bi-es1] loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, paramedic, HQ, whatson]
[2015-01-06 21:29:45,587][INFO ][node                     ] [bi-es1] initialized
[2015-01-06 21:29:45,588][INFO ][node                     ] [bi-es1] starting ...
[2015-01-06 21:29:45,960][INFO ][transport                ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.83.231:9300]}
[2015-01-06 21:29:45,982][INFO ][discovery                ] [bi-es1] bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
[2015-01-06 21:29:51,906][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28805963] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:51,915][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:29:57,028][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28807127] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:57,036][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:02,245][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28808254] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:02,252][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:07,576][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28809377] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:07,583][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:12,689][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28810050] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:12,696][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:15,989][WARN ][discovery                ] [bi-es1] waited for 30s and no initial state was set by the discovery
[2015-01-06 21:30:16,002][INFO ][http                     ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.83.231:9200]}
[2015-01-06 21:30:16,003][INFO ][node                     ] [bi-es1] started
[2015-01-06 21:30:18,277][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28811164] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:18,292][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results in same errors. I have tried closing the _river index and removing all defined rivers (wasn't hopeful that would work but gave it a shot). 

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem good except still on 1.3.2. Would prefer not to do a full cluster shutdown to upgrade all nodes and jdbc plugin but that seems maybe it would solve this?

I have had issues in past with upgrades/jdbc river plugin (really love the plugin though) so my brain is focusing on that and might not be the issue? I used same process on a test cluster that I just realized did not have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Radu Gheorghe-2
Thanks Ben and Joerg. Too bad there was no good solution this particular upgrade, at least we all learned something (well, aside from Joerg who knew it all from the start :D).

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 12:28 AM, [hidden email] <[hidden email]> wrote:
Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it can only restart services of a plugin, i.e. lifecycle components with carefully implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions, parsers, functions etc. as modules by the onModule...() mechanism, so they get "baked" into the ES node as immutable objects before the node is going up. This means, the deploy plugin can not change or undo this behavior. (Maybe some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web server that does not use baked-in ES modules, just a restartable service. It is really delightful to just use a single curl PUT command for rapid prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:


Jörg



On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe <[hidden email]> wrote:
Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and do a rolling restart (sticking to 1.3.2), then do the upgrade with another rolling restart, then install the plugin back again with yet another rolling restart?

I would understand if you said "no way I do 3 restarts!" :) But maybe this will help in future: https://github.com/jprante/elasticsearch-plugin-deploy

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg <[hidden email]> wrote:
Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I upgraded the first node,removed and reinstalled latest versions of plugins - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the errors below in logs on upgraded node and node does not join cluster. If i downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin version 1.3.0.4 the node properly joins the cluster.
Errors from logs:
[2015-01-06 21:29:39,970][INFO ][node                     ] [bi-es1] version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
[2015-01-06 21:29:39,971][INFO ][node                     ] [bi-es1] initializing ...
[2015-01-06 21:29:40,009][INFO ][plugins                  ] [bi-es1] loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, paramedic, HQ, whatson]
[2015-01-06 21:29:45,587][INFO ][node                     ] [bi-es1] initialized
[2015-01-06 21:29:45,588][INFO ][node                     ] [bi-es1] starting ...
[2015-01-06 21:29:45,960][INFO ][transport                ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.83.231:9300]}
[2015-01-06 21:29:45,982][INFO ][discovery                ] [bi-es1] bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
[2015-01-06 21:29:51,906][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28805963] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:51,915][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:29:57,028][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28807127] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:57,036][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:02,245][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28808254] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:02,252][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:07,576][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28809377] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:07,583][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:12,689][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28810050] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:12,696][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:15,989][WARN ][discovery                ] [bi-es1] waited for 30s and no initial state was set by the discovery
[2015-01-06 21:30:16,002][INFO ][http                     ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.83.231:9200]}
[2015-01-06 21:30:16,003][INFO ][node                     ] [bi-es1] started
[2015-01-06 21:30:18,277][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28811164] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:18,292][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results in same errors. I have tried closing the _river index and removing all defined rivers (wasn't hopeful that would work but gave it a shot). 

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem good except still on 1.3.2. Would prefer not to do a full cluster shutdown to upgrade all nodes and jdbc plugin but that seems maybe it would solve this?

I have had issues in past with upgrades/jdbc river plugin (really love the plugin though) so my brain is focusing on that and might not be the issue? I used same process on a test cluster that I just realized did not have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

joergprante@gmail.com
No I didn't :(

I still struggle with stateful plugins - there is no good solution I know of and that is the reason of trouble. 

Jörg

On Fri, Jan 9, 2015 at 9:01 AM, Radu Gheorghe <[hidden email]> wrote:
Thanks Ben and Joerg. Too bad there was no good solution this particular upgrade, at least we all learned something (well, aside from Joerg who knew it all from the start :D).

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 12:28 AM, [hidden email] <[hidden email]> wrote:
Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it can only restart services of a plugin, i.e. lifecycle components with carefully implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions, parsers, functions etc. as modules by the onModule...() mechanism, so they get "baked" into the ES node as immutable objects before the node is going up. This means, the deploy plugin can not change or undo this behavior. (Maybe some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web server that does not use baked-in ES modules, just a restartable service. It is really delightful to just use a single curl PUT command for rapid prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:


Jörg



On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe <[hidden email]> wrote:
Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and do a rolling restart (sticking to 1.3.2), then do the upgrade with another rolling restart, then install the plugin back again with yet another rolling restart?

I would understand if you said "no way I do 3 restarts!" :) But maybe this will help in future: https://github.com/jprante/elasticsearch-plugin-deploy

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg <[hidden email]> wrote:
Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I upgraded the first node,removed and reinstalled latest versions of plugins - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the errors below in logs on upgraded node and node does not join cluster. If i downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin version 1.3.0.4 the node properly joins the cluster.
Errors from logs:
[2015-01-06 21:29:39,970][INFO ][node                     ] [bi-es1] version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
[2015-01-06 21:29:39,971][INFO ][node                     ] [bi-es1] initializing ...
[2015-01-06 21:29:40,009][INFO ][plugins                  ] [bi-es1] loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, paramedic, HQ, whatson]
[2015-01-06 21:29:45,587][INFO ][node                     ] [bi-es1] initialized
[2015-01-06 21:29:45,588][INFO ][node                     ] [bi-es1] starting ...
[2015-01-06 21:29:45,960][INFO ][transport                ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.83.231:9300]}
[2015-01-06 21:29:45,982][INFO ][discovery                ] [bi-es1] bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
[2015-01-06 21:29:51,906][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28805963] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:51,915][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:29:57,028][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28807127] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:57,036][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:02,245][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28808254] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:02,252][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:07,576][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28809377] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:07,583][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:12,689][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28810050] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:12,696][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:15,989][WARN ][discovery                ] [bi-es1] waited for 30s and no initial state was set by the discovery
[2015-01-06 21:30:16,002][INFO ][http                     ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.83.231:9200]}
[2015-01-06 21:30:16,003][INFO ][node                     ] [bi-es1] started
[2015-01-06 21:30:18,277][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28811164] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:18,292][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results in same errors. I have tried closing the _river index and removing all defined rivers (wasn't hopeful that would work but gave it a shot). 

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem good except still on 1.3.2. Would prefer not to do a full cluster shutdown to upgrade all nodes and jdbc plugin but that seems maybe it would solve this?

I have had issues in past with upgrades/jdbc river plugin (really love the plugin though) so my brain is focusing on that and might not be the issue? I used same process on a test cluster that I just realized did not have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Radu Gheorghe-2
Yeah, my general approach so far was to stay away from plugins as much as possible and use external services instead. Something that your JDBC river can do as well, as far as I understand.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 10:20 AM, [hidden email] <[hidden email]> wrote:
No I didn't :(

I still struggle with stateful plugins - there is no good solution I know of and that is the reason of trouble. 

Jörg

On Fri, Jan 9, 2015 at 9:01 AM, Radu Gheorghe <[hidden email]> wrote:
Thanks Ben and Joerg. Too bad there was no good solution this particular upgrade, at least we all learned something (well, aside from Joerg who knew it all from the start :D).

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 12:28 AM, [hidden email] <[hidden email]> wrote:
Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it can only restart services of a plugin, i.e. lifecycle components with carefully implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions, parsers, functions etc. as modules by the onModule...() mechanism, so they get "baked" into the ES node as immutable objects before the node is going up. This means, the deploy plugin can not change or undo this behavior. (Maybe some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web server that does not use baked-in ES modules, just a restartable service. It is really delightful to just use a single curl PUT command for rapid prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:


Jörg



On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe <[hidden email]> wrote:
Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and do a rolling restart (sticking to 1.3.2), then do the upgrade with another rolling restart, then install the plugin back again with yet another rolling restart?

I would understand if you said "no way I do 3 restarts!" :) But maybe this will help in future: https://github.com/jprante/elasticsearch-plugin-deploy

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg <[hidden email]> wrote:
Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I upgraded the first node,removed and reinstalled latest versions of plugins - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the errors below in logs on upgraded node and node does not join cluster. If i downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin version 1.3.0.4 the node properly joins the cluster.
Errors from logs:
[2015-01-06 21:29:39,970][INFO ][node                     ] [bi-es1] version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
[2015-01-06 21:29:39,971][INFO ][node                     ] [bi-es1] initializing ...
[2015-01-06 21:29:40,009][INFO ][plugins                  ] [bi-es1] loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, paramedic, HQ, whatson]
[2015-01-06 21:29:45,587][INFO ][node                     ] [bi-es1] initialized
[2015-01-06 21:29:45,588][INFO ][node                     ] [bi-es1] starting ...
[2015-01-06 21:29:45,960][INFO ][transport                ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.83.231:9300]}
[2015-01-06 21:29:45,982][INFO ][discovery                ] [bi-es1] bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
[2015-01-06 21:29:51,906][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28805963] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:51,915][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:29:57,028][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28807127] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:57,036][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:02,245][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28808254] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:02,252][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:07,576][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28809377] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:07,583][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:12,689][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28810050] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:12,696][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:15,989][WARN ][discovery                ] [bi-es1] waited for 30s and no initial state was set by the discovery
[2015-01-06 21:30:16,002][INFO ][http                     ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.83.231:9200]}
[2015-01-06 21:30:16,003][INFO ][node                     ] [bi-es1] started
[2015-01-06 21:30:18,277][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28811164] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:18,292][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results in same errors. I have tried closing the _river index and removing all defined rivers (wasn't hopeful that would work but gave it a shot). 

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem good except still on 1.3.2. Would prefer not to do a full cluster shutdown to upgrade all nodes and jdbc plugin but that seems maybe it would solve this?

I have had issues in past with upgrades/jdbc river plugin (really love the plugin though) so my brain is focusing on that and might not be the issue? I used same process on a test cluster that I just realized did not have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

joergprante@gmail.com
I could switch away from plugin mode, but that would mean people must adapt and sell another JDBC bridge component to their boss and worse, that will keep the SPOF weakness of rivers but with more burden of responsibility on myself. Not in my sparetime.

Maybe a compromise will work better: external service, but also attached to the ES nodes as a "lightweight" restartable plugin. It looks attractive to me to move over to Groovy and run SQL / JDBC from there. The picture is an enhanced deploy plugin and a general harvester job distribution module with UI, where JDBC is just one option. With the deploy plugin it is easy to move JDBC driver jars and other dependencies around in the cluster, for stop/start, keep state etc.

Jörg

On Fri, Jan 9, 2015 at 9:22 AM, Radu Gheorghe <[hidden email]> wrote:
Yeah, my general approach so far was to stay away from plugins as much as possible and use external services instead. Something that your JDBC river can do as well, as far as I understand.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 10:20 AM, [hidden email] <[hidden email]> wrote:
No I didn't :(

I still struggle with stateful plugins - there is no good solution I know of and that is the reason of trouble. 

Jörg

On Fri, Jan 9, 2015 at 9:01 AM, Radu Gheorghe <[hidden email]> wrote:
Thanks Ben and Joerg. Too bad there was no good solution this particular upgrade, at least we all learned something (well, aside from Joerg who knew it all from the start :D).

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 12:28 AM, [hidden email] <[hidden email]> wrote:
Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it can only restart services of a plugin, i.e. lifecycle components with carefully implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions, parsers, functions etc. as modules by the onModule...() mechanism, so they get "baked" into the ES node as immutable objects before the node is going up. This means, the deploy plugin can not change or undo this behavior. (Maybe some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web server that does not use baked-in ES modules, just a restartable service. It is really delightful to just use a single curl PUT command for rapid prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:


Jörg



On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe <[hidden email]> wrote:
Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and do a rolling restart (sticking to 1.3.2), then do the upgrade with another rolling restart, then install the plugin back again with yet another rolling restart?

I would understand if you said "no way I do 3 restarts!" :) But maybe this will help in future: https://github.com/jprante/elasticsearch-plugin-deploy

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg <[hidden email]> wrote:
Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I upgraded the first node,removed and reinstalled latest versions of plugins - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the errors below in logs on upgraded node and node does not join cluster. If i downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin version 1.3.0.4 the node properly joins the cluster.
Errors from logs:
[2015-01-06 21:29:39,970][INFO ][node                     ] [bi-es1] version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
[2015-01-06 21:29:39,971][INFO ][node                     ] [bi-es1] initializing ...
[2015-01-06 21:29:40,009][INFO ][plugins                  ] [bi-es1] loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, paramedic, HQ, whatson]
[2015-01-06 21:29:45,587][INFO ][node                     ] [bi-es1] initialized
[2015-01-06 21:29:45,588][INFO ][node                     ] [bi-es1] starting ...
[2015-01-06 21:29:45,960][INFO ][transport                ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.83.231:9300]}
[2015-01-06 21:29:45,982][INFO ][discovery                ] [bi-es1] bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
[2015-01-06 21:29:51,906][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28805963] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:51,915][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:29:57,028][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28807127] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:29:57,036][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:02,245][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28808254] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:02,252][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:07,576][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28809377] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:07,583][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:12,689][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28810050] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:12,696][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:15,989][WARN ][discovery                ] [bi-es1] waited for 30s and no initial state was set by the discovery
[2015-01-06 21:30:16,002][INFO ][http                     ] [bi-es1] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.83.231:9200]}
[2015-01-06 21:30:16,003][INFO ][node                     ] [bi-es1] started
[2015-01-06 21:30:18,277][WARN ][transport.netty          ] [bi-es1] Message not fully read (request) for [28811164] and action [discovery/zen/join/validate], resetting
[2015-01-06 21:30:18,292][INFO ][discovery.zen            ] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ]
[2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results in same errors. I have tried closing the _river index and removing all defined rivers (wasn't hopeful that would work but gave it a shot). 

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem good except still on 1.3.2. Would prefer not to do a full cluster shutdown to upgrade all nodes and jdbc plugin but that seems maybe it would solve this?

I have had issues in past with upgrades/jdbc river plugin (really love the plugin though) so my brain is focusing on that and might not be the issue? I used same process on a test cluster that I just realized did not have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEs%3D8BFHS59eXES0gSwY2Z6M1K9Ctz3J7Z1ZZ8CJk1K7g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.