Another odd ES freak out...

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Another odd ES freak out...

Grant
So we seem to be having recurring incidences of ES nodes getting into
a very odd state. In this particular case, one node because
unresponsive to test polls. I'm not really sure what to make of this,
because while this is ongoing, the cluster remains green, but the
borked node continues to try and service traffic, which means our app
is sporadically failing in the meantime.

[UTC Feb  5 22:29:23] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable
[UTC Feb  5 22:30:28] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable
[UTC Feb  5 22:31:33] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable


Here's the corresponding logs from the node in question:

Some of this:

[2012-02-05 22:32:11,552][INFO ][discovery.zen            ] [prod-es-
r07] master_left [[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg][inet[prod-es-
r08.ihost.brewster.com/10.180.48.255:9300]]], reason [no longer
master]
[2012-02-05 22:32:11,552][INFO ][cluster.service          ] [prod-es-
r07] master {new [prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
r04.ihost.brewster.com/10.180.35.110:9300]], previous [prod-es-r08]
[mlrGPzm3QeCm7d_E_Lvozg][inet[prod-es-r08.ihost.brewster.com/
10.180.48.255:9300]]}, removed {[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
[inet[prod-es-r08.ihost.brewster.com/10.180.48.255:9300]],}, reason:
zen-disco-master_failed ([prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
[inet[prod-es-r08.ihost.brewster.com/10.180.48.255:9300]])
[2012-02-05 22:32:12,557][INFO ][discovery.zen            ] [prod-es-
r07] master_left [[prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
r04.ihost.brewster.com/10.180.35.110:9300]]], reason [no longer
master]
[2012-02-05 22:32:12,558][WARN ][discovery.zen            ] [prod-es-
r07] not enough master nodes after master left (reason = no longer
master), current nodes: {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg]
[inet[prod-es-r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r07]
[zqJRs5e6S5eWfL0kVuolJg][inet[prod-es-r07.ihost.brewster.com/
10.180.48.216:9300]],}
[2012-02-05 22:32:12,559][INFO ][cluster.service          ] [prod-es-
r07] removed {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg][inet[prod-es-
r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r04]
[uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-r04.ihost.brewster.com/
10.180.35.110:9300]],}, reason: zen-disco-master_failed ([prod-es-r04]
[uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-r04.ihost.brewster.com/
10.180.35.110:9300]])
[2012-02-05 22:32:12,565][WARN ][http.netty               ] [prod-es-
r07] Caught exception while handling client http traffic, closing
connection [id: 0x09be0e53, /10.180.48.216:54645 => /
10.180.48.216:9200]
java.lang.IllegalArgumentException: empty text
        at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
103)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
68)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
81)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
198)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
107)
        at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
470)
        at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
443)
        at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
        at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
        at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
        at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
        at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
[2012-02-05 22:32:12,814][DEBUG][action.search.type       ] [prod-es-
r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
[scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
[2012-02-05 22:32:12,815][DEBUG][action.search.type       ] [prod-es-
r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
[scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
[2012-02-05 22:32:14,066][WARN ][http.netty               ] [prod-es-
r07] Caught exception while handling client http traffic, closing
connection [id: 0x2cb594cf, /10.180.48.216:54651 => /
10.180.48.216:9200]





Followed by tons of this:


[2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
r07] [contact_documents-33-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [R],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@5c67fa3d]
org.elasticsearch.transport.SendRequestTransportException: [prod-es-
r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
phase/query]
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
196)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
168)
        at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:
140)
        at org.elasticsearch.action.search.type.TransportSearchCountAction
$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
279)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
        at org.elasticsearch.search.action.SearchServiceTransportAction
$2.handleException(SearchServiceTransportAction.java:151)
        at org.elasticsearch.transport.TransportService
$2.run(TransportService.java:199)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[prod-es-r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
Node not connected
        at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:
636)
        at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:
448)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
181)
        ... 11 more
[2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
r07] [contact_documents-4-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@718585ec]
org.elasticsearch.transport.SendRequestTransportException: [prod-es-
r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
phase/query]
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
196)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
168)
        at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:
140)
        at org.elasticsearch.action.search.type.TransportSearchCountAction
$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
279)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
        at org.elasticsearch.search.action.SearchServiceTransportAction
$2.handleException(SearchServiceTransportAction.java:151)
        at org.elasticsearch.transport.TransportService
$2.run(TransportService.java:199)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[prod-es-r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
Node not connected
        at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:
636)
        at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:
448)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
181)
        ... 11 more

etc
Reply | Threaded
Open this post in threaded view
|

Re: Another odd ES freak out...

Grant
One ammedment: the cluster was indeed yellow while the node was having
issues.

On Feb 5, 5:48 pm, Grant <[hidden email]> wrote:

> So we seem to be having recurring incidences of ES nodes getting into
> a very odd state. In this particular case, one node because
> unresponsive to test polls. I'm not really sure what to make of this,
> because while this is ongoing, the cluster remains green, but the
> borked node continues to try and service traffic, which means our app
> is sporadically failing in the meantime.
>
> [UTC Feb  5 22:29:23] error    : 'prod_elasticsearch_cluster_health'
> failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
> health] via TCP -- HTTP: Error receiving data -- Resource temporarily
> unavailable
> [UTC Feb  5 22:30:28] error    : 'prod_elasticsearch_cluster_health'
> failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
> health] via TCP -- HTTP: Error receiving data -- Resource temporarily
> unavailable
> [UTC Feb  5 22:31:33] error    : 'prod_elasticsearch_cluster_health'
> failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
> health] via TCP -- HTTP: Error receiving data -- Resource temporarily
> unavailable
>
> Here's the corresponding logs from the node in question:
>
> Some of this:
>
> [2012-02-05 22:32:11,552][INFO ][discovery.zen            ] [prod-es-
> r07] master_left [[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg][inet[prod-es-
> r08.ihost.brewster.com/10.180.48.255:9300]]], reason [no longer
> master]
> [2012-02-05 22:32:11,552][INFO ][cluster.service          ] [prod-es-
> r07] master {new [prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
> r04.ihost.brewster.com/10.180.35.110:9300]], previous [prod-es-r08]
> [mlrGPzm3QeCm7d_E_Lvozg][inet[prod-es-r08.ihost.brewster.com/
> 10.180.48.255:9300]]}, removed {[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
> [inet[prod-es-r08.ihost.brewster.com/10.180.48.255:9300]],}, reason:
> zen-disco-master_failed ([prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
> [inet[prod-es-r08.ihost.brewster.com/10.180.48.255:9300]])
> [2012-02-05 22:32:12,557][INFO ][discovery.zen            ] [prod-es-
> r07] master_left [[prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
> r04.ihost.brewster.com/10.180.35.110:9300]]], reason [no longer
> master]
> [2012-02-05 22:32:12,558][WARN ][discovery.zen            ] [prod-es-
> r07] not enough master nodes after master left (reason = no longer
> master), current nodes: {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg]
> [inet[prod-es-r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r07]
> [zqJRs5e6S5eWfL0kVuolJg][inet[prod-es-r07.ihost.brewster.com/
> 10.180.48.216:9300]],}
> [2012-02-05 22:32:12,559][INFO ][cluster.service          ] [prod-es-
> r07] removed {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg][inet[prod-es-
> r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r04]
> [uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-r04.ihost.brewster.com/
> 10.180.35.110:9300]],}, reason: zen-disco-master_failed ([prod-es-r04]
> [uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-r04.ihost.brewster.com/
> 10.180.35.110:9300]])
> [2012-02-05 22:32:12,565][WARN ][http.netty               ] [prod-es-
> r07] Caught exception while handling client http traffic, closing
> connection [id: 0x09be0e53, /10.180.48.216:54645 => /
> 10.180.48.216:9200]
> java.lang.IllegalArgumentException: empty text
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVe rsion.java:
> 103)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpV ersion.java:
> 68)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.create Message(HttpRequestDecoder.java:
> 81)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode (HttpMessageDecoder.java:
> 198)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode (HttpMessageDecoder.java:
> 107)
>         at
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDe code(ReplayingDecoder.java:
> 470)
>         at
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messag eReceived(ReplayingDecoder.java:
> 443)
>         at
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleU pstream(SimpleChannelUpstreamHandler.java:
> 80)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
> 564)
>         at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
> $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> 783)
>         at
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChann elsHandler.java:
> 81)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
> 564)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
> 559)
>         at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
> 274)
>         at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
> 261)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker. java:
> 351)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelected Keys(NioWorker.java:
> 282)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
> 202)
>         at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
> 108)
>         at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
> $1.run(DeadLockProofWorker.java:44)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1110)
>         at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> [2012-02-05 22:32:12,814][DEBUG][action.search.type       ] [prod-es-
> r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
> [scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
> [2012-02-05 22:32:12,815][DEBUG][action.search.type       ] [prod-es-
> r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
> [scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
> [2012-02-05 22:32:14,066][WARN ][http.netty               ] [prod-es-
> r07] Caught exception while handling client http traffic, closing
> connection [id: 0x2cb594cf, /10.180.48.216:54651 => /
> 10.180.48.216:9200]
>
> Followed by tons of this:
>
> [2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
> r07] [contact_documents-33-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [R],
> s[STARTED]: Failed to execute
> [org.elasticsearch.action.search.SearchRequest@5c67fa3d]
> org.elasticsearch.transport.SendRequestTransportException: [prod-es-
> r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
> phase/query]
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 196)
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 168)
>         at
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQue ry(SearchServiceTransportAction.java:
> 140)
>         at org.elasticsearch.action.search.type.TransportSearchCountAction
> $AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
> 279)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
>         at org.elasticsearch.search.action.SearchServiceTransportAction
> $2.handleException(SearchServiceTransportAction.java:151)
>         at org.elasticsearch.transport.TransportService
> $2.run(TransportService.java:199)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1110)
>         at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> Caused by: org.elasticsearch.transport.NodeNotConnectedException:
> [prod-es-r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
> Node not connected
>         at
> org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport .java:
> 636)
>         at
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport .java:
> 448)
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 181)
>         ... 11 more
> [2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
> r07] [contact_documents-4-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [P],
> s[STARTED]: Failed to execute
> [org.elasticsearch.action.search.SearchRequest@718585ec]
> org.elasticsearch.transport.SendRequestTransportException: [prod-es-
> r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
> phase/query]
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 196)
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 168)
>         at
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQue ry(SearchServiceTransportAction.java:
> 140)
>         at org.elasticsearch.action.search.type.TransportSearchCountAction
> $AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
> 279)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
>         at org.elasticsearch.search.action.SearchServiceTransportAction
> $2.handleException(SearchServiceTransportAction.java:151)
>         at org.elasticsearch.transport.TransportService
> $2.run(TransportService.java:199)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1110)
>         at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> Caused by: org.elasticsearch.transport.NodeNotConnectedException:
> [prod-es-r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
> Node not connected
>         at
> org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport .java:
> 636)
>         at
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport .java:
> 448)
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 181)
>         ... 11 more
>
> etc
Reply | Threaded
Open this post in threaded view
|

Re: Another odd ES freak out...

Grant
In reply to this post by Grant
Also, these are 4gb nodes with 2gb of allocated heap. I noticed right
before this node went down the entire 2gb was used.


On Feb 5, 5:48 pm, Grant <[hidden email]> wrote:

> So we seem to be having recurring incidences of ES nodes getting into
> a very odd state. In this particular case, one node because
> unresponsive to test polls. I'm not really sure what to make of this,
> because while this is ongoing, the cluster remains green, but the
> borked node continues to try and service traffic, which means our app
> is sporadically failing in the meantime.
>
> [UTC Feb  5 22:29:23] error    : 'prod_elasticsearch_cluster_health'
> failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
> health] via TCP -- HTTP: Error receiving data -- Resource temporarily
> unavailable
> [UTC Feb  5 22:30:28] error    : 'prod_elasticsearch_cluster_health'
> failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
> health] via TCP -- HTTP: Error receiving data -- Resource temporarily
> unavailable
> [UTC Feb  5 22:31:33] error    : 'prod_elasticsearch_cluster_health'
> failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
> health] via TCP -- HTTP: Error receiving data -- Resource temporarily
> unavailable
>
> Here's the corresponding logs from the node in question:
>
> Some of this:
>
> [2012-02-05 22:32:11,552][INFO ][discovery.zen            ] [prod-es-
> r07] master_left [[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg][inet[prod-es-
> r08.ihost.brewster.com/10.180.48.255:9300]]], reason [no longer
> master]
> [2012-02-05 22:32:11,552][INFO ][cluster.service          ] [prod-es-
> r07] master {new [prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
> r04.ihost.brewster.com/10.180.35.110:9300]], previous [prod-es-r08]
> [mlrGPzm3QeCm7d_E_Lvozg][inet[prod-es-r08.ihost.brewster.com/
> 10.180.48.255:9300]]}, removed {[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
> [inet[prod-es-r08.ihost.brewster.com/10.180.48.255:9300]],}, reason:
> zen-disco-master_failed ([prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
> [inet[prod-es-r08.ihost.brewster.com/10.180.48.255:9300]])
> [2012-02-05 22:32:12,557][INFO ][discovery.zen            ] [prod-es-
> r07] master_left [[prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
> r04.ihost.brewster.com/10.180.35.110:9300]]], reason [no longer
> master]
> [2012-02-05 22:32:12,558][WARN ][discovery.zen            ] [prod-es-
> r07] not enough master nodes after master left (reason = no longer
> master), current nodes: {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg]
> [inet[prod-es-r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r07]
> [zqJRs5e6S5eWfL0kVuolJg][inet[prod-es-r07.ihost.brewster.com/
> 10.180.48.216:9300]],}
> [2012-02-05 22:32:12,559][INFO ][cluster.service          ] [prod-es-
> r07] removed {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg][inet[prod-es-
> r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r04]
> [uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-r04.ihost.brewster.com/
> 10.180.35.110:9300]],}, reason: zen-disco-master_failed ([prod-es-r04]
> [uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-r04.ihost.brewster.com/
> 10.180.35.110:9300]])
> [2012-02-05 22:32:12,565][WARN ][http.netty               ] [prod-es-
> r07] Caught exception while handling client http traffic, closing
> connection [id: 0x09be0e53, /10.180.48.216:54645 => /
> 10.180.48.216:9200]
> java.lang.IllegalArgumentException: empty text
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVe rsion.java:
> 103)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpV ersion.java:
> 68)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.create Message(HttpRequestDecoder.java:
> 81)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode (HttpMessageDecoder.java:
> 198)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode (HttpMessageDecoder.java:
> 107)
>         at
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDe code(ReplayingDecoder.java:
> 470)
>         at
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messag eReceived(ReplayingDecoder.java:
> 443)
>         at
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleU pstream(SimpleChannelUpstreamHandler.java:
> 80)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
> 564)
>         at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
> $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> 783)
>         at
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChann elsHandler.java:
> 81)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
> 564)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream( DefaultChannelPipeline.java:
> 559)
>         at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
> 274)
>         at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channel s.java:
> 261)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker. java:
> 351)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelected Keys(NioWorker.java:
> 282)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.j ava:
> 202)
>         at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenami ngRunnable.java:
> 108)
>         at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
> $1.run(DeadLockProofWorker.java:44)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1110)
>         at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> [2012-02-05 22:32:12,814][DEBUG][action.search.type       ] [prod-es-
> r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
> [scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
> [2012-02-05 22:32:12,815][DEBUG][action.search.type       ] [prod-es-
> r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
> [scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
> [2012-02-05 22:32:14,066][WARN ][http.netty               ] [prod-es-
> r07] Caught exception while handling client http traffic, closing
> connection [id: 0x2cb594cf, /10.180.48.216:54651 => /
> 10.180.48.216:9200]
>
> Followed by tons of this:
>
> [2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
> r07] [contact_documents-33-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [R],
> s[STARTED]: Failed to execute
> [org.elasticsearch.action.search.SearchRequest@5c67fa3d]
> org.elasticsearch.transport.SendRequestTransportException: [prod-es-
> r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
> phase/query]
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 196)
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 168)
>         at
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQue ry(SearchServiceTransportAction.java:
> 140)
>         at org.elasticsearch.action.search.type.TransportSearchCountAction
> $AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
> 279)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
>         at org.elasticsearch.search.action.SearchServiceTransportAction
> $2.handleException(SearchServiceTransportAction.java:151)
>         at org.elasticsearch.transport.TransportService
> $2.run(TransportService.java:199)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1110)
>         at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> Caused by: org.elasticsearch.transport.NodeNotConnectedException:
> [prod-es-r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
> Node not connected
>         at
> org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport .java:
> 636)
>         at
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport .java:
> 448)
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 181)
>         ... 11 more
> [2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
> r07] [contact_documents-4-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [P],
> s[STARTED]: Failed to execute
> [org.elasticsearch.action.search.SearchRequest@718585ec]
> org.elasticsearch.transport.SendRequestTransportException: [prod-es-
> r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
> phase/query]
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 196)
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 168)
>         at
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQue ry(SearchServiceTransportAction.java:
> 140)
>         at org.elasticsearch.action.search.type.TransportSearchCountAction
> $AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
> 279)
>         at org.elasticsearch.action.search.type.TransportSearchTypeAction
> $BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
>         at org.elasticsearch.search.action.SearchServiceTransportAction
> $2.handleException(SearchServiceTransportAction.java:151)
>         at org.elasticsearch.transport.TransportService
> $2.run(TransportService.java:199)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1110)
>         at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> Caused by: org.elasticsearch.transport.NodeNotConnectedException:
> [prod-es-r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
> Node not connected
>         at
> org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport .java:
> 636)
>         at
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport .java:
> 448)
>         at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.j ava:
> 181)
>         ... 11 more
>
> etc
Reply | Threaded
Open this post in threaded view
|

Re: Another odd ES freak out...

Vishal Bhasin
In reply to this post by Grant
Hello - we are seeing the same issues, were you able to resolve this? thanks!

On Sunday, 5 February 2012 16:48:25 UTC-6, Grant wrote:
So we seem to be having recurring incidences of ES nodes getting into
a very odd state. In this particular case, one node because
unresponsive to test polls. I'm not really sure what to make of this,
because while this is ongoing, the cluster remains green, but the
borked node continues to try and service traffic, which means our app
is sporadically failing in the meantime.

[UTC Feb  5 22:29:23] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[<a href="http://10.180.48.216:9200/_cluster/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;">10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable
[UTC Feb  5 22:30:28] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[<a href="http://10.180.48.216:9200/_cluster/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;">10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable
[UTC Feb  5 22:31:33] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[<a href="http://10.180.48.216:9200/_cluster/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;">10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable


Here's the corresponding logs from the node in question:

Some of this:

[2012-02-05 22:32:11,552][INFO ][discovery.zen            ] [prod-es-
r07] master_left [[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg][inet[prod-es-
<a href="http://r08.ihost.brewster.com/10.180.48.255:9300%5D%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr08.ihost.brewster.com%2F10.180.48.255%3A9300%255D%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNFIZHt2ZexK5ysgXKD8tHN-o7F8Ow';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr08.ihost.brewster.com%2F10.180.48.255%3A9300%255D%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNFIZHt2ZexK5ysgXKD8tHN-o7F8Ow';return true;">r08.ihost.brewster.com/10.180.48.255:9300]]], reason [no longer
master]
[2012-02-05 22:32:11,552][INFO ][cluster.service          ] [prod-es-
r07] master {new [prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
<a href="http://r04.ihost.brewster.com/10.180.35.110:9300%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr04.ihost.brewster.com%2F10.180.35.110%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNFCKV7VJw3TCMUkjYlLvIPUN-LDBw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr04.ihost.brewster.com%2F10.180.35.110%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNFCKV7VJw3TCMUkjYlLvIPUN-LDBw';return true;">r04.ihost.brewster.com/10.180.35.110:9300]], previous [prod-es-r08]
[mlrGPzm3QeCm7d_E_Lvozg][inet[<a href="http://prod-es-r08.ihost.brewster.com/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFrw6ixfqMSj2mvnognP28SqipbbA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFrw6ixfqMSj2mvnognP28SqipbbA';return true;">prod-es-r08.ihost.brewster.com/
10.180.48.255:9300]]}, removed {[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
[inet[<a href="http://prod-es-r08.ihost.brewster.com/10.180.48.255:9300%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F10.180.48.255%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF4w3OhamIMsrsYhvh42-i5BH9NYA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F10.180.48.255%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF4w3OhamIMsrsYhvh42-i5BH9NYA';return true;">prod-es-r08.ihost.brewster.com/10.180.48.255:9300]],}, reason:
zen-disco-master_failed ([prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
[inet[<a href="http://prod-es-r08.ihost.brewster.com/10.180.48.255:9300%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F10.180.48.255%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF4w3OhamIMsrsYhvh42-i5BH9NYA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F10.180.48.255%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF4w3OhamIMsrsYhvh42-i5BH9NYA';return true;">prod-es-r08.ihost.brewster.com/10.180.48.255:9300]])
[2012-02-05 22:32:12,557][INFO ][discovery.zen            ] [prod-es-
r07] master_left [[prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
<a href="http://r04.ihost.brewster.com/10.180.35.110:9300%5D%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr04.ihost.brewster.com%2F10.180.35.110%3A9300%255D%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF2oPcBQ9SAyB2q9xsxraIkMsRreA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr04.ihost.brewster.com%2F10.180.35.110%3A9300%255D%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF2oPcBQ9SAyB2q9xsxraIkMsRreA';return true;">r04.ihost.brewster.com/10.180.35.110:9300]]], reason [no longer
master]
[2012-02-05 22:32:12,558][WARN ][discovery.zen            ] [prod-es-
r07] not enough master nodes after master left (reason = no longer
master), current nodes: {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg]
[inet[<a href="http://prod-es-r02.ihost.brewster.com/10.182.14.95:9300%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r02.ihost.brewster.com%2F10.182.14.95%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNEvoTxSTURYJj-C3SlpUA6c8CUx0Q';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r02.ihost.brewster.com%2F10.182.14.95%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNEvoTxSTURYJj-C3SlpUA6c8CUx0Q';return true;">prod-es-r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r07]
[zqJRs5e6S5eWfL0kVuolJg][inet[<a href="http://prod-es-r07.ihost.brewster.com/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r07.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGhVNkD_PqMdyhB4apIara2ndLuNA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r07.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGhVNkD_PqMdyhB4apIara2ndLuNA';return true;">prod-es-r07.ihost.brewster.com/
10.180.48.216:9300]],}
[2012-02-05 22:32:12,559][INFO ][cluster.service          ] [prod-es-
r07] removed {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg][inet[prod-es-
<a href="http://r02.ihost.brewster.com/10.182.14.95:9300%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr02.ihost.brewster.com%2F10.182.14.95%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNE2ohjSaJXEbZHNMfORykB4Sl9R3A';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr02.ihost.brewster.com%2F10.182.14.95%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNE2ohjSaJXEbZHNMfORykB4Sl9R3A';return true;">r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r04]
[uOUyy7p_TBuNEbwmqWF9-w][inet[<a href="http://prod-es-r04.ihost.brewster.com/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r04.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNE1Cbf2wOqrtrVL1scDl4zGXrwPtg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r04.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNE1Cbf2wOqrtrVL1scDl4zGXrwPtg';return true;">prod-es-r04.ihost.brewster.com/
10.180.35.110:9300]],}, reason: zen-disco-master_failed ([prod-es-r04]
[uOUyy7p_TBuNEbwmqWF9-w][inet[<a href="http://prod-es-r04.ihost.brewster.com/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r04.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNE1Cbf2wOqrtrVL1scDl4zGXrwPtg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r04.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNE1Cbf2wOqrtrVL1scDl4zGXrwPtg';return true;">prod-es-r04.ihost.brewster.com/
10.180.35.110:9300]])
[2012-02-05 22:32:12,565][WARN ][http.netty               ] [prod-es-
r07] Caught exception while handling client http traffic, closing
connection [id: 0x09be0e53, /<a href="http://10.180.48.216:54645" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A54645\46sa\75D\46sntz\0751\46usg\75AFQjCNFFz7G85e12vstnkddSBSYpGB2ZvQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A54645\46sa\75D\46sntz\0751\46usg\75AFQjCNFFz7G85e12vstnkddSBSYpGB2ZvQ';return true;">10.180.48.216:54645 => /
<a href="http://10.180.48.216:9200" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200\46sa\75D\46sntz\0751\46usg\75AFQjCNE9wtHwePKQbo-DKYPYeVEscO1-DA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200\46sa\75D\46sntz\0751\46usg\75AFQjCNE9wtHwePKQbo-DKYPYeVEscO1-DA';return true;">10.180.48.216:9200]
java.lang.IllegalArgumentException: empty text
        at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
103)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
68)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
81)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
198)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
107)
        at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
470)
        at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
443)
        at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
        at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
        at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
        at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
        at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
[2012-02-05 22:32:12,814][DEBUG][action.search.type       ] [prod-es-
r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
[scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
[2012-02-05 22:32:12,815][DEBUG][action.search.type       ] [prod-es-
r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
[scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
[2012-02-05 22:32:14,066][WARN ][http.netty               ] [prod-es-
r07] Caught exception while handling client http traffic, closing
connection [id: 0x2cb594cf, /<a href="http://10.180.48.216:54651" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A54651\46sa\75D\46sntz\0751\46usg\75AFQjCNGMAvblABAE3-qsNtvsriVr7rR_0w';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A54651\46sa\75D\46sntz\0751\46usg\75AFQjCNGMAvblABAE3-qsNtvsriVr7rR_0w';return true;">10.180.48.216:54651 => /
<a href="http://10.180.48.216:9200" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200\46sa\75D\46sntz\0751\46usg\75AFQjCNE9wtHwePKQbo-DKYPYeVEscO1-DA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200\46sa\75D\46sntz\0751\46usg\75AFQjCNE9wtHwePKQbo-DKYPYeVEscO1-DA';return true;">10.180.48.216:9200]





Followed by tons of this:


[2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
r07] [contact_documents-33-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [R],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@5c67fa3d]
org.elasticsearch.transport.SendRequestTransportException: [prod-es-
r06][inet[<a href="http://prod-es-r06.ihost.brewster.com/10.180.46.203:9300%5D%5D%5Bsearch/phase/query" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300%255D%255D%255Bsearch%2Fphase%2Fquery\46sa\75D\46sntz\0751\46usg\75AFQjCNH2IB7w98zPhSoupRTsddWsNmPFuw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300%255D%255D%255Bsearch%2Fphase%2Fquery\46sa\75D\46sntz\0751\46usg\75AFQjCNH2IB7w98zPhSoupRTsddWsNmPFuw';return true;">prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
phase/query]
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
196)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
168)
        at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:
140)
        at org.elasticsearch.action.search.type.TransportSearchCountAction
$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
279)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
        at org.elasticsearch.search.action.SearchServiceTransportAction
$2.handleException(SearchServiceTransportAction.java:151)
        at org.elasticsearch.transport.TransportService
$2.run(TransportService.java:199)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[prod-es-r06][inet[<a href="http://prod-es-r06.ihost.brewster.com/10.180.46.203:9300" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNE-CO2MbVGXna5s3XmFpb9Aj1mfqQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNE-CO2MbVGXna5s3XmFpb9Aj1mfqQ';return true;">prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
Node not connected
        at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:
636)
        at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:
448)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
181)
        ... 11 more
[2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
r07] [contact_documents-4-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@718585ec]
org.elasticsearch.transport.SendRequestTransportException: [prod-es-
r06][inet[<a href="http://prod-es-r06.ihost.brewster.com/10.180.46.203:9300%5D%5D%5Bsearch/phase/query" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300%255D%255D%255Bsearch%2Fphase%2Fquery\46sa\75D\46sntz\0751\46usg\75AFQjCNH2IB7w98zPhSoupRTsddWsNmPFuw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300%255D%255D%255Bsearch%2Fphase%2Fquery\46sa\75D\46sntz\0751\46usg\75AFQjCNH2IB7w98zPhSoupRTsddWsNmPFuw';return true;">prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
phase/query]
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
196)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
168)
        at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:
140)
        at org.elasticsearch.action.search.type.TransportSearchCountAction
$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
279)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
        at org.elasticsearch.search.action.SearchServiceTransportAction
$2.handleException(SearchServiceTransportAction.java:151)
        at org.elasticsearch.transport.TransportService
$2.run(TransportService.java:199)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[prod-es-r06][inet[<a href="http://prod-es-r06.ihost.brewster.com/10.180.46.203:9300" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNE-CO2MbVGXna5s3XmFpb9Aj1mfqQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNE-CO2MbVGXna5s3XmFpb9Aj1mfqQ';return true;">prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
Node not connected
        at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:
636)
        at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:
448)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
181)
        ... 11 more

etc

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52d57a44-25b1-4d7c-8a03-d183959b40b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Another odd ES freak out...

Shobana Neelakantan
We are seeing the same issue as well.
Were you able to resolve this?

On Thursday, April 17, 2014 2:38:06 AM UTC+5:30, Vishal Bhasin wrote:
Hello - we are seeing the same issues, were you able to resolve this? thanks!

On Sunday, 5 February 2012 16:48:25 UTC-6, Grant wrote:
So we seem to be having recurring incidences of ES nodes getting into
a very odd state. In this particular case, one node because
unresponsive to test polls. I'm not really sure what to make of this,
because while this is ongoing, the cluster remains green, but the
borked node continues to try and service traffic, which means our app
is sporadically failing in the meantime.

[UTC Feb  5 22:29:23] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[<a href="http://10.180.48.216:9200/_cluster/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;">10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable
[UTC Feb  5 22:30:28] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[<a href="http://10.180.48.216:9200/_cluster/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;">10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable
[UTC Feb  5 22:31:33] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[<a href="http://10.180.48.216:9200/_cluster/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200%2F_cluster%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEzV1FzIps2HWAm_9a3mqr_Sf_wxg';return true;">10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable


Here's the corresponding logs from the node in question:

Some of this:

[2012-02-05 22:32:11,552][INFO ][discovery.zen            ] [prod-es-
r07] master_left [[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg][inet[prod-es-
<a href="http://r08.ihost.brewster.com/10.180.48.255:9300%5D%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr08.ihost.brewster.com%2F10.180.48.255%3A9300%255D%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNFIZHt2ZexK5ysgXKD8tHN-o7F8Ow';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr08.ihost.brewster.com%2F10.180.48.255%3A9300%255D%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNFIZHt2ZexK5ysgXKD8tHN-o7F8Ow';return true;">r08.ihost.brewster.com/10.180.48.255:9300]]], reason [no longer
master]
[2012-02-05 22:32:11,552][INFO ][cluster.service          ] [prod-es-
r07] master {new [prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
<a href="http://r04.ihost.brewster.com/10.180.35.110:9300%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr04.ihost.brewster.com%2F10.180.35.110%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNFCKV7VJw3TCMUkjYlLvIPUN-LDBw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr04.ihost.brewster.com%2F10.180.35.110%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNFCKV7VJw3TCMUkjYlLvIPUN-LDBw';return true;">r04.ihost.brewster.com/10.180.35.110:9300]], previous [prod-es-r08]
[mlrGPzm3QeCm7d_E_Lvozg][inet[<a href="http://prod-es-r08.ihost.brewster.com/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFrw6ixfqMSj2mvnognP28SqipbbA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFrw6ixfqMSj2mvnognP28SqipbbA';return true;">prod-es-r08.ihost.brewster.com/
10.180.48.255:9300]]}, removed {[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
[inet[<a href="http://prod-es-r08.ihost.brewster.com/10.180.48.255:9300%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F10.180.48.255%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF4w3OhamIMsrsYhvh42-i5BH9NYA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F10.180.48.255%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF4w3OhamIMsrsYhvh42-i5BH9NYA';return true;">prod-es-r08.ihost.brewster.com/10.180.48.255:9300]],}, reason:
zen-disco-master_failed ([prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
[inet[<a href="http://prod-es-r08.ihost.brewster.com/10.180.48.255:9300%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F10.180.48.255%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF4w3OhamIMsrsYhvh42-i5BH9NYA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r08.ihost.brewster.com%2F10.180.48.255%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF4w3OhamIMsrsYhvh42-i5BH9NYA';return true;">prod-es-r08.ihost.brewster.com/10.180.48.255:9300]])
[2012-02-05 22:32:12,557][INFO ][discovery.zen            ] [prod-es-
r07] master_left [[prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
<a href="http://r04.ihost.brewster.com/10.180.35.110:9300%5D%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr04.ihost.brewster.com%2F10.180.35.110%3A9300%255D%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF2oPcBQ9SAyB2q9xsxraIkMsRreA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr04.ihost.brewster.com%2F10.180.35.110%3A9300%255D%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNF2oPcBQ9SAyB2q9xsxraIkMsRreA';return true;">r04.ihost.brewster.com/10.180.35.110:9300]]], reason [no longer
master]
[2012-02-05 22:32:12,558][WARN ][discovery.zen            ] [prod-es-
r07] not enough master nodes after master left (reason = no longer
master), current nodes: {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg]
[inet[<a href="http://prod-es-r02.ihost.brewster.com/10.182.14.95:9300%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r02.ihost.brewster.com%2F10.182.14.95%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNEvoTxSTURYJj-C3SlpUA6c8CUx0Q';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r02.ihost.brewster.com%2F10.182.14.95%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNEvoTxSTURYJj-C3SlpUA6c8CUx0Q';return true;">prod-es-r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r07]
[zqJRs5e6S5eWfL0kVuolJg][inet[<a href="http://prod-es-r07.ihost.brewster.com/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r07.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGhVNkD_PqMdyhB4apIara2ndLuNA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r07.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGhVNkD_PqMdyhB4apIara2ndLuNA';return true;">prod-es-r07.ihost.brewster.com/
10.180.48.216:9300]],}
[2012-02-05 22:32:12,559][INFO ][cluster.service          ] [prod-es-
r07] removed {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg][inet[prod-es-
<a href="http://r02.ihost.brewster.com/10.182.14.95:9300%5D" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr02.ihost.brewster.com%2F10.182.14.95%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNE2ohjSaJXEbZHNMfORykB4Sl9R3A';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fr02.ihost.brewster.com%2F10.182.14.95%3A9300%255D\46sa\75D\46sntz\0751\46usg\75AFQjCNE2ohjSaJXEbZHNMfORykB4Sl9R3A';return true;">r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r04]
[uOUyy7p_TBuNEbwmqWF9-w][inet[<a href="http://prod-es-r04.ihost.brewster.com/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r04.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNE1Cbf2wOqrtrVL1scDl4zGXrwPtg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r04.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNE1Cbf2wOqrtrVL1scDl4zGXrwPtg';return true;">prod-es-r04.ihost.brewster.com/
10.180.35.110:9300]],}, reason: zen-disco-master_failed ([prod-es-r04]
[uOUyy7p_TBuNEbwmqWF9-w][inet[<a href="http://prod-es-r04.ihost.brewster.com/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r04.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNE1Cbf2wOqrtrVL1scDl4zGXrwPtg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r04.ihost.brewster.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNE1Cbf2wOqrtrVL1scDl4zGXrwPtg';return true;">prod-es-r04.ihost.brewster.com/
10.180.35.110:9300]])
[2012-02-05 22:32:12,565][WARN ][http.netty               ] [prod-es-
r07] Caught exception while handling client http traffic, closing
connection [id: 0x09be0e53, /<a href="http://10.180.48.216:54645" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A54645\46sa\75D\46sntz\0751\46usg\75AFQjCNFFz7G85e12vstnkddSBSYpGB2ZvQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A54645\46sa\75D\46sntz\0751\46usg\75AFQjCNFFz7G85e12vstnkddSBSYpGB2ZvQ';return true;">10.180.48.216:54645 => /
<a href="http://10.180.48.216:9200" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200\46sa\75D\46sntz\0751\46usg\75AFQjCNE9wtHwePKQbo-DKYPYeVEscO1-DA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200\46sa\75D\46sntz\0751\46usg\75AFQjCNE9wtHwePKQbo-DKYPYeVEscO1-DA';return true;">10.180.48.216:9200]
java.lang.IllegalArgumentException: empty text
        at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
103)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
68)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
81)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
198)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
107)
        at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
470)
        at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
443)
        at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
        at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
        at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
        at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
        at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
[2012-02-05 22:32:12,814][DEBUG][action.search.type       ] [prod-es-
r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
[scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
[2012-02-05 22:32:12,815][DEBUG][action.search.type       ] [prod-es-
r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
[scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
[2012-02-05 22:32:14,066][WARN ][http.netty               ] [prod-es-
r07] Caught exception while handling client http traffic, closing
connection [id: 0x2cb594cf, /<a href="http://10.180.48.216:54651" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A54651\46sa\75D\46sntz\0751\46usg\75AFQjCNGMAvblABAE3-qsNtvsriVr7rR_0w';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A54651\46sa\75D\46sntz\0751\46usg\75AFQjCNGMAvblABAE3-qsNtvsriVr7rR_0w';return true;">10.180.48.216:54651 => /
<a href="http://10.180.48.216:9200" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200\46sa\75D\46sntz\0751\46usg\75AFQjCNE9wtHwePKQbo-DKYPYeVEscO1-DA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F10.180.48.216%3A9200\46sa\75D\46sntz\0751\46usg\75AFQjCNE9wtHwePKQbo-DKYPYeVEscO1-DA';return true;">10.180.48.216:9200]





Followed by tons of this:


[2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
r07] [contact_documents-33-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [R],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@5c67fa3d]
org.elasticsearch.transport.SendRequestTransportException: [prod-es-
r06][inet[<a href="http://prod-es-r06.ihost.brewster.com/10.180.46.203:9300%5D%5D%5Bsearch/phase/query" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300%255D%255D%255Bsearch%2Fphase%2Fquery\46sa\75D\46sntz\0751\46usg\75AFQjCNH2IB7w98zPhSoupRTsddWsNmPFuw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300%255D%255D%255Bsearch%2Fphase%2Fquery\46sa\75D\46sntz\0751\46usg\75AFQjCNH2IB7w98zPhSoupRTsddWsNmPFuw';return true;">prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
phase/query]
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
196)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
168)
        at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:
140)
        at org.elasticsearch.action.search.type.TransportSearchCountAction
$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
279)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
        at org.elasticsearch.search.action.SearchServiceTransportAction
$2.handleException(SearchServiceTransportAction.java:151)
        at org.elasticsearch.transport.TransportService
$2.run(TransportService.java:199)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[prod-es-r06][inet[<a href="http://prod-es-r06.ihost.brewster.com/10.180.46.203:9300" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNE-CO2MbVGXna5s3XmFpb9Aj1mfqQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNE-CO2MbVGXna5s3XmFpb9Aj1mfqQ';return true;">prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
Node not connected
        at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:
636)
        at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:
448)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
181)
        ... 11 more
[2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
r07] [contact_documents-4-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@718585ec]
org.elasticsearch.transport.SendRequestTransportException: [prod-es-
r06][inet[<a href="http://prod-es-r06.ihost.brewster.com/10.180.46.203:9300%5D%5D%5Bsearch/phase/query" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300%255D%255D%255Bsearch%2Fphase%2Fquery\46sa\75D\46sntz\0751\46usg\75AFQjCNH2IB7w98zPhSoupRTsddWsNmPFuw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300%255D%255D%255Bsearch%2Fphase%2Fquery\46sa\75D\46sntz\0751\46usg\75AFQjCNH2IB7w98zPhSoupRTsddWsNmPFuw';return true;">prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
phase/query]
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
196)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
168)
        at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:
140)
        at org.elasticsearch.action.search.type.TransportSearchCountAction
$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
279)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
        at org.elasticsearch.search.action.SearchServiceTransportAction
$2.handleException(SearchServiceTransportAction.java:151)
        at org.elasticsearch.transport.TransportService
$2.run(TransportService.java:199)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[prod-es-r06][inet[<a href="http://prod-es-r06.ihost.brewster.com/10.180.46.203:9300" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNE-CO2MbVGXna5s3XmFpb9Aj1mfqQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fprod-es-r06.ihost.brewster.com%2F10.180.46.203%3A9300\46sa\75D\46sntz\0751\46usg\75AFQjCNE-CO2MbVGXna5s3XmFpb9Aj1mfqQ';return true;">prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
Node not connected
        at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:
636)
        at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:
448)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
181)
        ... 11 more

etc

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87767368-7870-4e24-a33f-a618e2c045ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Another odd ES freak out...

Jack Park-2
I saw a few of those which I ultimately attributed to a flaky network connection. That's not going to explain every such episode, but it did solve one of mine.

On Thu, Oct 2, 2014 at 11:27 PM, Shobana Neelakantan <[hidden email]> wrote:
We are seeing the same issue as well.
Were you able to resolve this?

On Thursday, April 17, 2014 2:38:06 AM UTC+5:30, Vishal Bhasin wrote:
Hello - we are seeing the same issues, were you able to resolve this? thanks!

On Sunday, 5 February 2012 16:48:25 UTC-6, Grant wrote:
So we seem to be having recurring incidences of ES nodes getting into
a very odd state. In this particular case, one node because
unresponsive to test polls. I'm not really sure what to make of this,
because while this is ongoing, the cluster remains green, but the
borked node continues to try and service traffic, which means our app
is sporadically failing in the meantime.

[UTC Feb  5 22:29:23] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable
[UTC Feb  5 22:30:28] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable
[UTC Feb  5 22:31:33] error    : 'prod_elasticsearch_cluster_health'
failed protocol test [HTTP] at INET[10.180.48.216:9200/_cluster/
health] via TCP -- HTTP: Error receiving data -- Resource temporarily
unavailable


Here's the corresponding logs from the node in question:

Some of this:

[2012-02-05 22:32:11,552][INFO ][discovery.zen            ] [prod-es-
r07] master_left [[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg][inet[prod-es-
r08.ihost.brewster.com/10.180.48.255:9300]]], reason [no longer
master]
[2012-02-05 22:32:11,552][INFO ][cluster.service          ] [prod-es-
r07] master {new [prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
r04.ihost.brewster.com/10.180.35.110:9300]], previous [prod-es-r08]
[mlrGPzm3QeCm7d_E_Lvozg][inet[prod-es-r08.ihost.brewster.com/
10.180.48.255:9300]]}, removed {[prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
[inet[prod-es-r08.ihost.brewster.com/10.180.48.255:9300]],}, reason:
zen-disco-master_failed ([prod-es-r08][mlrGPzm3QeCm7d_E_Lvozg]
[inet[prod-es-r08.ihost.brewster.com/10.180.48.255:9300]])
[2012-02-05 22:32:12,557][INFO ][discovery.zen            ] [prod-es-
r07] master_left [[prod-es-r04][uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-
r04.ihost.brewster.com/10.180.35.110:9300]]], reason [no longer
master]
[2012-02-05 22:32:12,558][WARN ][discovery.zen            ] [prod-es-
r07] not enough master nodes after master left (reason = no longer
master), current nodes: {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg]
[inet[prod-es-r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r07]
[zqJRs5e6S5eWfL0kVuolJg][inet[prod-es-r07.ihost.brewster.com/
10.180.48.216:9300]],}
[2012-02-05 22:32:12,559][INFO ][cluster.service          ] [prod-es-
r07] removed {[prod-es-r02][uuh4KmeHR-eUeIr7J97zCg][inet[prod-es-
r02.ihost.brewster.com/10.182.14.95:9300]],[prod-es-r04]
[uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-r04.ihost.brewster.com/
10.180.35.110:9300]],}, reason: zen-disco-master_failed ([prod-es-r04]
[uOUyy7p_TBuNEbwmqWF9-w][inet[prod-es-r04.ihost.brewster.com/
10.180.35.110:9300]])
[2012-02-05 22:32:12,565][WARN ][http.netty               ] [prod-es-
r07] Caught exception while handling client http traffic, closing
connection [id: 0x09be0e53, /10.180.48.216:54645 => /
10.180.48.216:9200]
java.lang.IllegalArgumentException: empty text
        at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
103)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
68)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
81)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
198)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
107)
        at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
470)
        at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
443)
        at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
        at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
        at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
        at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
        at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
[2012-02-05 22:32:12,814][DEBUG][action.search.type       ] [prod-es-
r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
[scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
[2012-02-05 22:32:12,815][DEBUG][action.search.type       ] [prod-es-
r07] Node [Bbaoza_KTP2DJQgxM4JN-A] not available for scroll request
[scan;1;5092799:Bbaoza_KTP2DJQgxM4JN-A;1;total_hits:7200;]
[2012-02-05 22:32:14,066][WARN ][http.netty               ] [prod-es-
r07] Caught exception while handling client http traffic, closing
connection [id: 0x2cb594cf, /10.180.48.216:54651 => /
10.180.48.216:9200]





Followed by tons of this:


[2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
r07] [contact_documents-33-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [R],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@5c67fa3d]
org.elasticsearch.transport.SendRequestTransportException: [prod-es-
r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
phase/query
]
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
196)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
168)
        at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:
140)
        at org.elasticsearch.action.search.type.TransportSearchCountAction
$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
279)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
        at org.elasticsearch.search.action.SearchServiceTransportAction
$2.handleException(SearchServiceTransportAction.java:151)
        at org.elasticsearch.transport.TransportService
$2.run(TransportService.java:199)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[prod-es-r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
Node not connected
        at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:
636)
        at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:
448)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
181)
        ... 11 more
[2012-02-05 22:32:24,572][DEBUG][action.search.type       ] [prod-es-
r07] [contact_documents-4-0][0], node[ar6qMqYnRSm5f0zvpKDirA], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@718585ec]
org.elasticsearch.transport.SendRequestTransportException: [prod-es-
r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]][search/
phase/query
]
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
196)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
168)
        at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:
140)
        at org.elasticsearch.action.search.type.TransportSearchCountAction
$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:74)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:
279)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:211)
        at org.elasticsearch.search.action.SearchServiceTransportAction
$2.handleException(SearchServiceTransportAction.java:151)
        at org.elasticsearch.transport.TransportService
$2.run(TransportService.java:199)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[prod-es-r06][inet[prod-es-r06.ihost.brewster.com/10.180.46.203:9300]]
Node not connected
        at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:
636)
        at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:
448)
        at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:
181)
        ... 11 more

etc

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87767368-7870-4e24-a33f-a618e2c045ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fybrLioU5OxgMnj4s5GCtzLzj8ZHuU7%2BMgm6%3Dp3KNwuZw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.