Not stable some time.

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Not stable some time.

darkyoung
Hi,

I'm using php client Elastica to communicate with ES, and I have two nodes running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and the load average raised.

I checked the ES logs and get the info below, is there something I can do to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES server.

[2012-03-04 13:49:29,639][DEBUG][http.netty               ] [Wizard] Caught exception while handling client http traffic, closing connection [id: 0x477966e4, /192.168.1.6:54837 :> /192.168.1.27:9200]
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)
at org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)
at org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)
at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Reply | Threaded
Open this post in threaded view
|

Re: Not stable some time.

darkyoung
If I restart the server, it will be better for some time, hours or days.

在 2012年3月4日星期日UTC+8下午2时05分57秒,Ocean Wu写道:
Hi,

I'm using php client Elastica to communicate with ES, and I have two nodes running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and the load average raised.

I checked the ES logs and get the info below, is there something I can do to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES server.

[2012-03-04 13:49:29,639][DEBUG][http.netty               ] [Wizard] Caught exception while handling client http traffic, closing connection [id: 0x477966e4, /192.168.1.6:54837 :> /192.168.1.27:9200]
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)
at org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)
at org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)
at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Reply | Threaded
Open this post in threaded view
|

Re: Not stable some time.

zhouxiang
i meet the same problerm.
did you fix it now ?
3ks

在 2012年3月4日星期日UTC+8下午2时23分09秒,Sean Wu写道:
If I restart the server, it will be better for some time, hours or days.

在 2012年3月4日星期日UTC+8下午2时05分57秒,Ocean Wu写道:
Hi,

I'm using php client Elastica to communicate with ES, and I have two nodes running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and the load average raised.

I checked the ES logs and get the info below, is there something I can do to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES server.

[2012-03-04 13:49:29,639][DEBUG][http.netty               ] [Wizard] Caught exception while handling client http traffic, closing connection [id: 0x477966e4, /192.168.1.6:54837 :> /192.168.1.27:9200]
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)
at org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)
at org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)
at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Reply | Threaded
Open this post in threaded view
|

Re: Not stable some time.

Shairon Toledo
I got same issue twice this week, the cluster only get back when all nodes are restarted. The http port hangs all connection too.
From Netty code I see 

 void writeFromUserCode(final AbstractNioChannel<?> channel) {
        if (!channel.isConnected()) {
            cleanUpWriteBuffer(channel);
            return;
        }

I infer that any node is closing the stream before buffer in the message event be read or write in transport layer.

I will try reproduce it by ES tests. Meanwhile if you have an idea please let me know.


ES 0.19.3
Indexer: Java ES client jar 0.19.0
Searcher: rubberband

My log:

[2012-08-23 20:12:37,704][DEBUG][http.netty               ] [prod-search-202] Caught exception while handling client http traffic, closing connection [id: 0x3fe40812, /10.4.70.11:53934 :> /10.251.94.255:9200]
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:673)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:400)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:59)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:733)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:65)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:712)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:679)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:245)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:150)
at org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:116)
at org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:106)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.finishHim(TransportNodesOperationAction.java:220)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onOperation(TransportNodesOperationAction.java:202)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$900(TransportNodesOperationAction.java:102)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:180)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:172)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:262)
at org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:233)
at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:141)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:95)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)


On Friday, August 10, 2012 1:03:53 AM UTC-3, shinezhou wrote:
i meet the same problerm.
did you fix it now ?
3ks

在 2012年3月4日星期日UTC+8下午2时23分09秒,Sean Wu写道:
If I restart the server, it will be better for some time, hours or days.

在 2012年3月4日星期日UTC+8下午2时05分57秒,Ocean Wu写道:
Hi,

I'm using php client Elastica to communicate with ES, and I have two nodes running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and the load average raised.

I checked the ES logs and get the info below, is there something I can do to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES server.

[2012-03-04 13:49:29,639][DEBUG][http.netty               ] [Wizard] Caught exception while handling client http traffic, closing connection [id: 0x477966e4, /192.168.1.6:54837 :> /192.168.1.27:9200]
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)
at org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)
at org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)
at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Not stable some time.

Shairon Toledo
I took a snapshot via jstack when it happened, the cpu was in 100%


 

On Thu, Aug 23, 2012 at 6:02 PM, Shairon Toledo <[hidden email]> wrote:
I got same issue twice this week, the cluster only get back when all nodes are restarted. The http port hangs all connection too.
From Netty code I see 

 void writeFromUserCode(final AbstractNioChannel<?> channel) {
        if (!channel.isConnected()) {
            cleanUpWriteBuffer(channel);
            return;
        }

I infer that any node is closing the stream before buffer in the message event be read or write in transport layer.

I will try reproduce it by ES tests. Meanwhile if you have an idea please let me know.


ES 0.19.3
Indexer: Java ES client jar 0.19.0
Searcher: rubberband

My log:

[2012-08-23 20:12:37,704][DEBUG][http.netty               ] [prod-search-202] Caught exception while handling client http traffic, closing connection [id: 0x3fe40812, /10.4.70.11:53934 :> /10.251.94.255:9200]
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:673)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:400)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:59)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:733)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:65)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:712)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:679)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:245)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:150)
at org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:116)
at org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:106)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.finishHim(TransportNodesOperationAction.java:220)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onOperation(TransportNodesOperationAction.java:202)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$900(TransportNodesOperationAction.java:102)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:180)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:172)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:262)
at org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:233)
at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:141)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:95)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)


On Friday, August 10, 2012 1:03:53 AM UTC-3, shinezhou wrote:
i meet the same problerm.
did you fix it now ?
3ks

在 2012年3月4日星期日UTC+8下午2时23分09秒,Sean Wu写道:
If I restart the server, it will be better for some time, hours or days.

在 2012年3月4日星期日UTC+8下午2时05分57秒,Ocean Wu写道:
Hi,

I'm using php client Elastica to communicate with ES, and I have two nodes running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and the load average raised.

I checked the ES logs and get the info below, is there something I can do to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES server.

[2012-03-04 13:49:29,639][DEBUG][http.netty               ] [Wizard] Caught exception while handling client http traffic, closing connection [id: 0x477966e4, /192.168.1.6:54837 :> /192.168.1.27:9200]
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)
at org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)
at org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)
at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

--
 
 



--











Shairon Toledo
http://hashcode.me

--