Quantcast

Errors on replicated setup

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Errors on replicated setup

vibrant
I have a setup with 2 servers:

openjdk version "1.7.0_06"
OpenJDK Runtime Environment (build 1.7.0_06-b24)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

elasticsearch-0.19.9

I have the following config on both:

cluster.name: elasticsearch
node.name: "death" (and on the other node.name "life")
discovery.zen.ping.multicast.enabled: true
discovery.zen.ping.unicast.hosts: ["10.24.233.196","10.17.57.4"]
 
So I start adding documents to the 'life' node, and everything seems to be going good except from time to time I get the below logs/exceptions on both machines.  Please tell me what I could be doing wrong.

[2012-09-26 08:45:15,963][WARN ][cluster.action.shard     ] [death] received shard failed for [prod_threads][0], node[mu_BJdkWRkyYewRElqxzaw], [R], s[STARTED], reason [
Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[F
ailed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]


[2012-09-26 08:45:15,942][WARN ][action.index             ] [life] Failed to perform index on replica [prod_threads][0]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream
        at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:171)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:793)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:390)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

Caused by: java.io.StreamCorruptedException: unexpected end of block data
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
        at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:498)
        at java.lang.Throwable.readObject(Throwable.java:913)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
        at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:498)
        at java.lang.Throwable.readObject(Throwable.java:913)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
        at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
        ... 22 more
[2012-09-26 08:45:15,943][WARN ][cluster.action.shard     ] [life] sending failed shard for [prod_threads][0], node[mu_BJdkWRkyYewRElqxzaw], [R], s[STARTED], reason [Failed to perform [index] on replica, message [Remote
TransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of
 block data]; ]]
[2012-09-26 08:45:15,943][WARN ][transport.netty          ] [life] Message not fully read (response) for [6224] handler org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOpe
rationAction$4@29a198c9, error [true], resetting

--
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Errors on replicated setup

Sapana Patel
Hi,

I am facing same issue in my application
Have you found the solution? please share the solution with all

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Errors on replicated setup

Germain Maurice
In reply to this post by vibrant
I have the same kind of this problem.
I have a 2 nodes cluster on version 0.20.5, but previously have the same problem with 0.19.11.


Le mercredi 26 septembre 2012 10:51:47 UTC+2, vibrant a écrit :
I have a setup with 2 servers:

openjdk version "1.7.0_06"
OpenJDK Runtime Environment (build 1.7.0_06-b24)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

elasticsearch-0.19.9

I have the following config on both:

cluster.name: elasticsearch
node.name: "death" (and on the other node.name "life")
discovery.zen.ping.multicast.enabled: true
discovery.zen.ping.unicast.hosts: ["10.24.233.196","10.17.57.4"]
 
So I start adding documents to the 'life' node, and everything seems to be going good except from time to time I get the below logs/exceptions on both machines.  Please tell me what I could be doing wrong.

[2012-09-26 08:45:15,963][WARN ][cluster.action.shard     ] [death] received shard failed for [prod_threads][0], node[mu_BJdkWRkyYewRElqxzaw], [R], s[STARTED], reason [
Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[F
ailed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]


[2012-09-26 08:45:15,942][WARN ][action.index             ] [life] Failed to perform index on replica [prod_threads][0]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream
        at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:171)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:793)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:390)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

Caused by: java.io.StreamCorruptedException: unexpected end of block data
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
        at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:498)
        at java.lang.Throwable.readObject(Throwable.java:913)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
        at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:498)
        at java.lang.Throwable.readObject(Throwable.java:913)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
        at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
        ... 22 more
[2012-09-26 08:45:15,943][WARN ][cluster.action.shard     ] [life] sending failed shard for [prod_threads][0], node[mu_BJdkWRkyYewRElqxzaw], [R], s[STARTED], reason [Failed to perform [index] on replica, message [Remote
TransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of
 block data]; ]]
[2012-09-26 08:45:15,943][WARN ][transport.netty          ] [life] Message not fully read (response) for [6224] handler org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOpe
rationAction$4@29a198c9, error [true], resetting

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Errors on replicated setup

Mohammady Mahdy
I've seen this before when your nodes are not the exact same version

On Thursday, March 7, 2013 5:08:21 PM UTC+2, Gmaurice wrote:
I have the same kind of this problem.
I have a 2 nodes cluster on version 0.20.5, but previously have the same problem with 0.19.11.


Le mercredi 26 septembre 2012 10:51:47 UTC+2, vibrant a écrit :
I have a setup with 2 servers:

openjdk version "1.7.0_06"
OpenJDK Runtime Environment (build 1.7.0_06-b24)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

elasticsearch-0.19.9

I have the following config on both:

cluster.name: elasticsearch
node.name: "death" (and on the other node.name "life")
discovery.zen.ping.multicast.enabled: true
discovery.zen.ping.unicast.hosts: ["10.24.233.196","10.17.57.4"]
 
So I start adding documents to the 'life' node, and everything seems to be going good except from time to time I get the below logs/exceptions on both machines.  Please tell me what I could be doing wrong.

[2012-09-26 08:45:15,963][WARN ][cluster.action.shard     ] [death] received shard failed for [prod_threads][0], node[mu_BJdkWRkyYewRElqxzaw], [R], s[STARTED], reason [
Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[F
ailed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]


[2012-09-26 08:45:15,942][WARN ][action.index             ] [life] Failed to perform index on replica [prod_threads][0]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream
        at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:171)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:793)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:390)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

Caused by: java.io.StreamCorruptedException: unexpected end of block data
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
        at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:498)
        at java.lang.Throwable.readObject(Throwable.java:913)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
        at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:498)
        at java.lang.Throwable.readObject(Throwable.java:913)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
        at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
        ... 22 more
[2012-09-26 08:45:15,943][WARN ][cluster.action.shard     ] [life] sending failed shard for [prod_threads][0], node[mu_BJdkWRkyYewRElqxzaw], [R], s[STARTED], reason [Failed to perform [index] on replica, message [Remote
TransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of
 block data]; ]]
[2012-09-26 08:45:15,943][WARN ][transport.netty          ] [life] Message not fully read (response) for [6224] handler org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOpe
rationAction$4@29a198c9, error [true], resetting

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Loading...