Quantcast

High failure rates on REST indexing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

High failure rates on REST indexing

Tim Dysinger-2-3
I am rapidly pushing 750k documents to one elasticsearch node
(localhost).  About 3% of them fail to push over rest. The json is
fine.  I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty               ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
        at
org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:
39)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
391)
        at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:
499)
        at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
490)
        at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
427)
        at
org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:
156)
        at
org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
49)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
345)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
332)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
        at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
275)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
        at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High failure rates on REST indexing

kimchy
Administrator
Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger <[hidden email]> wrote:

> I am rapidly pushing 750k documents to one elasticsearch node
> (localhost).  About 3% of them fail to push over rest. The json is
> fine.  I've pushed all these to CouchDB before.
>
> The console just fills up with this
>
> [11:55:54,225][WARN ][http.netty               ] [Reilly, Ben] Caught
> exception while handling client http trafic
> java.lang.ClassCastException:
> org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
> org.jboss.netty.handler.codec.http.HttpRequest
> at
> org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:
> 39)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> 391)
> at
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:
> 499)
> at
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> 490)
> at
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> 427)
> at
> org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:
> 156)
> at
> org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> 49)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> 345)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> 332)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> 323)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> 275)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> 196)
> at
> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> 46)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>

--
Sent from my mobile device
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High failure rates on REST indexing

Tim Dysinger-2-3
Using this client http://github.com/francois/rest-client

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon <[hidden email]> wrote:

> Hey,
>
> This happens basically because there is no support for http chunk yet,
> which I guess the http client you use does for large requests.
>
> Is there a chance that you can disable chunking for now? Which http
> client are you using?
>
> -shay.banon
>
> On 2/13/10, Tim Dysinger <[hidden email]> wrote:
>
>
>
>
>
> > I am rapidly pushing 750k documents to one elasticsearch node
> > (localhost).  About 3% of them fail to push over rest. The json is
> > fine.  I've pushed all these to CouchDB before.
>
> > The console just fills up with this
>
> > [11:55:54,225][WARN ][http.netty               ] [Reilly, Ben] Caught
> > exception while handling client http trafic
> > java.lang.ClassCastException:
> > org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
> > org.jboss.netty.handler.codec.http.HttpRequest
> >    at
> > org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest Handler.java:
> > 39)
> >    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 391)
> >    at
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR eceived(ReplayingDecoder.java:
> > 499)
> >    at
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD ecoder.java:
> > 490)
> >    at
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla yingDecoder.java:
> > 427)
> >    at
> > org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime outHandler.java:
> > 156)
> >    at
> > org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel sHandler.java:
> > 49)
> >    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 345)
> >    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 332)
> >    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> > 323)
> >    at
> > org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker. java:
> > 275)
> >    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> > 196)
> >    at
> > org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> > 46)
> >    at java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:886)
> >    at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:908)
> >    at java.lang.Thread.run(Thread.java:619)
>
> --
> Sent from my mobile device
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High failure rates on REST indexing

kimchy
Administrator
Support for HTTP chunks has been added to master: http://github.com/elasticsearch/elasticsearch/issues/closed#issue/8. You can get it here: http://www.elasticsearch.com/download/ (with instructions on how to build it).

-shay.banon

On Sat, Feb 13, 2010 at 10:28 PM, Tim Dysinger <[hidden email]> wrote:
Using this client http://github.com/francois/rest-client

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon <[hidden email]> wrote:
> Hey,
>
> This happens basically because there is no support for http chunk yet,
> which I guess the http client you use does for large requests.
>
> Is there a chance that you can disable chunking for now? Which http
> client are you using?
>
> -shay.banon
>
> On 2/13/10, Tim Dysinger <[hidden email]> wrote:
>
>
>
>
>
> > I am rapidly pushing 750k documents to one elasticsearch node
> > (localhost).  About 3% of them fail to push over rest. The json is
> > fine.  I've pushed all these to CouchDB before.
>
> > The console just fills up with this
>
> > [11:55:54,225][WARN ][http.netty               ] [Reilly, Ben] Caught
> > exception while handling client http trafic
> > java.lang.ClassCastException:
> > org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
> > org.jboss.netty.handler.codec.http.HttpRequest
> >    at
> > org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest Handler.java:
> > 39)
> >    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 391)
> >    at
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR eceived(ReplayingDecoder.java:
> > 499)
> >    at
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD ecoder.java:
> > 490)
> >    at
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla yingDecoder.java:
> > 427)
> >    at
> > org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime outHandler.java:
> > 156)
> >    at
> > org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel sHandler.java:
> > 49)
> >    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 345)
> >    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 332)
> >    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> > 323)
> >    at
> > org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker. java:
> > 275)
> >    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> > 196)
> >    at
> > org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> > 46)
> >    at java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:886)
> >    at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:908)
> >    at java.lang.Thread.run(Thread.java:619)
>
> --
> Sent from my mobile device

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High failure rates on REST indexing

kimchy
Administrator
By the way, it would be interesting the know the numbers you are getting. If you can, also post the environment you are running on (os, number of CPUs), the client type, number of clients (concurrent), typical document size and content, elasticsearch configuration (both node level and index level), cpu taken by client and by elasticsearch, and anything else you find useful.

-shay.banon

On Sat, Feb 13, 2010 at 11:46 PM, Shay Banon <[hidden email]> wrote:
Support for HTTP chunks has been added to master: http://github.com/elasticsearch/elasticsearch/issues/closed#issue/8. You can get it here: http://www.elasticsearch.com/download/ (with instructions on how to build it).

-shay.banon


On Sat, Feb 13, 2010 at 10:28 PM, Tim Dysinger <[hidden email]> wrote:
Using this client http://github.com/francois/rest-client

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon <[hidden email]> wrote:
> Hey,
>
> This happens basically because there is no support for http chunk yet,
> which I guess the http client you use does for large requests.
>
> Is there a chance that you can disable chunking for now? Which http
> client are you using?
>
> -shay.banon
>
> On 2/13/10, Tim Dysinger <[hidden email]> wrote:
>
>
>
>
>
> > I am rapidly pushing 750k documents to one elasticsearch node
> > (localhost).  About 3% of them fail to push over rest. The json is
> > fine.  I've pushed all these to CouchDB before.
>
> > The console just fills up with this
>
> > [11:55:54,225][WARN ][http.netty               ] [Reilly, Ben] Caught
> > exception while handling client http trafic
> > java.lang.ClassCastException:
> > org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
> > org.jboss.netty.handler.codec.http.HttpRequest
> >    at
> > org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest Handler.java:
> > 39)
> >    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 391)
> >    at
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR eceived(ReplayingDecoder.java:
> > 499)
> >    at
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD ecoder.java:
> > 490)
> >    at
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla yingDecoder.java:
> > 427)
> >    at
> > org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime outHandler.java:
> > 156)
> >    at
> > org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel sHandler.java:
> > 49)
> >    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 345)
> >    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 332)
> >    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> > 323)
> >    at
> > org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker. java:
> > 275)
> >    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> > 196)
> >    at
> > org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> > 46)
> >    at java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:886)
> >    at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:908)
> >    at java.lang.Thread.run(Thread.java:619)
>
> --
> Sent from my mobile device


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High failure rates on REST indexing

Tim Dysinger-2-3
In reply to this post by kimchy
Thanks.  It works now.

On Feb 13, 1:46 pm, Shay Banon <[hidden email]> wrote:

> Support for HTTP chunks has been added to master:http://github.com/elasticsearch/elasticsearch/issues/closed#issue/8. You can
> get it here:http://www.elasticsearch.com/download/(with instructions on
> how to build it).
>
> -shay.banon
>
>
>
> On Sat, Feb 13, 2010 at 10:28 PM, Tim Dysinger <[hidden email]> wrote:
> > Using this clienthttp://github.com/francois/rest-client
>
> > I am not explicitly doing chunking but it's possible it's doing this
> > behind the scenes on larger documents.
>
> > On Feb 13, 12:19 pm, Shay Banon <[hidden email]> wrote:
> > > Hey,
>
> > > This happens basically because there is no support for http chunk yet,
> > > which I guess the http client you use does for large requests.
>
> > > Is there a chance that you can disable chunking for now? Which http
> > > client are you using?
>
> > > -shay.banon
>
> > > On 2/13/10, Tim Dysinger <[hidden email]> wrote:
>
> > > > I am rapidly pushing 750k documents to one elasticsearch node
> > > > (localhost).  About 3% of them fail to push over rest. The json is
> > > > fine.  I've pushed all these to CouchDB before.
>
> > > > The console just fills up with this
>
> > > > [11:55:54,225][WARN ][http.netty               ] [Reilly, Ben] Caught
> > > > exception while handling client http trafic
> > > > java.lang.ClassCastException:
> > > > org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
> > > > org.jboss.netty.handler.codec.http.HttpRequest
> > > >    at
>
> > org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest
> > Handler.java:
> > > > 39)
> > > >    at
> > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > > 391)
> > > >    at
>
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR
> > eceived(ReplayingDecoder.java:
> > > > 499)
> > > >    at
>
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD
> > ecoder.java:
> > > > 490)
> > > >    at
>
> > org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla
> > yingDecoder.java:
> > > > 427)
> > > >    at
>
> > org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime
> > outHandler.java:
> > > > 156)
> > > >    at
>
> > org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel
> > sHandler.java:
> > > > 49)
> > > >    at
> > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > > 345)
> > > >    at
> > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > > 332)
> > > >    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> > > > 323)
> > > >    at
>
> > org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.
> > java:
> > > > 275)
> > > >    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> > > > 196)
> > > >    at
>
> > org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> > > > 46)
> > > >    at java.util.concurrent.ThreadPoolExecutor
> > > > $Worker.runTask(ThreadPoolExecutor.java:886)
> > > >    at java.util.concurrent.ThreadPoolExecutor
> > > > $Worker.run(ThreadPoolExecutor.java:908)
> > > >    at java.lang.Thread.run(Thread.java:619)
>
> > > --
> > > Sent from my mobile device
Loading...