Indexing hangs via REST API

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Indexing hangs via REST API

ppearcy
Hello,
  I seem to be getting some hanging when indexing documents. I'm still
playing around to see if I can pin down exactly what is causing the
issue. Also, I am not sure if this is an ES issue, an issue with the
HTTP objects I am using to connect or most likely an OS network/socket
issue, as my indexer is running on Windows, submitting files to ES
running on linux. I've had my fair share of windows socket issues in
the past, caused by horrible default socket connections for the number
of ephemeral ports.

I am on master from last night.

I am using the python HTTPConnection object via pyelasticsearch to
index. When everything starts up indexing churns away for a while as
it should, but eventually my connection hangs and when this occurs I
see the exception below. I've hit this using a single index thread and
multiple (well, in my case it is actually seperate indexer process to
get around the GIL, but should be the same from ES perspective). This
is the same exception that I receive when I kill my indexer while
there is an open socket performing an index. Does the exception below
mean that the socket was closed by the client side before the data was
received or while it was flowing over the socket?

Any other thoughts or ideas would be much appreciated.

Thanks,
Paul

[18:01:47,706][INFO ][node                     ] [Jackhammer]
{elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
[18:17:18,553][WARN ][http.netty               ] [Jackhammer] Caught
exception while handling client http traffic
java.lang.IllegalArgumentException: empty text
        at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
103)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
68)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
81)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
198)
        at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
107)
        at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
461)
        at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
434)
        at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
545)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
754)
        at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
51)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
545)
        at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
540)
        at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
        at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
349)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
281)
        at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
201)
        at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
        at
org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
        at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Reply | Threaded
Open this post in threaded view
|

Re: Indexing hangs via REST API

ppearcy
Actually, the exception I get when I kill my indexer refers to
fireChannelDisconnected vs the exception on index hangs which points
at fireMessageReceived, so they are different cases. Apologies for any
confusion.

Thanks

On Aug 4, 12:41 pm, Paul <[hidden email]> wrote:

> Hello,
>   I seem to be getting some hanging when indexing documents. I'm still
> playing around to see if I can pin down exactly what is causing the
> issue. Also, I am not sure if this is an ES issue, an issue with the
> HTTP objects I am using to connect or most likely an OS network/socket
> issue, as my indexer is running on Windows, submitting files to ES
> running on linux. I've had my fair share of windows socket issues in
> the past, caused by horrible default socket connections for the number
> of ephemeral ports.
>
> I am on master from last night.
>
> I am using the python HTTPConnection object via pyelasticsearch to
> index. When everything starts up indexing churns away for a while as
> it should, but eventually my connection hangs and when this occurs I
> see the exception below. I've hit this using a single index thread and
> multiple (well, in my case it is actually seperate indexer process to
> get around the GIL, but should be the same from ES perspective). This
> is the same exception that I receive when I kill my indexer while
> there is an open socket performing an index. Does the exception below
> mean that the socket was closed by the client side before the data was
> received or while it was flowing over the socket?
>
> Any other thoughts or ideas would be much appreciated.
>
> Thanks,
> Paul
>
> [18:01:47,706][INFO ][node                     ] [Jackhammer]
> {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
> [18:17:18,553][WARN ][http.netty               ] [Jackhammer] Caught
> exception while handling client http traffic
> java.lang.IllegalArgumentException: empty text
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
> 103)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> 68)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> 81)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> 198)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> 107)
>         at
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> 461)
>         at
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> 434)
>         at
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> 80)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 545)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline
> $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> 754)
>         at
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> 51)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 545)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 540)
>         at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> 274)
>         at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> 261)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> 349)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> 281)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> 201)
>         at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> 108)
>         at
> org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> 46)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1110)
>         at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
Reply | Threaded
Open this post in threaded view
|

Re: Indexing hangs via REST API

kimchy
Administrator
Strange... . Can you netstat and see if the number of open sockets keep getting bigger? Might not use HTTP keep alive and you are running out of sockets on windows (not sure what happens in windows in this case...)

-shay.banon

On Wed, Aug 4, 2010 at 9:52 PM, Paul <[hidden email]> wrote:
Actually, the exception I get when I kill my indexer refers to
fireChannelDisconnected vs the exception on index hangs which points
at fireMessageReceived, so they are different cases. Apologies for any
confusion.

Thanks

On Aug 4, 12:41 pm, Paul <[hidden email]> wrote:
> Hello,
>   I seem to be getting some hanging when indexing documents. I'm still
> playing around to see if I can pin down exactly what is causing the
> issue. Also, I am not sure if this is an ES issue, an issue with the
> HTTP objects I am using to connect or most likely an OS network/socket
> issue, as my indexer is running on Windows, submitting files to ES
> running on linux. I've had my fair share of windows socket issues in
> the past, caused by horrible default socket connections for the number
> of ephemeral ports.
>
> I am on master from last night.
>
> I am using the python HTTPConnection object via pyelasticsearch to
> index. When everything starts up indexing churns away for a while as
> it should, but eventually my connection hangs and when this occurs I
> see the exception below. I've hit this using a single index thread and
> multiple (well, in my case it is actually seperate indexer process to
> get around the GIL, but should be the same from ES perspective). This
> is the same exception that I receive when I kill my indexer while
> there is an open socket performing an index. Does the exception below
> mean that the socket was closed by the client side before the data was
> received or while it was flowing over the socket?
>
> Any other thoughts or ideas would be much appreciated.
>
> Thanks,
> Paul
>
> [18:01:47,706][INFO ][node                     ] [Jackhammer]
> {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
> [18:17:18,553][WARN ][http.netty               ] [Jackhammer] Caught
> exception while handling client http traffic
> java.lang.IllegalArgumentException: empty text
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
> 103)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> 68)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> 81)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> 198)
>         at
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> 107)
>         at
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> 461)
>         at
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> 434)
>         at
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> 80)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 545)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline
> $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> 754)
>         at
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> 51)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 545)
>         at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 540)
>         at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> 274)
>         at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> 261)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> 349)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> 281)
>         at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> 201)
>         at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> 108)
>         at
> org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> 46)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1110)
>         at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)

Reply | Threaded
Open this post in threaded view
|

Re: Indexing hangs via REST API

ppearcy
When Windows hits its max ephemeral ports it will fail on new attempts
and nothing will ever make it over the wire.

Just to be safe, I applied the Windows registry settings here:
http://www.ibm.com/developerworks/lotus/library/quickr-tuning/

However, no luck. I did confirm via netstat that keep-alive as my
ephemeral port numbers aren't changing between documents and there
aren't any ES sockets in the WAIT_CLOSE state.

I just hit the condition in the multithreaded case (3 threads) and at
the time on two of the threads and my machine only has a few hundred
sockets active. When I brought down ES, the hung connections were
freed and my indexers started trying to process.

I am going to see if I can get a sniffer trace to see exactly what is
occurring over the wire. I may also play around with running all
processes on linux to avoid potential Windows gotchas.

I think using a HTTP timeout will allow me to work around whatever the
issue is. I have a hunch though that Netty or ES is not properly
closing the socket in this error case.

Thanks,
Paul

On Aug 4, 1:13 pm, Shay Banon <[hidden email]> wrote:

> Strange... . Can you netstat and see if the number of open sockets keep
> getting bigger? Might not use HTTP keep alive and you are running out of
> sockets on windows (not sure what happens in windows in this case...)
>
> -shay.banon
>
> On Wed, Aug 4, 2010 at 9:52 PM, Paul <[hidden email]> wrote:
> > Actually, the exception I get when I kill my indexer refers to
> > fireChannelDisconnected vs the exception on index hangs which points
> > at fireMessageReceived, so they are different cases. Apologies for any
> > confusion.
>
> > Thanks
>
> > On Aug 4, 12:41 pm, Paul <[hidden email]> wrote:
> > > Hello,
> > >   I seem to be getting some hanging when indexing documents. I'm still
> > > playing around to see if I can pin down exactly what is causing the
> > > issue. Also, I am not sure if this is an ES issue, an issue with the
> > > HTTP objects I am using to connect or most likely an OS network/socket
> > > issue, as my indexer is running on Windows, submitting files to ES
> > > running on linux. I've had my fair share of windows socket issues in
> > > the past, caused by horrible default socket connections for the number
> > > of ephemeral ports.
>
> > > I am on master from last night.
>
> > > I am using the python HTTPConnection object via pyelasticsearch to
> > > index. When everything starts up indexing churns away for a while as
> > > it should, but eventually my connection hangs and when this occurs I
> > > see the exception below. I've hit this using a single index thread and
> > > multiple (well, in my case it is actually seperate indexer process to
> > > get around the GIL, but should be the same from ES perspective). This
> > > is the same exception that I receive when I kill my indexer while
> > > there is an open socket performing an index. Does the exception below
> > > mean that the socket was closed by the client side before the data was
> > > received or while it was flowing over the socket?
>
> > > Any other thoughts or ideas would be much appreciated.
>
> > > Thanks,
> > > Paul
>
> > > [18:01:47,706][INFO ][node                     ] [Jackhammer]
> > > {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
> > > [18:17:18,553][WARN ][http.netty               ] [Jackhammer] Caught
> > > exception while handling client http traffic
> > > java.lang.IllegalArgumentException: empty text
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
> > > 103)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> > > 68)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> > > 81)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > > 198)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > > 107)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> > > 461)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> > > 434)
> > >         at
>
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> > > 80)
> > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > 545)
> > >         at
> > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline
> > > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> > > 754)
> > >         at
>
> > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> > > 51)
> > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > 545)
> > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > 540)
> > >         at
>
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > 274)
> > >         at
>
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > 261)
> > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> > > 349)
> > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> > > 281)
> > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> > > 201)
> > >         at
>
> > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> > > 108)
> > >         at
>
> > org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> > > 46)
> > >         at
>
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> > > 1110)
> > >         at java.util.concurrent.ThreadPoolExecutor
> > > $Worker.run(ThreadPoolExecutor.java:603)
> > >         at java.lang.Thread.run(Thread.java:636)
Reply | Threaded
Open this post in threaded view
|

Re: Indexing hangs via REST API

kimchy
Administrator
It can either be that, or a problem in elasticsearch and it does not send back a response for some reason. I suggest you try the second option, try and run everything on the linux box and see if it happens there?

-shay.banon

On Wed, Aug 4, 2010 at 11:03 PM, Paul <[hidden email]> wrote:
When Windows hits its max ephemeral ports it will fail on new attempts
and nothing will ever make it over the wire.

Just to be safe, I applied the Windows registry settings here:
http://www.ibm.com/developerworks/lotus/library/quickr-tuning/

However, no luck. I did confirm via netstat that keep-alive as my
ephemeral port numbers aren't changing between documents and there
aren't any ES sockets in the WAIT_CLOSE state.

I just hit the condition in the multithreaded case (3 threads) and at
the time on two of the threads and my machine only has a few hundred
sockets active. When I brought down ES, the hung connections were
freed and my indexers started trying to process.

I am going to see if I can get a sniffer trace to see exactly what is
occurring over the wire. I may also play around with running all
processes on linux to avoid potential Windows gotchas.

I think using a HTTP timeout will allow me to work around whatever the
issue is. I have a hunch though that Netty or ES is not properly
closing the socket in this error case.

Thanks,
Paul

On Aug 4, 1:13 pm, Shay Banon <[hidden email]> wrote:
> Strange... . Can you netstat and see if the number of open sockets keep
> getting bigger? Might not use HTTP keep alive and you are running out of
> sockets on windows (not sure what happens in windows in this case...)
>
> -shay.banon
>
> On Wed, Aug 4, 2010 at 9:52 PM, Paul <[hidden email]> wrote:
> > Actually, the exception I get when I kill my indexer refers to
> > fireChannelDisconnected vs the exception on index hangs which points
> > at fireMessageReceived, so they are different cases. Apologies for any
> > confusion.
>
> > Thanks
>
> > On Aug 4, 12:41 pm, Paul <[hidden email]> wrote:
> > > Hello,
> > >   I seem to be getting some hanging when indexing documents. I'm still
> > > playing around to see if I can pin down exactly what is causing the
> > > issue. Also, I am not sure if this is an ES issue, an issue with the
> > > HTTP objects I am using to connect or most likely an OS network/socket
> > > issue, as my indexer is running on Windows, submitting files to ES
> > > running on linux. I've had my fair share of windows socket issues in
> > > the past, caused by horrible default socket connections for the number
> > > of ephemeral ports.
>
> > > I am on master from last night.
>
> > > I am using the python HTTPConnection object via pyelasticsearch to
> > > index. When everything starts up indexing churns away for a while as
> > > it should, but eventually my connection hangs and when this occurs I
> > > see the exception below. I've hit this using a single index thread and
> > > multiple (well, in my case it is actually seperate indexer process to
> > > get around the GIL, but should be the same from ES perspective). This
> > > is the same exception that I receive when I kill my indexer while
> > > there is an open socket performing an index. Does the exception below
> > > mean that the socket was closed by the client side before the data was
> > > received or while it was flowing over the socket?
>
> > > Any other thoughts or ideas would be much appreciated.
>
> > > Thanks,
> > > Paul
>
> > > [18:01:47,706][INFO ][node                     ] [Jackhammer]
> > > {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
> > > [18:17:18,553][WARN ][http.netty               ] [Jackhammer] Caught
> > > exception while handling client http traffic
> > > java.lang.IllegalArgumentException: empty text
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
> > > 103)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> > > 68)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> > > 81)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > > 198)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > > 107)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> > > 461)
> > >         at
>
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> > > 434)
> > >         at
>
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> > > 80)
> > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > 545)
> > >         at
> > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline
> > > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> > > 754)
> > >         at
>
> > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> > > 51)
> > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > 545)
> > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > 540)
> > >         at
>
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > 274)
> > >         at
>
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > 261)
> > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> > > 349)
> > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> > > 281)
> > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> > > 201)
> > >         at
>
> > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> > > 108)
> > >         at
>
> > org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> > > 46)
> > >         at
>
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> > > 1110)
> > >         at java.util.concurrent.ThreadPoolExecutor
> > > $Worker.run(ThreadPoolExecutor.java:603)
> > >         at java.lang.Thread.run(Thread.java:636)

Reply | Threaded
Open this post in threaded view
|

Re: Indexing hangs via REST API

ppearcy
OK, got to the bottom of it... Major bug on my part and minor bug on
the part of ES. I'm throwing a lot of different types of data at the
system and should have checked if this was occurring on a specific
type.

It appears that I had a newline in the path that I was putting indexed
data to. In this case, it appears that ES fails to write an error back
into the connection and a hang occurs.

I've opened an issue for it and included a test to reproduce:

http://github.com/elasticsearch/elasticsearch/issues/issue/297

Thanks!

On Aug 4, 3:14 pm, Shay Banon <[hidden email]> wrote:

> It can either be that, or a problem in elasticsearch and it does not send
> back a response for some reason. I suggest you try the second option, try
> and run everything on the linux box and see if it happens there?
>
> -shay.banon
>
> On Wed, Aug 4, 2010 at 11:03 PM, Paul <[hidden email]> wrote:
> > When Windows hits its max ephemeral ports it will fail on new attempts
> > and nothing will ever make it over the wire.
>
> > Just to be safe, I applied the Windows registry settings here:
> >http://www.ibm.com/developerworks/lotus/library/quickr-tuning/
>
> > However, no luck. I did confirm via netstat that keep-alive as my
> > ephemeral port numbers aren't changing between documents and there
> > aren't any ES sockets in the WAIT_CLOSE state.
>
> > I just hit the condition in the multithreaded case (3 threads) and at
> > the time on two of the threads and my machine only has a few hundred
> > sockets active. When I brought down ES, the hung connections were
> > freed and my indexers started trying to process.
>
> > I am going to see if I can get a sniffer trace to see exactly what is
> > occurring over the wire. I may also play around with running all
> > processes on linux to avoid potential Windows gotchas.
>
> > I think using a HTTP timeout will allow me to work around whatever the
> > issue is. I have a hunch though that Netty or ES is not properly
> > closing the socket in this error case.
>
> > Thanks,
> > Paul
>
> > On Aug 4, 1:13 pm, Shay Banon <[hidden email]> wrote:
> > > Strange... . Can you netstat and see if the number of open sockets keep
> > > getting bigger? Might not use HTTP keep alive and you are running out of
> > > sockets on windows (not sure what happens in windows in this case...)
>
> > > -shay.banon
>
> > > On Wed, Aug 4, 2010 at 9:52 PM, Paul <[hidden email]> wrote:
> > > > Actually, the exception I get when I kill my indexer refers to
> > > > fireChannelDisconnected vs the exception on index hangs which points
> > > > at fireMessageReceived, so they are different cases. Apologies for any
> > > > confusion.
>
> > > > Thanks
>
> > > > On Aug 4, 12:41 pm, Paul <[hidden email]> wrote:
> > > > > Hello,
> > > > >   I seem to be getting some hanging when indexing documents. I'm
> > still
> > > > > playing around to see if I can pin down exactly what is causing the
> > > > > issue. Also, I am not sure if this is an ES issue, an issue with the
> > > > > HTTP objects I am using to connect or most likely an OS
> > network/socket
> > > > > issue, as my indexer is running on Windows, submitting files to ES
> > > > > running on linux. I've had my fair share of windows socket issues in
> > > > > the past, caused by horrible default socket connections for the
> > number
> > > > > of ephemeral ports.
>
> > > > > I am on master from last night.
>
> > > > > I am using the python HTTPConnection object via pyelasticsearch to
> > > > > index. When everything starts up indexing churns away for a while as
> > > > > it should, but eventually my connection hangs and when this occurs I
> > > > > see the exception below. I've hit this using a single index thread
> > and
> > > > > multiple (well, in my case it is actually seperate indexer process to
> > > > > get around the GIL, but should be the same from ES perspective). This
> > > > > is the same exception that I receive when I kill my indexer while
> > > > > there is an open socket performing an index. Does the exception below
> > > > > mean that the socket was closed by the client side before the data
> > was
> > > > > received or while it was flowing over the socket?
>
> > > > > Any other thoughts or ideas would be much appreciated.
>
> > > > > Thanks,
> > > > > Paul
>
> > > > > [18:01:47,706][INFO ][node                     ] [Jackhammer]
> > > > > {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
> > > > > [18:17:18,553][WARN ][http.netty               ] [Jackhammer] Caught
> > > > > exception while handling client http traffic
> > > > > java.lang.IllegalArgumentException: empty text
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
> > > > > 103)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> > > > > 68)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> > > > > 81)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > > > > 198)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > > > > 107)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> > > > > 461)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> > > > > 434)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> > > > > 80)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > > > 545)
> > > > >         at
> > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline
>
> > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> > > > > 754)
> > > > >         at
>
> > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> > > > > 51)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > > > 545)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > > > 540)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > > > 274)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > > > 261)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> > > > > 349)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> > > > > 281)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> > > > > 201)
> > > > >         at
>
> > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> > > > > 108)
> > > > >         at
>
> > org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> > > > > 46)
> > > > >         at
>
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> > > > > 1110)
> > > > >         at java.util.concurrent.ThreadPoolExecutor
> > > > > $Worker.run(ThreadPoolExecutor.java:603)
> > > > >         at java.lang.Thread.run(Thread.java:636)
Reply | Threaded
Open this post in threaded view
|

Re: Indexing hangs via REST API

kimchy
Administrator
great, thanks for the work!. I will look at the case and fix it.

On Thu, Aug 5, 2010 at 7:08 AM, Paul <[hidden email]> wrote:
OK, got to the bottom of it... Major bug on my part and minor bug on
the part of ES. I'm throwing a lot of different types of data at the
system and should have checked if this was occurring on a specific
type.

It appears that I had a newline in the path that I was putting indexed
data to. In this case, it appears that ES fails to write an error back
into the connection and a hang occurs.

I've opened an issue for it and included a test to reproduce:

http://github.com/elasticsearch/elasticsearch/issues/issue/297

Thanks!

On Aug 4, 3:14 pm, Shay Banon <[hidden email]> wrote:
> It can either be that, or a problem in elasticsearch and it does not send
> back a response for some reason. I suggest you try the second option, try
> and run everything on the linux box and see if it happens there?
>
> -shay.banon
>
> On Wed, Aug 4, 2010 at 11:03 PM, Paul <[hidden email]> wrote:
> > When Windows hits its max ephemeral ports it will fail on new attempts
> > and nothing will ever make it over the wire.
>
> > Just to be safe, I applied the Windows registry settings here:
> >http://www.ibm.com/developerworks/lotus/library/quickr-tuning/
>
> > However, no luck. I did confirm via netstat that keep-alive as my
> > ephemeral port numbers aren't changing between documents and there
> > aren't any ES sockets in the WAIT_CLOSE state.
>
> > I just hit the condition in the multithreaded case (3 threads) and at
> > the time on two of the threads and my machine only has a few hundred
> > sockets active. When I brought down ES, the hung connections were
> > freed and my indexers started trying to process.
>
> > I am going to see if I can get a sniffer trace to see exactly what is
> > occurring over the wire. I may also play around with running all
> > processes on linux to avoid potential Windows gotchas.
>
> > I think using a HTTP timeout will allow me to work around whatever the
> > issue is. I have a hunch though that Netty or ES is not properly
> > closing the socket in this error case.
>
> > Thanks,
> > Paul
>
> > On Aug 4, 1:13 pm, Shay Banon <[hidden email]> wrote:
> > > Strange... . Can you netstat and see if the number of open sockets keep
> > > getting bigger? Might not use HTTP keep alive and you are running out of
> > > sockets on windows (not sure what happens in windows in this case...)
>
> > > -shay.banon
>
> > > On Wed, Aug 4, 2010 at 9:52 PM, Paul <[hidden email]> wrote:
> > > > Actually, the exception I get when I kill my indexer refers to
> > > > fireChannelDisconnected vs the exception on index hangs which points
> > > > at fireMessageReceived, so they are different cases. Apologies for any
> > > > confusion.
>
> > > > Thanks
>
> > > > On Aug 4, 12:41 pm, Paul <[hidden email]> wrote:
> > > > > Hello,
> > > > >   I seem to be getting some hanging when indexing documents. I'm
> > still
> > > > > playing around to see if I can pin down exactly what is causing the
> > > > > issue. Also, I am not sure if this is an ES issue, an issue with the
> > > > > HTTP objects I am using to connect or most likely an OS
> > network/socket
> > > > > issue, as my indexer is running on Windows, submitting files to ES
> > > > > running on linux. I've had my fair share of windows socket issues in
> > > > > the past, caused by horrible default socket connections for the
> > number
> > > > > of ephemeral ports.
>
> > > > > I am on master from last night.
>
> > > > > I am using the python HTTPConnection object via pyelasticsearch to
> > > > > index. When everything starts up indexing churns away for a while as
> > > > > it should, but eventually my connection hangs and when this occurs I
> > > > > see the exception below. I've hit this using a single index thread
> > and
> > > > > multiple (well, in my case it is actually seperate indexer process to
> > > > > get around the GIL, but should be the same from ES perspective). This
> > > > > is the same exception that I receive when I kill my indexer while
> > > > > there is an open socket performing an index. Does the exception below
> > > > > mean that the socket was closed by the client side before the data
> > was
> > > > > received or while it was flowing over the socket?
>
> > > > > Any other thoughts or ideas would be much appreciated.
>
> > > > > Thanks,
> > > > > Paul
>
> > > > > [18:01:47,706][INFO ][node                     ] [Jackhammer]
> > > > > {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
> > > > > [18:17:18,553][WARN ][http.netty               ] [Jackhammer] Caught
> > > > > exception while handling client http traffic
> > > > > java.lang.IllegalArgumentException: empty text
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:
> > > > > 103)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> > > > > 68)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> > > > > 81)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > > > > 198)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > > > > 107)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> > > > > 461)
> > > > >         at
>
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> > > > > 434)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> > > > > 80)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > > > 545)
> > > > >         at
> > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline
>
> > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> > > > > 754)
> > > > >         at
>
> > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> > > > > 51)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > > > 545)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > > > > 540)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > > > 274)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > > > > 261)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> > > > > 349)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> > > > > 281)
> > > > >         at
>
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> > > > > 201)
> > > > >         at
>
> > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> > > > > 108)
> > > > >         at
>
> > org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> > > > > 46)
> > > > >         at
>
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> > > > > 1110)
> > > > >         at java.util.concurrent.ThreadPoolExecutor
> > > > > $Worker.run(ThreadPoolExecutor.java:603)
> > > > >         at java.lang.Thread.run(Thread.java:636)