Rejected execution exception for multiple parallel requests.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Rejected execution exception for multiple parallel requests.

vipins
Hi,

I am querying elasticsearch for multiple parallel requests using single transport client instance in my application.

I got the below exception for the parallel execution. Hot to overcome the issue.

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@5f804c60
                at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
                at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
                at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
                at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:509)
                at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteScan(SearchServiceTransportAction.java:441)
                at org.elasticsearch.action.search.type.TransportSearchScanAction$AsyncAction.sendExecuteFirstPhase(TransportSearchScanAction.java:68)
                at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
                at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
                at org.elasticsearch.action.search.type.TransportSearchScanAction.doExecute(TransportSearchScanAction.java:52)
                at org.elasticsearch.action.search.type.TransportSearchScanAction.doExecute(TransportSearchScanAction.java:42)
                at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
                at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:107)
                at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
                at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
                at org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:124)
                at org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:113)
                at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:212)
                at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:109)
                at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
                at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
                at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
                at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
                at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
                at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
                at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
                at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
                at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
                at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
                at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
                at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
                at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
                at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
                at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
                at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
                at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
                at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
                at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
                at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
                at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
                at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                at java.lang.Thread.run(Thread.java:745)
Reply | Threaded
Open this post in threaded view
|

Re: Rejected execution exception for multiple parallel requests.

Radu Gheorghe-2
Hello,

I assume you query lots of shards/indices? If not, then it might just be that ES is overloaded with that many requests and you have to add nodes.

If yes, you'll can increase the queue size of the search thread pool. Something like:

curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "threadpool.search.size" : 5000
    }
}'

Note that ES will use more memory to keep the search requests until they get executed.

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Tue, Jan 6, 2015 at 8:44 AM, vipins <[hidden email]> wrote:
Hi,

I am querying elasticsearch for multiple parallel requests using single
transport client instance in my application.

I got the below exception for the parallel execution. Hot to overcome the
issue.

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 1000) on
org.elasticsearch.search.action.SearchServiceTransportAction$23@5f804c60
                at
org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
                at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
                at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
                at
org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:509)
                at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteScan(SearchServiceTransportAction.java:441)
                at
org.elasticsearch.action.search.type.TransportSearchScanAction$AsyncAction.sendExecuteFirstPhase(TransportSearchScanAction.java:68)
                at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
                at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
                at
org.elasticsearch.action.search.type.TransportSearchScanAction.doExecute(TransportSearchScanAction.java:52)
                at
org.elasticsearch.action.search.type.TransportSearchScanAction.doExecute(TransportSearchScanAction.java:42)
                at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
                at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:107)
                at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
                at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
                at
org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:124)
                at
org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:113)
                at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:212)
                at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:109)
                at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
                at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
                at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
                at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
                at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
                at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
                at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
                at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
                at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
                at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
                at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
                at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
                at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
                at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
                at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
                at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
                at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                at java.lang.Thread.run(Thread.java:745)



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Rejected-execution-exception-for-multiple-parallel-requests-tp4068538.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1420526659152-4068538.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_0eLYiZjhc33Dn-Mjy3KAFXzUwQ038w56d8JtQ4RhGicA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Rejected execution exception for multiple parallel requests.

vineeth mohan-2
Hi , 


Thanks
             Vineeth Mohan,
             Elasticsearch consultant,
             qbox.ioElasticsearch service provider)
   

On Wed, Jan 7, 2015 at 6:02 PM, Radu Gheorghe <[hidden email]> wrote:
Hello,

I assume you query lots of shards/indices? If not, then it might just be that ES is overloaded with that many requests and you have to add nodes.

If yes, you'll can increase the queue size of the search thread pool. Something like:

curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "threadpool.search.size" : 5000
    }
}'

Note that ES will use more memory to keep the search requests until they get executed.

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Tue, Jan 6, 2015 at 8:44 AM, vipins <[hidden email]> wrote:
Hi,

I am querying elasticsearch for multiple parallel requests using single
transport client instance in my application.

I got the below exception for the parallel execution. Hot to overcome the
issue.

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 1000) on
org.elasticsearch.search.action.SearchServiceTransportAction$23@5f804c60
                at
org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
                at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
                at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
                at
org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:509)
                at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteScan(SearchServiceTransportAction.java:441)
                at
org.elasticsearch.action.search.type.TransportSearchScanAction$AsyncAction.sendExecuteFirstPhase(TransportSearchScanAction.java:68)
                at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
                at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
                at
org.elasticsearch.action.search.type.TransportSearchScanAction.doExecute(TransportSearchScanAction.java:52)
                at
org.elasticsearch.action.search.type.TransportSearchScanAction.doExecute(TransportSearchScanAction.java:42)
                at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
                at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:107)
                at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
                at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
                at
org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:124)
                at
org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:113)
                at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:212)
                at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:109)
                at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
                at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
                at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
                at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
                at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
                at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
                at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
                at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
                at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
                at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
                at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
                at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
                at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
                at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
                at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
                at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
                at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
                at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                at java.lang.Thread.run(Thread.java:745)



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Rejected-execution-exception-for-multiple-parallel-requests-tp4068538.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1420526659152-4068538.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_0eLYiZjhc33Dn-Mjy3KAFXzUwQ038w56d8JtQ4RhGicA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5m3%2B9Zd0mDf1hUQpLFXu%3DKC5zDLYRs6EXZHM1kmvYzVhQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.