elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

bagui
Hi,

I'm using elasticsearch using TransportClient for multiple operation. The issue I'm facing now is if my es server goes down my client side app getting OutOfMemoryError.  Getting the below exception. I had to restart my tomcat every time after this to make my application up. Can some one please suggest how to prevent this. 


Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [spring] in context with path [/aricloud] threw exception [Handler processing failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread] with root cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)


Thanks,
Subhadip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6930af3b-13ad-476c-82cc-7c8d792dea67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
laltu
Software Engineer
Reply | Threaded
Open this post in threaded view
|

Re: elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

Ed Kim
Other members can correct me if I'm wrong, but I notice that when you lose connection with the server, the transportclient queues retries of whatever operations you try to execute, and it starts to queue listeners into a 'generic' threadpool (which I read somewhere that it was unbounded). We've seen this problem when we thrash ES until it eventually stops responding, and our bulk requests start to back up and eventually cause the application to halt due to OOM.

I don't know exactly what your application is doing when your ES node(s) go down, but perhaps you can proactively stop requests to ES servers once your application sees the no node exception error (which you should get when ES goes down). You could also close the transportclient and shutdown its threadpool and reconnect/instantiate after a timed delay to clean up whatever is floating around in the transportclient. We have been able to solve most of our native thread issues by protecting our use of transportclient and doing a soft restart of this client. 


On Saturday, January 10, 2015 at 9:29:56 AM UTC-8, Subhadip Bagui wrote:
Hi,

I'm using elasticsearch using TransportClient for multiple operation. The issue I'm facing now is if my es server goes down my client side app getting OutOfMemoryError.  Getting the below exception. I had to restart my tomcat every time after this to make my application up. Can some one please suggest how to prevent this. 


Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [spring] in context with path [/aricloud] threw exception [Handler processing failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread] with root cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)


Thanks,
Subhadip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

Jason Wee
Hi Ed,

soft restart as in client.close()  ? I did that as well, does not seem to work, hence I resolve to tomcat restart instead... 

Jason

On Sun, Jan 11, 2015 at 3:03 PM, Ed Kim <[hidden email]> wrote:
Other members can correct me if I'm wrong, but I notice that when you lose connection with the server, the transportclient queues retries of whatever operations you try to execute, and it starts to queue listeners into a 'generic' threadpool (which I read somewhere that it was unbounded). We've seen this problem when we thrash ES until it eventually stops responding, and our bulk requests start to back up and eventually cause the application to halt due to OOM.

I don't know exactly what your application is doing when your ES node(s) go down, but perhaps you can proactively stop requests to ES servers once your application sees the no node exception error (which you should get when ES goes down). You could also close the transportclient and shutdown its threadpool and reconnect/instantiate after a timed delay to clean up whatever is floating around in the transportclient. We have been able to solve most of our native thread issues by protecting our use of transportclient and doing a soft restart of this client. 


On Saturday, January 10, 2015 at 9:29:56 AM UTC-8, Subhadip Bagui wrote:
Hi,

I'm using elasticsearch using TransportClient for multiple operation. The issue I'm facing now is if my es server goes down my client side app getting OutOfMemoryError.  Getting the below exception. I had to restart my tomcat every time after this to make my application up. Can some one please suggest how to prevent this. 


Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [spring] in context with path [/aricloud] threw exception [Handler processing failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread] with root cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)


Thanks,
Subhadip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4ityTv%2B2-tQLrqWQetKfL0juxDVqhsJeNbYQkZbzr6kjtDg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

Ed Kim
Yes, but in addition I would also call client.threadPool().shutdown() to make sure you free up the threadpool.

On Sunday, January 11, 2015 at 7:22:20 PM UTC-8, Jason Wee wrote:
Hi Ed,

soft restart as in client.close()  ? I did that as well, does not seem to work, hence I resolve to tomcat restart instead... 

Jason

On Sun, Jan 11, 2015 at 3:03 PM, Ed Kim <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="BE_Aw2lH9JcJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">edk...@...> wrote:
Other members can correct me if I'm wrong, but I notice that when you lose connection with the server, the transportclient queues retries of whatever operations you try to execute, and it starts to queue listeners into a 'generic' threadpool (which I read somewhere that it was unbounded). We've seen this problem when we thrash ES until it eventually stops responding, and our bulk requests start to back up and eventually cause the application to halt due to OOM.

I don't know exactly what your application is doing when your ES node(s) go down, but perhaps you can proactively stop requests to ES servers once your application sees the no node exception error (which you should get when ES goes down). You could also close the transportclient and shutdown its threadpool and reconnect/instantiate after a timed delay to clean up whatever is floating around in the transportclient. We have been able to solve most of our native thread issues by protecting our use of transportclient and doing a soft restart of this client. 


On Saturday, January 10, 2015 at 9:29:56 AM UTC-8, Subhadip Bagui wrote:
Hi,

I'm using elasticsearch using TransportClient for multiple operation. The issue I'm facing now is if my es server goes down my client side app getting OutOfMemoryError.  Getting the below exception. I had to restart my tomcat every time after this to make my application up. Can some one please suggest how to prevent this. 


Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [spring] in context with path [/aricloud] threw exception [Handler processing failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread] with root cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)


Thanks,
Subhadip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="BE_Aw2lH9JcJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a124503b-ad60-42a0-bb17-de56c8c6a726%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

bagui
In reply to this post by bagui
Hi Ed,

I my case I have created a singleton TransportClient and querying ES with the same in frequent interval to get data. I'm not doing any bulk operations only searching index. threadPool needs to be shutdown once Tomcat stops I guess, But when my webapplication is up and running is there any need to shutdown the threadPool ? I'm getting this error recently. Previously if the ES server goes down I used to get NoNodeAvailableException and that's OK with me. But now the whole tomcat goes down and no app is working showing OutOfMemoryError. Please suggest.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00bad24d-b036-4613-b3f3-7a2802778d24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
laltu
Software Engineer
Reply | Threaded
Open this post in threaded view
|

Re: elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

Ed Kim
It's hard to say what's truly causing it, but when your server goes down, you should also see NoNodeAvailableException. Also, I don't think it matters too much whether you are doing index or search requests, as all requests queue handlers (Runnable) to the generic threadpool. What is your query per second like when searching? What timeout have you set? I'm wondering if TransportClient is flooding the threadpool with a whole bunch of requests and causing your server to crash as a result. In this case though, I might also expect to see timeout exceptions as well, but since you haven't mentioned them, I'm not really sure.

For the threadpool shutdown, I typically include threadpool.shutdown() whenever I call close() since my intention is to completely shutdown the transportclient. I wouldn't suggest simply killing the threadpool only.

Having said that, have you tried protecting TransportClient from handling/executing requests when NoNodeAvailableException occurs? We've set up a wrapper class around TransportClient to queue up requests until TransportClient recovers. We will periodically check to see if TransportClient can communicate with ES, and then execute the queued requests if things are back to normal. Worst case, we will drop the queued ES requests and return an error message back to the client.



On Monday, January 12, 2015 at 4:38:09 AM UTC-8, Subhadip Bagui wrote:
Hi Ed,

I my case I have created a singleton TransportClient and querying ES with the same in frequent interval to get data. I'm not doing any bulk operations only searching index. threadPool needs to be shutdown once Tomcat stops I guess, But when my webapplication is up and running is there any need to shutdown the threadPool ? I'm getting this error recently. Previously if the ES server goes down I used to get NoNodeAvailableException and that's OK with me. But now the whole tomcat goes down and no app is working showing OutOfMemoryError. Please suggest.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a01d7ebb-b86b-43f9-a93f-77c49b9be17f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.