Quantcast

Netty threads eats all of CPU

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Netty threads eats all of CPU

Mustafa Sener
Hi,
Several weeks ago I reported same issue.
http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html

We upgraded JRE to latest one which is 1.6.0_24-b07 as it is suggested by Shay. But the problem occurred again. The problem is on TransportClient side not on ES server. It seems that it is related to following threads which eats all CPU:

New I/O client boss #5
New I/O client boss #7

I took multiple stack traces at different times
=====================================================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
	at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
	at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
	- locked <0x00000007871eadf0> (a java.lang.Object)
	at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
	at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x000000078721e8d8> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
	- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
=====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x00000007871eadf0> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x000000078721e8d8> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

=======================================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap$KeySet.iterator(HashMap.java:874)
	at java.util.HashSet.iterator(HashSet.java:153)
	at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
	- locked <0x00000007871e5538> (a java.util.HashSet)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)


"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
	at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
	at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
	- locked <0x000000078721e8d8> (a java.lang.Object)
	at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
	at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
	- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
====================================================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x00000007871eadf0> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

--
Mustafa Sener
www.ifountain.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Netty threads eats all of CPU

kimchy
Administrator
Can you try and move to OIO in the transport client and see if it still happens?

On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote:

Hi,
Several weeks ago I reported same issue.
http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html

We upgraded JRE to latest one which is 1.6.0_24-b07 as it is suggested by Shay. But the problem occurred again. The problem is on TransportClient side not on ES server. It seems that it is related to following threads which eats all CPU:

New I/O client boss #5
New I/O client boss #7

I took multiple stack traces at different times
=====================================================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
	at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
	at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
	- locked <0x00000007871eadf0> (a java.lang.Object)
	at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
	at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x000000078721e8d8> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
	- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
=====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x00000007871eadf0> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x000000078721e8d8> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

=======================================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap$KeySet.iterator(HashMap.java:874)
	at java.util.HashSet.iterator(HashSet.java:153)
	at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
	- locked <0x00000007871e5538> (a java.util.HashSet)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)


"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
	at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
	at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
	- locked <0x000000078721e8d8> (a java.lang.Object)
	at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
	at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
	- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
====================================================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x00000007871eadf0> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

--
Mustafa Sener
www.ifountain.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Netty threads eats all of CPU

Mustafa Sener
what do you mean by O|O?

On Fri, May 6, 2011 at 6:54 PM, Shay Banon <[hidden email]> wrote:
Can you try and move to OIO in the transport client and see if it still happens?

On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote:

Hi,
Several weeks ago I reported same issue.
http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html

We upgraded JRE to latest one which is 1.6.0_24-b07 as it is suggested by Shay. But the problem occurred again. The problem is on TransportClient side not on ES server. It seems that it is related to following threads which eats all CPU:

New I/O client boss #5
New I/O client boss #7

I took multiple stack traces at different times
=====================================================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
	at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
	at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
	- locked <0x00000007871eadf0> (a java.lang.Object)
	at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
	at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x000000078721e8d8> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
	- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
=====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x00000007871eadf0> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x000000078721e8d8> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

=======================================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap$KeySet.iterator(HashMap.java:874)
	at java.util.HashSet.iterator(HashSet.java:153)
	at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
	- locked <0x00000007871e5538> (a java.util.HashSet)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)


"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
	at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
	at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
	- locked <0x000000078721e8d8> (a java.lang.Object)
	at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
	at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
	- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
====================================================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x00000007871eadf0> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

--
Mustafa Sener
www.ifountain.com




--
Mustafa Sener
www.ifountain.com
WebRep
Overall rating
 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Netty threads eats all of CPU

kimchy
Administrator
Sorry, thats the netty lingo. Set network.tcp.blocking setting to true.

On Friday, May 6, 2011 at 7:01 PM, Mustafa Sener wrote:

what do you mean by O|O?

On Fri, May 6, 2011 at 6:54 PM, Shay Banon <[hidden email]> wrote:
Can you try and move to OIO in the transport client and see if it still happens?

On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote:

Hi,
Several weeks ago I reported same issue.
http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html

We upgraded JRE to latest one which is 1.6.0_24-b07 as it is suggested by Shay. But the problem occurred again. The problem is on TransportClient side not on ES server. It seems that it is related to following threads which eats all CPU:

New I/O client boss #5
New I/O client boss #7

I took multiple stack traces at different times
=====================================================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
	at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
	at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
	- locked <0x00000007871eadf0> (a java.lang.Object)
	at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
	at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x000000078721e8d8> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
	- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
=====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x00000007871eadf0> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x000000078721e8d8> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

=======================================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap$KeySet.iterator(HashMap.java:874)
	at java.util.HashSet.iterator(HashSet.java:153)
	at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
	- locked <0x00000007871e5538> (a java.util.HashSet)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)


"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
	at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
	at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
	- locked <0x000000078721e8d8> (a java.lang.Object)
	at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
	at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
	- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
====================================================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x00000007871eadf0> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

--
Mustafa Sener
www.ifountain.com




--
Mustafa Sener
www.ifountain.com
WebRep
Overall rating
 


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Netty threads eats all of CPU

Mustafa Sener
I applied setting to TransportClient. I will notify you if this problem occurs again.

Thanks.

On Fri, May 6, 2011 at 7:06 PM, Shay Banon <[hidden email]> wrote:
Sorry, thats the netty lingo. Set network.tcp.blocking setting to true.

On Friday, May 6, 2011 at 7:01 PM, Mustafa Sener wrote:

what do you mean by O|O?

On Fri, May 6, 2011 at 6:54 PM, Shay Banon <[hidden email]> wrote:
Can you try and move to OIO in the transport client and see if it still happens?

On Friday, May 6, 2011 at 12:39 PM, Mustafa Sener wrote:

Hi,
Several weeks ago I reported same issue.
http://elasticsearch-users.115913.n3.nabble.com/Netty-threads-eats-all-of-CPU-td2791974.html

We upgraded JRE to latest one which is 1.6.0_24-b07 as it is suggested by Shay. But the problem occurred again. The problem is on TransportClient side not on ES server. It seems that it is related to following threads which eats all CPU:

New I/O client boss #5
New I/O client boss #7

I took multiple stack traces at different times
=====================================================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
	at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
	at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
	- locked <0x00000007871eadf0> (a java.lang.Object)
	at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
	at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x000000078721e8d8> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
	- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
=====================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x00000007871eadf0> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007871eac48> (a sun.nio.ch.Util$2) - locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76) - locked <0x000000078721e8d8> (a java.lang.Object) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078721e730> (a sun.nio.ch.Util$2) - locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

=======================================================================================

"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap$KeySet.iterator(HashMap.java:874)
	at java.util.HashSet.iterator(HashSet.java:153)
	at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
	- locked <0x00000007871e5538> (a java.util.HashSet)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)


"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
	at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
	at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
	- locked <0x000000078721e8d8> (a java.lang.Object)
	at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:193)
	at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:198)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:64)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000078721e730> (a sun.nio.ch.Util$2)
	- locked <0x000000078721e720> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000078721df90> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
====================================================================
"New I/O client boss #5" daemon prio=10 tid=0x0000000053e12800 nid=0x7b03 runnable [0x00002aaaf8fca000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x00000007871eadf0> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000007871eac48> (a sun.nio.ch.Util$2)
	- locked <0x00000007871eac38> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007871e54e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:239)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007871eae58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"New I/O client boss #7" daemon prio=10 tid=0x0000000053281800 nid=0x7afc runnable [0x00002aaaf8dc8000]
   java.lang.Thread.State: RUNNABLE
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:328)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x000000078721dc08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

--
Mustafa Sener
www.ifountain.com




--
Mustafa Sener
www.ifountain.com
WebRep
Overall rating
 





--
Mustafa Sener
www.ifountain.com

 

Loading...