0.19.11 StackOverflowError

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

0.19.11 StackOverflowError

davrob
Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try to start ES, also I get a SnappyError, which may be related.  

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):


- David.

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: 0.19.11 StackOverflowError

Drew Raines-2
davrob2 wrote:

> After my move to 0.19.11 - I am getting a StackOverflowError when I try to
> start ES, also I get a SnappyError, which may be related.  
>
> The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):
>
> 0.19.11 StackOverflowError - Debug Log <https://gist.github.com/4060559>

What version of ES were you using before?  Seems like what was on
disk is not compatible with the snappy support in 0.19.11, but that's
a guess.  Any changes besides ES version?

-Drew

--


Reply | Threaded
Open this post in threaded view
|

Re: 0.19.11 StackOverflowError

joergprante@gmail.com
In reply to this post by davrob
The Snappy message is just a warning and is not related. Your stack trace shows you have JVM stack settings that are too small for your JVM to run ES. What Java are you running? Are you running ES inside a Solaris zone?

Jörg

On Monday, November 12, 2012 6:20:03 PM UTC+1, davrob2 wrote:
Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try to start ES, also I get a SnappyError, which may be related.  

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):


- David.

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: 0.19.11 StackOverflowError

davrob
Hi,

I changed -d64 JVM flag to -server, on Solaris - just see see what happened, and the error changed to this:

[2012-11-13 09:51:09,877][WARN ][transport.netty          ] [Boomerang] Message not fully read (request) for [36986] and action [], resetting
[2012-11-13 09:51:09,894][WARN ][netty.channel.socket.nio.AbstractNioWorker] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)

which is definitely memory related.  I'm using the same settings I was running on 0.17.6, on the same machine:

JDK: /cs/java/jdk160_16-64b
Memory Settings: ES_MIN_MEM=3600m, ES_MAX_MEM=3600m

- David.

On Monday, 12 November 2012 20:45:49 UTC, Jörg Prante wrote:
The Snappy message is just a warning and is not related. Your stack trace shows you have JVM stack settings that are too small for your JVM to run ES. What Java are you running? Are you running ES inside a Solaris zone?

Jörg

On Monday, November 12, 2012 6:20:03 PM UTC+1, davrob2 wrote:
Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try to start ES, also I get a SnappyError, which may be related.  

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):


- David.

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: 0.19.11 StackOverflowError

davrob
Just for the record, these are basic Java Settings:

Java home is /cs/java/jdk160_16-64b

 Java bin is /cs/java/jdk160_16-64b/bin/java

JAVA_OPTS is  -server -Xms3600m -Xmx3600m -Xss128k -Djline.enabled=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly

CLASSPATH is :/app/securities/rapport/deploy/dev2/ESServer/lib/elasticsearch-0.19.11.jar:/app/securities/rapport/deploy/dev2/ESServer/lib/*:/app/securities/rapport/deploy/dev2/ESServer/lib/sigar/*


On Tuesday, 13 November 2012 14:57:11 UTC, davrob2 wrote:
Hi,

I changed -d64 JVM flag to -server, on Solaris - just see see what happened, and the error changed to this:

[2012-11-13 09:51:09,877][WARN ][transport.netty          ] [Boomerang] Message not fully read (request) for [36986] and action [], resetting
[2012-11-13 09:51:09,894][WARN ][netty.channel.socket.nio.AbstractNioWorker] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)

which is definitely memory related.  I'm using the same settings I was running on 0.17.6, on the same machine:

JDK: /cs/java/jdk160_16-64b
Memory Settings: ES_MIN_MEM=3600m, ES_MAX_MEM=3600m

- David.

On Monday, 12 November 2012 20:45:49 UTC, Jörg Prante wrote:
The Snappy message is just a warning and is not related. Your stack trace shows you have JVM stack settings that are too small for your JVM to run ES. What Java are you running? Are you running ES inside a Solaris zone?

Jörg

On Monday, November 12, 2012 6:20:03 PM UTC+1, davrob2 wrote:
Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try to start ES, also I get a SnappyError, which may be related.  

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):


- David.

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: 0.19.11 StackOverflowError

joergprante@gmail.com
You should check your custom Xss setting, 128k is too small for 64bit stacks on Solaris, 512k is the default.

If this is Java JDK 6u16, this version is very old, released back in October, 2009. It is not expected that you can use applications like ES flawlessly without unexpected behavior on such old JVMs. And, many bugs and security issues have been fixed within the last three years. 

Please update to the latest JDK6 (which is 6u37) or JDK7 (the current JDK, which is 7u9).

Jörg

On Tuesday, November 13, 2012 4:10:16 PM UTC+1, davrob2 wrote:
Just for the record, these are basic Java Settings:

Java home is /cs/java/jdk160_16-64b

 Java bin is /cs/java/jdk160_16-64b/bin/java

JAVA_OPTS is  -server -Xms3600m -Xmx3600m -Xss128k -Djline.enabled=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly

CLASSPATH is :/app/securities/rapport/deploy/dev2/ESServer/lib/elasticsearch-0.19.11.jar:/app/securities/rapport/deploy/dev2/ESServer/lib/*:/app/securities/rapport/deploy/dev2/ESServer/lib/sigar/*


On Tuesday, 13 November 2012 14:57:11 UTC, davrob2 wrote:
Hi,

I changed -d64 JVM flag to -server, on Solaris - just see see what happened, and the error changed to this:

[2012-11-13 09:51:09,877][WARN ][transport.netty          ] [Boomerang] Message not fully read (request) for [36986] and action [], resetting
[2012-11-13 09:51:09,894][WARN ][netty.channel.socket.nio.AbstractNioWorker] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)

which is definitely memory related.  I'm using the same settings I was running on 0.17.6, on the same machine:

JDK: /cs/java/jdk160_16-64b
Memory Settings: ES_MIN_MEM=3600m, ES_MAX_MEM=3600m

- David.

On Monday, 12 November 2012 20:45:49 UTC, Jörg Prante wrote:
The Snappy message is just a warning and is not related. Your stack trace shows you have JVM stack settings that are too small for your JVM to run ES. What Java are you running? Are you running ES inside a Solaris zone?

Jörg

On Monday, November 12, 2012 6:20:03 PM UTC+1, davrob2 wrote:
Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try to start ES, also I get a SnappyError, which may be related.  

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):


- David.

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: 0.19.11 StackOverflowError

davrob
Thanks for the advice on the xss settings - I'll try an xss of 512k, though I did up it to the standard in the 0.20 release without any luck:

# reduce the per-thread stack size
JAVA_OPTS="$JAVA_OPTS -Xss256k"

Unfortunately, I'm probably stuck with my Java version, unless I try to get an exception from my company, which is probably more trouble than its worth.

Luckily enough - I'll be moving my remaining Solaris environments to Linux in the next couple of months, so I may stay with the old version of ES until then.


On Tuesday, 13 November 2012 19:14:03 UTC, Jörg Prante wrote:
You should check your custom Xss setting, 128k is too small for 64bit stacks on Solaris, 512k is the default.

If this is Java JDK 6u16, this version is very old, released back in October, 2009. It is not expected that you can use applications like ES flawlessly without unexpected behavior on such old JVMs. And, many bugs and security issues have been fixed within the last three years. 

Please update to the latest JDK6 (which is 6u37) or JDK7 (the current JDK, which is 7u9).

Jörg

On Tuesday, November 13, 2012 4:10:16 PM UTC+1, davrob2 wrote:
Just for the record, these are basic Java Settings:

Java home is /cs/java/jdk160_16-64b

 Java bin is /cs/java/jdk160_16-64b/bin/java

JAVA_OPTS is  -server -Xms3600m -Xmx3600m -Xss128k -Djline.enabled=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly

CLASSPATH is :/app/securities/rapport/deploy/dev2/ESServer/lib/elasticsearch-0.19.11.jar:/app/securities/rapport/deploy/dev2/ESServer/lib/*:/app/securities/rapport/deploy/dev2/ESServer/lib/sigar/*


On Tuesday, 13 November 2012 14:57:11 UTC, davrob2 wrote:
Hi,

I changed -d64 JVM flag to -server, on Solaris - just see see what happened, and the error changed to this:

[2012-11-13 09:51:09,877][WARN ][transport.netty          ] [Boomerang] Message not fully read (request) for [36986] and action [], resetting
[2012-11-13 09:51:09,894][WARN ][netty.channel.socket.nio.AbstractNioWorker] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)

which is definitely memory related.  I'm using the same settings I was running on 0.17.6, on the same machine:

JDK: /cs/java/jdk160_16-64b
Memory Settings: ES_MIN_MEM=3600m, ES_MAX_MEM=3600m

- David.

On Monday, 12 November 2012 20:45:49 UTC, Jörg Prante wrote:
The Snappy message is just a warning and is not related. Your stack trace shows you have JVM stack settings that are too small for your JVM to run ES. What Java are you running? Are you running ES inside a Solaris zone?

Jörg

On Monday, November 12, 2012 6:20:03 PM UTC+1, davrob2 wrote:
Hi,

After my move to 0.19.11 - I am getting a StackOverflowError when I try to start ES, also I get a SnappyError, which may be related.  

The problem happens on Solaris 10 but not my Suse Linux build (2.6.16):


- David.

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: 0.19.11 StackOverflowError

joergprante@gmail.com
FYI -Xss values are only estimates, they differ between operating systems and machine architectures, so vendors adjust the JVMs for different default values in different environments. The ES preconfigured value is not guaranteed to be safe. To be honest I don't know why there is a preconfigured Xss value at all, maybe there was once assumed that thread space consumption could be lowered because ES uses a lot of threads.

Quoting: 

"You may be running into a problem with the default stack size for threads. In Java SE 6, the default on Sparc is 512k in the 32-bit VM, and 1024k in the 64-bit VM. On x86 Solaris/Linux it is 320k in the 32-bit VM and 1024k in the 64-bit VM.

On Windows, the default thread stack size is read from the binary (java.exe). As of Java SE 6, this value is 320k in the 32-bit VM and 1024k in the 64-bit VM.

You can reduce your stack size by running with the -Xss option. For example:

java -server -Xss64k

Note that on some versions of Windows, the OS may round up thread stack sizes using very coarse granularity. If the requested size is less than the default size by 1K or more, the stack size is rounded up to the default; otherwise, the stack size is rounded up to a multiple of 1 MB.

64k is the least amount of stack space allowed per thread."

http://www.oracle.com/technetwork/java/hotspotfaq-138619.html

Jörg

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: 0.19.11 StackOverflowError

davrob
Hi Jorg,

Thanks for your detailed explanation.

-David.

On Wednesday, 14 November 2012 00:18:52 UTC, Jörg Prante wrote:
FYI -Xss values are only estimates, they differ between operating systems and machine architectures, so vendors adjust the JVMs for different default values in different environments. The ES preconfigured value is not guaranteed to be safe. To be honest I don't know why there is a preconfigured Xss value at all, maybe there was once assumed that thread space consumption could be lowered because ES uses a lot of threads.

Quoting: 

"You may be running into a problem with the default stack size for threads. In Java SE 6, the default on Sparc is 512k in the 32-bit VM, and 1024k in the 64-bit VM. On x86 Solaris/Linux it is 320k in the 32-bit VM and 1024k in the 64-bit VM.

On Windows, the default thread stack size is read from the binary (java.exe). As of Java SE 6, this value is 320k in the 32-bit VM and 1024k in the 64-bit VM.

You can reduce your stack size by running with the -Xss option. For example:

java -server -Xss64k

Note that on some versions of Windows, the OS may round up thread stack sizes using very coarse granularity. If the requested size is less than the default size by 1K or more, the stack size is rounded up to the default; otherwise, the stack size is rounded up to a multiple of 1 MB.

64k is the least amount of stack space allowed per thread."


Jörg

--