Could someone perhaps help me troubleshoot this problem... Over the weekend I ran a test on my 5-node cluster, which started failing after a while. After looking at the logs (which came up to be around 9 GB of data...) I found this error being thrown repeatedly:
[2012-03-26 14:00:15,270][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink] Failed to accept a connection.
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
It seems as if there's some problem with "Too many open files", but this cannot be the case, or at least not the way I understand it. I set the user running the ElasticSearch instance's file limit to 32000 as per the suggestion on the website, but even if I check with "lsof | wc -l" there is in total only about 6000 descriptors open, so I doubt it's really the problem. What bugs me, but I really do not know much about these things, is that it seems to be a "NioServerSocketPipelineSink" connection, for which I'm not entirely sure how it relates to disc-IO.
Another seemingly related issue is the following error which is also thrown into the mix:
org.elasticsearch.index.engine.IndexFailedEngineException: [dev-index] Index failed for [info#< omitted >]
Caused by: java.io.FileNotFoundException: /data2/esc/cluster0/nodes/0/indices/dev-index/1/index/_j3.fdx (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
... 7 more
Again there's that "Too many open files" message. Any idea as to what might be causing this problem? Am I using the Java API wrong? Thanks for the help - I appreciate it.
Am Montag, 26. März 2012 14:11:08 UTC+2 schrieb Thinus Prinsloo:
You need to raise the "ulimit" for the user that is running ES. Depends on your Linux distribution where you have to do it. On RedHat you can add a file to /etc/security/limits.d.
Ours looks like this:
elastic - memlock unlimited
elastic soft nofile 80000
elastic hard nofile 100000
Our user is called "elastic". The first line allows the ES JVM to lock as much memory as it wants to (memory mapped files count as memory as well!). When the user reaches the soft limit a warning is written to the log.
Also, check nodes info and nodes stats API, they provide information on the max open files desc limit, and hte current open files. Verify through the nodes info API that your setting has actually taken affect.
On Mon, Mar 26, 2012 at 4:10 PM, Thomas Peuss <[hidden email]> wrote:
Thanks - that's basically what I did. I realised that the configuration I used to set the limits did not work, so I added it to the ElasticSearch launch script. I realised that when I was trying to verify through the API as you suggested. What made it very difficult then was that even though it reported the right limit, I still got the error and when viewing the associated file descriptors for that thread the number was not near the limit. All I could gather here was that because of the previous failure of running out of file-id's, the index was now severely corrupted (not sure if it can happen like that). So this time, though it showed the same error, it could actually not find the file. So I started deleting some shards and had ES recover until it sorted my problem. It was a bit messy though...
Bottom-line, make sure that setting is working properly by using the API, and don't ever allow it to try and index millions of files when that error occurs. Rather stop... :)
On Tue, Mar 27, 2012 at 2:56 PM, Shay Banon <[hidden email]> wrote:
E-mail: [hidden email]
Cell: +27 82 339 2226
In reply to this post by kimchy
in 0.19.9, the node stats don't seem to mention max open files desc limit.
Do you have documentation somewhere which request I have to make (with curl for example)?
On Tuesday, March 27, 2012 2:56:59 PM UTC+2, kimchy wrote:
Maybe you could also update the docs:
to point at some of the tutorials that mention the service wrapper. I almost tried to configure the servicewrapper with elasticsearch myself, but luckyly found your github project (https://github.com/elasticsearch/elasticsearch-servicewrapper).
On Monday, October 15, 2012 6:34:47 PM UTC+2, Jaap Taal wrote:
|Free forum by Nabble||Edit this page|