ES hangs up when sending too much index requests

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

ES hangs up when sending too much index requests

Ihor Hnedysh
Good day,

we have faced with the ES handing by some reason, could you please take a look in it:


Elastic search configuration if the following:


Can you comment this somehow? Do you need any additional data?

PS:
1. Problem reproduces sporadically with different requests
2. Log shows nothing
Reply | Threaded
Open this post in threaded view
|

Re: ES hangs up when sending too much index requests

sborar
This post has NOT been accepted by the mailing list yet.
How many search requests are you sending ?
Reply | Threaded
Open this post in threaded view
|

Re: ES hangs up when sending too much index requests

Ihor Hnedysh
differently. This time only 25 requests were sent (portionally. by 5 requests per time). All requests have similar structure. userId criteria only varies.
Reply | Threaded
Open this post in threaded view
|

Re: ES hangs up when sending too much index requests

kimchy
Administrator
- Are the search request concurrent? If so, how many concurrent search requests are you sending? 
- Are you opening a new socket on each request? The OS can stall opening new sockets if you do that excessively over time.
- How the machine stats on the two nodes? CPU, load?
- How much memory do you ahve allocated on the machine, how many CPUs/cores? How much memory have you allocated to ES?

On Mon, Mar 26, 2012 at 9:15 PM, Ihor Hnedysh <[hidden email]> wrote:
differently. This time only 25 requests were sent (portionally. by 5 requests
per time). All requests have similar structure. userId criteria only varies.

--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ES-hangs-up-when-sending-too-much-index-requests-tp3857628p3859097.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: ES hangs up when sending too much index requests

Ihor Hnedysh
>>Are the search request concurrent? If so, how many concurrent search requests are you sending?
Yes. Requests are sent in parallel. Portionally. By five requests per portion. Total 25 requests.

>>Are you opening a new socket on each request? The OS can stall opening new sockets if you do that excessively over time.
I'm not sure about your question since we're not using sockets directly. I will recheck it. But as far as I understand problem is that response did not came from elastic search in the same time while it received it. I believe in it because request that was previously hanging is aborted right after elastic search instance stopped. Also, frequency of problem reproducibility has been decreased after we have added additional data node.

>>How the machine stats on the two nodes? CPU, load?
No CPU load. ProcessExplorer shows some cycles performed by elastic search.

>>How much memory do you ahve allocated on the machine, how many CPUs/cores? How much memory Have you allocated to ES?
I have standard JVM settings set. I did not tune them. PC is the following:
RAM: 6GB
CPU: 4 cores 3.2 GHz
I have win7 on board but we have current problem reproduced not only on windows machine but on Ubuntu as well.
Reply | Threaded
Open this post in threaded view
|

Re: ES hangs up when sending too much index requests

Ihor Hnedysh
up
Reply | Threaded
Open this post in threaded view
|

Re: ES hangs up when sending too much index requests

kimchy
Administrator
In reply to this post by Ihor Hnedysh
Can you try and explain the problem again? How you run your tests, how many concurrent clients, what are you running.

On Wed, Mar 28, 2012 at 11:18 AM, Ihor Hnedysh <[hidden email]> wrote:
>>Are the search request concurrent? If so, how many concurrent search
requests are you sending?
Yes. Requests are sent in parallel. Portionally. By five requests per
portion. Total 25 requests.

>>Are you opening a new socket on each request? The OS can stall opening new
sockets if you do that excessively over time.
I'm not sure about your question since we're not using sockets directly. I
will recheck it. But as far as I understand problem is that response did not
came from elastic search in the same time while it received it. I believe in
it because request that was previously hanging is aborted right after
elastic search instance stopped. Also, frequency of problem reproducibility
has been decreased after we have added additional data node.

>>How the machine stats on the two nodes? CPU, load?
No CPU load. ProcessExplorer shows some cycles performed by elastic search.

>>How much memory do you ahve allocated on the machine, how many CPUs/cores?
How much memory Have you allocated to ES?
I have standard JVM settings set. I did not tune them. PC is the following:
RAM: 6GB
CPU: 4 cores 3.2 GHz
I have win7 on board but we have current problem reproduced not only on
windows machine but on Ubuntu as well.

--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ES-hangs-up-when-sending-too-much-index-requests-tp3857628p3863994.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: ES hangs up when sending too much index requests

Alexander Gamanyuk
This post has NOT been accepted by the mailing list yet.
Ok, I will try to explain it in details.

We have about 120 tests. All the tests have some business logic and send different requests to ES. About 900 requests for all the tests.
They are run by executing a batch file. We start the batch file with tests simultaneously 2 or more times. In average, every batch file sends about 4 requests per second. Since we started 2 ones we have about 7-8 requests per second.
As a result we have about 3-7 hanged requests.
I tested it in different configurations of ES, detailed raw result are here

Note, using tests is just an easy way of reproducing of the problem. The problem exist in a real application as well.
Hope it helps.
Reply | Threaded
Open this post in threaded view
|

Re: ES hangs up when sending too much index requests

Alexander Gamanyuk
This post has NOT been accepted by the mailing list yet.
We're using version 0.18.4, but the problem is reproducing on 0.19.2 as well.