concurrent search request to elasticsearch

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

concurrent search request to elasticsearch

vipins
What is the maximum limit on the concurrent search requests with default Elastic search server settings.

I am able to perform only 5 parallel search requests in my application with default settings.

how can we improve the scalability of ES server search requests apart from increasing number of node,shards and queue size in search thread pool.

thanks.
Reply | Threaded
Open this post in threaded view
|

Re: concurrent search request to elasticsearch

Radu Gheorghe-2
Hello,

The search threadpool size (that is, how many requests can be actually worked on at once) defaults to 3 times the number of processors. This might be reduced in future, though, see: https://github.com/elasticsearch/elasticsearch/pull/9165

The queue size (how many requests ES can accept before starting to reject them) defaults to 1000.

From what I understand, this is per thread, so the answer to your question depends on how many processors you have and how many shards get hit by each search. For example, if a search runs on 3 indices, each with 2 shards (number of replicas is irrelevant, because the search will only hit one complete set of data) you'll get 6 requests in the threadpool per search. If you have two servers with 8 cores each, you 8*3*2=48 threads available. So the cluster can work on 8 requests at once. On top of that it can still queue round-down(2 nodes * 1000 queue size/6 requests per search)=333 searches until it starts rejecting them.

Regarding your scaling question, I can't give you a direct answer, unfortunately, because it depends on a whole lot of variables, mainly how your data, queries and hardware look like and what can be changed. The fact that your threadpool queue got full is just a symptom, it's not clear to me what happens in there. I usually see this when there are lots of indices and/or those indices have lots of shards. So a single request takes a lot of requests in the threadpool, filling it up, even if the ES cluster can keep up with the load. If that's your case increase the threadpool queue size and make sure you don't have too many shards per index.

If your cluster can't keep up with the load (a monitoring tool like SPM should show you that), then the first step is to see where is the bottleneck. Again, monitoring can give some insight: are queries too expensive, can they be optimized? do you have too many cache evictions? is the heap size too large or too small? is memory, I/O or CPU the bottleneck? Things like that. It could also be that you need more/different hardware.

Finally, you can make scaling ES someone else's problem by using a hosted service like Logsene. Especially if you're using ES for log- or metric-like data, you'll get lots of features out of the box, and we expose most of the ES API to plug in your custom stuff.

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thu, Jan 8, 2015 at 1:32 PM, vipins <[hidden email]> wrote:
What is the maximum limit on the concurrent search requests with default
Elastic search server settings.

I am able to perform only 5 parallel search requests in my application with
default settings.

how can we improve the scalability of ES server search requests apart from
increasing number of node,shards and queue size in search thread pool.

thanks.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1420716748150-4068702.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_2ZoQ6bewJ_aSxFMxAO0_5Mdsu%3D9WA4m_be7AfSb%3D2%2BTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: concurrent search request to elasticsearch

vipins
Thanks a lot for your detailed response.
We have got all default settings only.Single node and 5 shards. But there are lot of indices with huge number of records.
search settings:
          "threads" : 12,
          "queuesize" : 1000,
         
My query is very simple. which runs on a single index only.

Even with 5 requests in between it is throwing None of the configured nodes are available.

Thanks,
Reply | Threaded
Open this post in threaded view
|

Re: concurrent search request to elasticsearch

Radu Gheorghe-2
You're welcome.

So you're saying you're running 5 searches on a single index with 5 shards (25 per-shard queries in total) and you're getting an error? I assume that error doesn't say the queue is full because the queue is 1000. Can you post the full error and also a gist where you reproduce the issue? I might be missing an essential bit here.

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thu, Jan 8, 2015 at 3:14 PM, vipins <[hidden email]> wrote:
Thanks a lot for your detailed response.
We have got all default settings only.Single node and 5 shards. But there
are lot of indices with huge number of records.
search settings:
          "threads" : 12,
          "queuesize" : 1000,

My query is very simple. which runs on a single index only.

Even with 5 requests in between it is throwing None of the configured nodes
are available.

Thanks,



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702p4068707.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1420722888084-4068707.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_0DMNN91Xo_iQpFFdep%3DosynTs5tGr-UDX_EOCOX%3DTg5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: concurrent search request to elasticsearch

vipins
Sorry , I was wrong with number of shards. actual number of shards is 320 for the index which i am querying.

We are using rolling indices on a daily basis.

max queue size is 1000 for search thread pool.

We overcome the issue None of the configured nodes are available by keeping tcp connection alive as true.
Reply | Threaded
Open this post in threaded view
|

Re: concurrent search request to elasticsearch

Radu Gheorghe-2
OK, now it makes sense. 5 requests with 320 shards might saturate your queue.

But 320 shards sounds like a lot for one index. I assume you don't need to scale that very index to 320 nodes (+ replicas). If you can get the number of shards down (say, to the default of 5) things will surely look better not only from the queue's perspective, but it should also improve search performance.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thu, Jan 8, 2015 at 3:46 PM, vipins <[hidden email]> wrote:
Sorry , I was wrong with number of shards. actual number of shards is 320 for
the index which i am querying.

We are using rolling indices on a daily basis.

max queue size is 1000 for search thread pool.

We overcome the issue None of the configured nodes are available by keeping
tcp connection alive as true.




--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702p4068711.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1420724779413-4068711.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_00wG%2BNUQQm2_KtH7jKBC7ovN1AXnAf9Jot2VCTppMk9g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: concurrent search request to elasticsearch

vipins
Thanks for your prompt response.

Surely will reduce the number of shards with nodes/replicas addition for the better performance of the search.