intermittently search slow response

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

intermittently search slow response

Jason Wee
Hi, we are experience a strange issue, it happen intermittently. we have three elasticsearch nodes running version 0.90.7 and started a new cluster. Then as data coming in, we query one of the nodes, sometime the result return immediately. Sometime it just hang there forever until ctrl+z is issued. Not exactly sure what is wrong. Can anyone pinpoint where might be the possible cause?

[user@node1 ~]$ time curl -XGET 'http://localhost:9200/index_test/_search?q=key:keywords&pretty=true'
{
  "took" : 69,
  "timed_out" : false,
  "_shards" : {
    "total" : 4500,
    "successful" : 4500,
    "failed" : 0
  },
  "hits" : {
    "total" : 2201,
    "max_score" : 6.192957,
    "hits" : [ {
      "_index" : "index_test",
...
...
...      
real 0m0.072s
user 0m0.001s
sys 0m0.003s


[user@node1 ~]$ time curl -XGET 'http://localhost:9200/index_test/_search?q=key:keywords&pretty=true'

^C

real 16m34.575s
user 0m0.006s
sys 0m0.012s

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/868a0a1f-ee19-4a2d-807c-1379f023e9f2%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: intermittently search slow response

simonw-2
you are searching on 4500 shards? Is that right? I am just asking there could be a lot of stuff going on though.

simon
On Friday, November 29, 2013 11:51:35 AM UTC+1, Jason Wee wrote:
Hi, we are experience a strange issue, it happen intermittently. we have three elasticsearch nodes running version 0.90.7 and started a new cluster. Then as data coming in, we query one of the nodes, sometime the result return immediately. Sometime it just hang there forever until ctrl+z is issued. Not exactly sure what is wrong. Can anyone pinpoint where might be the possible cause?

[user@node1 ~]$ time curl -XGET '<a href="http://localhost:9200/index_test/_search?q=key:keywords&amp;pretty=true" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2Findex_test%2F_search%3Fq%3Dkey%3Akeywords%26pretty%3Dtrue\46sa\75D\46sntz\0751\46usg\75AFQjCNEb8R-Rmz_xtkw1hObSmyterrxi7w';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2Findex_test%2F_search%3Fq%3Dkey%3Akeywords%26pretty%3Dtrue\46sa\75D\46sntz\0751\46usg\75AFQjCNEb8R-Rmz_xtkw1hObSmyterrxi7w';return true;">http://localhost:9200/index_test/_search?q=key:keywords&pretty=true'
{
  "took" : 69,
  "timed_out" : false,
  "_shards" : {
    "total" : 4500,
    "successful" : 4500,
    "failed" : 0
  },
  "hits" : {
    "total" : 2201,
    "max_score" : 6.192957,
    "hits" : [ {
      "_index" : "index_test",
...
...
...      
real 0m0.072s
user 0m0.001s
sys 0m0.003s


[user@node1 ~]$ time curl -XGET '<a href="http://localhost:9200/index_test/_search?q=key:keywords&amp;pretty=true" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2Findex_test%2F_search%3Fq%3Dkey%3Akeywords%26pretty%3Dtrue\46sa\75D\46sntz\0751\46usg\75AFQjCNEb8R-Rmz_xtkw1hObSmyterrxi7w';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2Findex_test%2F_search%3Fq%3Dkey%3Akeywords%26pretty%3Dtrue\46sa\75D\46sntz\0751\46usg\75AFQjCNEb8R-Rmz_xtkw1hObSmyterrxi7w';return true;">http://localhost:9200/index_test/_search?q=key:keywords&pretty=true'

^C

real 16m34.575s
user 0m0.006s
sys 0m0.012s

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f7a0c71d-d54e-4be8-a487-47371ca61456%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: intermittently search slow response

Mark Walkom
4500 shards isn't too bad, we've got 5500 and queries are quick.

You haven't given us much of a clue on what size cluster you have, including node size, RAM usage, replica count etc, which will impact query performance. Try installing the elastichq plugin, it has some nice functionality under Node Diagnostics that lets you see where things might be getting out of hand.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email]
web: www.campaignmonitor.com


On 30 November 2013 08:38, simonw <[hidden email]> wrote:
you are searching on 4500 shards? Is that right? I am just asking there could be a lot of stuff going on though.

simon
On Friday, November 29, 2013 11:51:35 AM UTC+1, Jason Wee wrote:
Hi, we are experience a strange issue, it happen intermittently. we have three elasticsearch nodes running version 0.90.7 and started a new cluster. Then as data coming in, we query one of the nodes, sometime the result return immediately. Sometime it just hang there forever until ctrl+z is issued. Not exactly sure what is wrong. Can anyone pinpoint where might be the possible cause?

{
  "took" : 69,
  "timed_out" : false,
  "_shards" : {
    "total" : 4500,
    "successful" : 4500,
    "failed" : 0
  },
  "hits" : {
    "total" : 2201,
    "max_score" : 6.192957,
    "hits" : [ {
      "_index" : "index_test",
...
...
...      
real 0m0.072s
user 0m0.001s
sys 0m0.003s



^C

real 16m34.575s
user 0m0.006s
sys 0m0.012s

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f7a0c71d-d54e-4be8-a487-47371ca61456%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624agH6H2q-8YuknGzZH_7wWhVB1p9u8kscRRUZO5EOm%2Bzg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: intermittently search slow response

Jason Wee
Yes, the index is started with a total of 4500 shards. We are currently have three dedicated server running for elastic search cluster. The three nodes has identical hardware configuration. Each node is running intel i7-4770 cpu with 32GB of ram. Each node is allocated 16GB heap out of 32GB of total memory and it only run elasticsearch and no other jvm or intensive cpu processes. Our es cluster currently having 2 replicas and total active shards currently is 15000.

indices store is around 11GB at this moment. Heap usage hovering in between 6G and 13G. Non heap usage is low at around 60MB. Jvm threads count stay constant around 120. 

when curl on http://localhost:9200/_stats , the path to index current, indices/index_test/total/indexing/index_current hovering around 240 constant throughout.

Will these information give you any clue what might happen to the intermittently search slow response?

=> Try installing the elastichq plugin, it has some nice functionality under Node Diagnostics that lets you see where things might be getting out of hand.

Thank you, will install this.

/Jason


On Sat, Nov 30, 2013 at 5:52 AM, Mark Walkom <[hidden email]> wrote:
4500 shards isn't too bad, we've got 5500 and queries are quick.

You haven't given us much of a clue on what size cluster you have, including node size, RAM usage, replica count etc, which will impact query performance. Try installing the elastichq plugin, it has some nice functionality under Node Diagnostics that lets you see where things might be getting out of hand.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email]
web: www.campaignmonitor.com


On 30 November 2013 08:38, simonw <[hidden email]> wrote:
you are searching on 4500 shards? Is that right? I am just asking there could be a lot of stuff going on though.

simon
On Friday, November 29, 2013 11:51:35 AM UTC+1, Jason Wee wrote:
Hi, we are experience a strange issue, it happen intermittently. we have three elasticsearch nodes running version 0.90.7 and started a new cluster. Then as data coming in, we query one of the nodes, sometime the result return immediately. Sometime it just hang there forever until ctrl+z is issued. Not exactly sure what is wrong. Can anyone pinpoint where might be the possible cause?

{
  "took" : 69,
  "timed_out" : false,
  "_shards" : {
    "total" : 4500,
    "successful" : 4500,
    "failed" : 0
  },
  "hits" : {
    "total" : 2201,
    "max_score" : 6.192957,
    "hits" : [ {
      "_index" : "index_test",
...
...
...      
real 0m0.072s
user 0m0.001s
sys 0m0.003s



^C

real 16m34.575s
user 0m0.006s
sys 0m0.012s

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f7a0c71d-d54e-4be8-a487-47371ca61456%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624agH6H2q-8YuknGzZH_7wWhVB1p9u8kscRRUZO5EOm%2Bzg%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4ityaBRoGoOf9p3GcUNy75eiXtA9F5KK24YwXuLzRZ43txw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.