Troubleshooting ES Resharding. Nature of immediate tasks and other questions

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Troubleshooting ES Resharding. Nature of immediate tasks and other questions

tomas.nino
Hi all,

We've been facing some trouble with our Elastic Search installation (v 1.5.1), mostly trying
to bring it back up. Some questions have come up.

This is our situation. We're seeing about 200 unassigned shards, which are not being 
reassigned automatically, which in turn leads our ES to move into a red status. In this state, 
queries are disabled, however we keep storing data.

What we see is that this generates refresh/update mapping tasks which are never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been running _cluster/reroute on primary
shards:

curl -XPOST 'localhost:9290/_cluster/reroute' -d '{
     "commands": [
        {
            "allocate": {
                "index": "'$INDEX'",
                "shard": '$SHARD',
                "node": "'$NODE'",
                "allow_primary": true
          }
        }
    ]
  }'

Commands such as _nodes/_local/{stats, settings, os, processes} will fail on the master node (hang indefinetly)

We monitor the pending tasks (see _cat/pending_tasks  and _cluster/health output below) and see that
the IMMEDIATE tasks are queued. 

However, we're wondering what's the rationale behind the queuing of this tasks:

Is there a round robin mechanism for the IMMEDIATE or is there some way of prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task preempt any other (e.g URGENT or HIGH?)

We've noticed that when queuing two IMMEDIATE tasks, the second one may timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single thread? If so, anyway to change that? 


Thanks in advance!
Tomás
--- 

_cat/pending_tasks 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0220771  2.1m IMMEDIATE cluster_reroute (api)                                                                                       
220891 29.8s IMMEDIATE cluster_reroute (api)                                                                                       
196772 10.1h HIGH      update-mapping [beer.raw.cl.business.2015-05-04][GDS_search_scans] / node [v5SxZ7CdRou13tzy-N1DJg], order [1109] 
220892 25.9s IMMEDIATE cluster_reroute (api)                                                                                       
196773 10.1h HIGH      update-mapping [beer.raw.pe.business.2015-05-04][GDS_search_scans] / node [BTYaSC3cT8K_3xHDQYoNXQ], order [419] 
196779 10.1h HIGH      update-mapping [beer.raw.ar.business.2015-05-04][prism_retrieve] / node [iDVZlJycRdeOa1PGB4Oi9Q], order [127] 
220893 25.7s IMMEDIATE cluster_reroute (api)                                                                                       
196787 10.1h HIGH      refresh-mapping [beer.raw.pt.business.2015-05-03][[GDS_search_scans]]                                       
196786 10.1h HIGH      refresh-mapping [beer.raw.ca.business.2015-05-03][[GDS_search_scans]]                                       
196774 10.1h HIGH      update-mapping [beer.raw.pe.business.2015-05-04][GDS_search_scans] / node [Kx-HMg4qQKqepJb1qjjS3A], order [151] 
196790 10.1h HIGH      refresh-mapping [beer.raw.ae.search.2015-05-03][[vito]]                                                     
196792 10.1h HIGH      refresh-mapping [beer.raw.tr.business.2015-05-03][[GDS_search_scans]]                                       
218944 35.5m URGENT    shard-started ([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K_3xHDQYoNXQ], relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after recovery (replica) from node [[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/10.70.163.240:9300]]{master=false}]]                                                                     
220894 25.7s IMMEDIATE cluster_reroute (api)                                                                                       
196850 10.1h HIGH      refresh-mapping [beer.raw.fi.business.2015-05-03][[GDS_search_scans]]                                       
196788 10.1h HIGH      refresh-mapping [beer.raw.il.business.2015-05-03][[GDS_search_scans]]                                       
196789 10.1h HIGH      refresh-mapping [beer.raw.nl.business.2015-05-03][[GDS_search_scans]]                                       



Health

_cluster/health?pretty=true"

  "cluster_name" : "beer-elastic",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 20,
  "number_of_data_nodes" : 12,
  "active_primary_shards" : 32192,
  "active_shards" : 64384,
  "relocating_shards" : 2,
  "initializing_shards" : 14,
  "unassigned_shards" : 182,
  "number_of_pending_tasks" : 13686

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Troubleshooting ES Resharding. Nature of immediate tasks and other questions

Jason Wee
hello, just out of curiosity, what is the cluster index size and what is the rationale to have so many shards?

jason

On Tue, May 5, 2015 at 5:05 AM, <[hidden email]> wrote:
Hi all,

We've been facing some trouble with our Elastic Search installation (v 1.5.1), mostly trying
to bring it back up. Some questions have come up.

This is our situation. We're seeing about 200 unassigned shards, which are not being 
reassigned automatically, which in turn leads our ES to move into a red status. In this state, 
queries are disabled, however we keep storing data.

What we see is that this generates refresh/update mapping tasks which are never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been running _cluster/reroute on primary
shards:

curl -XPOST 'localhost:9290/_cluster/reroute' -d '{
     "commands": [
        {
            "allocate": {
                "index": "'$INDEX'",
                "shard": '$SHARD',
                "node": "'$NODE'",
                "allow_primary": true
          }
        }
    ]
  }'

Commands such as _nodes/_local/{stats, settings, os, processes} will fail on the master node (hang indefinetly)

We monitor the pending tasks (see _cat/pending_tasks  and _cluster/health output below) and see that
the IMMEDIATE tasks are queued. 

However, we're wondering what's the rationale behind the queuing of this tasks:

Is there a round robin mechanism for the IMMEDIATE or is there some way of prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task preempt any other (e.g URGENT or HIGH?)

We've noticed that when queuing two IMMEDIATE tasks, the second one may timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single thread? If so, anyway to change that? 


Thanks in advance!
Tomás
--- 

_cat/pending_tasks 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0220771  2.1m IMMEDIATE cluster_reroute (api)                                                                                       
220891 29.8s IMMEDIATE cluster_reroute (api)                                                                                       
196772 10.1h HIGH      update-mapping [beer.raw.cl.business.2015-05-04][GDS_search_scans] / node [v5SxZ7CdRou13tzy-N1DJg], order [1109] 
220892 25.9s IMMEDIATE cluster_reroute (api)                                                                                       
196773 10.1h HIGH      update-mapping [beer.raw.pe.business.2015-05-04][GDS_search_scans] / node [BTYaSC3cT8K_3xHDQYoNXQ], order [419] 
196779 10.1h HIGH      update-mapping [beer.raw.ar.business.2015-05-04][prism_retrieve] / node [iDVZlJycRdeOa1PGB4Oi9Q], order [127] 
220893 25.7s IMMEDIATE cluster_reroute (api)                                                                                       
196787 10.1h HIGH      refresh-mapping [beer.raw.pt.business.2015-05-03][[GDS_search_scans]]                                       
196786 10.1h HIGH      refresh-mapping [beer.raw.ca.business.2015-05-03][[GDS_search_scans]]                                       
196774 10.1h HIGH      update-mapping [beer.raw.pe.business.2015-05-04][GDS_search_scans] / node [Kx-HMg4qQKqepJb1qjjS3A], order [151] 
196790 10.1h HIGH      refresh-mapping [beer.raw.ae.search.2015-05-03][[vito]]                                                     
196792 10.1h HIGH      refresh-mapping [beer.raw.tr.business.2015-05-03][[GDS_search_scans]]                                       
218944 35.5m URGENT    shard-started ([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K_3xHDQYoNXQ], relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after recovery (replica) from node [[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/10.70.163.240:9300]]{master=false}]]                                                                     
220894 25.7s IMMEDIATE cluster_reroute (api)                                                                                       
196850 10.1h HIGH      refresh-mapping [beer.raw.fi.business.2015-05-03][[GDS_search_scans]]                                       
196788 10.1h HIGH      refresh-mapping [beer.raw.il.business.2015-05-03][[GDS_search_scans]]                                       
196789 10.1h HIGH      refresh-mapping [beer.raw.nl.business.2015-05-03][[GDS_search_scans]]                                       



Health

_cluster/health?pretty=true"

  "cluster_name" : "beer-elastic",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 20,
  "number_of_data_nodes" : 12,
  "active_primary_shards" : 32192,
  "active_shards" : 64384,
  "relocating_shards" : 2,
  "initializing_shards" : 14,
  "unassigned_shards" : 182,
  "number_of_pending_tasks" : 13686

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itwhNtE0bce-biK7m6MwH59yB9Q5r1tZ6CUz7U%2BjW-HTFw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Troubleshooting ES Resharding. Nature of immediate tasks and other questions

Mark Walkom-2
In reply to this post by tomas.nino
The rationale of queuing is to allow for instances where temporary load on the cluster might otherwise reject a request.
There is no way to prioritise tasks over other tasks.

Though it looks like your problem is you are overloading your nodes. 32192 primary shards is a massive amount for only 12 nodes, you really need to reduce this pretty dramatically to alleviate the pressure.

On 5 May 2015 at 07:05, <[hidden email]> wrote:
Hi all,

We've been facing some trouble with our Elastic Search installation (v 1.5.1), mostly trying
to bring it back up. Some questions have come up.

This is our situation. We're seeing about 200 unassigned shards, which are not being 
reassigned automatically, which in turn leads our ES to move into a red status. In this state, 
queries are disabled, however we keep storing data.

What we see is that this generates refresh/update mapping tasks which are never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been running _cluster/reroute on primary
shards:

curl -XPOST 'localhost:9290/_cluster/reroute' -d '{
     "commands": [
        {
            "allocate": {
                "index": "'$INDEX'",
                "shard": '$SHARD',
                "node": "'$NODE'",
                "allow_primary": true
          }
        }
    ]
  }'

Commands such as _nodes/_local/{stats, settings, os, processes} will fail on the master node (hang indefinetly)

We monitor the pending tasks (see _cat/pending_tasks  and _cluster/health output below) and see that
the IMMEDIATE tasks are queued. 

However, we're wondering what's the rationale behind the queuing of this tasks:

Is there a round robin mechanism for the IMMEDIATE or is there some way of prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task preempt any other (e.g URGENT or HIGH?)

We've noticed that when queuing two IMMEDIATE tasks, the second one may timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single thread? If so, anyway to change that? 


Thanks in advance!
Tomás
--- 

_cat/pending_tasks 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0220771  2.1m IMMEDIATE cluster_reroute (api)                                                                                       
220891 29.8s IMMEDIATE cluster_reroute (api)                                                                                       
196772 10.1h HIGH      update-mapping [beer.raw.cl.business.2015-05-04][GDS_search_scans] / node [v5SxZ7CdRou13tzy-N1DJg], order [1109] 
220892 25.9s IMMEDIATE cluster_reroute (api)                                                                                       
196773 10.1h HIGH      update-mapping [beer.raw.pe.business.2015-05-04][GDS_search_scans] / node [BTYaSC3cT8K_3xHDQYoNXQ], order [419] 
196779 10.1h HIGH      update-mapping [beer.raw.ar.business.2015-05-04][prism_retrieve] / node [iDVZlJycRdeOa1PGB4Oi9Q], order [127] 
220893 25.7s IMMEDIATE cluster_reroute (api)                                                                                       
196787 10.1h HIGH      refresh-mapping [beer.raw.pt.business.2015-05-03][[GDS_search_scans]]                                       
196786 10.1h HIGH      refresh-mapping [beer.raw.ca.business.2015-05-03][[GDS_search_scans]]                                       
196774 10.1h HIGH      update-mapping [beer.raw.pe.business.2015-05-04][GDS_search_scans] / node [Kx-HMg4qQKqepJb1qjjS3A], order [151] 
196790 10.1h HIGH      refresh-mapping [beer.raw.ae.search.2015-05-03][[vito]]                                                     
196792 10.1h HIGH      refresh-mapping [beer.raw.tr.business.2015-05-03][[GDS_search_scans]]                                       
218944 35.5m URGENT    shard-started ([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K_3xHDQYoNXQ], relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after recovery (replica) from node [[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/10.70.163.240:9300]]{master=false}]]                                                                     
220894 25.7s IMMEDIATE cluster_reroute (api)                                                                                       
196850 10.1h HIGH      refresh-mapping [beer.raw.fi.business.2015-05-03][[GDS_search_scans]]                                       
196788 10.1h HIGH      refresh-mapping [beer.raw.il.business.2015-05-03][[GDS_search_scans]]                                       
196789 10.1h HIGH      refresh-mapping [beer.raw.nl.business.2015-05-03][[GDS_search_scans]]                                       



Health

_cluster/health?pretty=true"

  "cluster_name" : "beer-elastic",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 20,
  "number_of_data_nodes" : 12,
  "active_primary_shards" : 32192,
  "active_shards" : 64384,
  "relocating_shards" : 2,
  "initializing_shards" : 14,
  "unassigned_shards" : 182,
  "number_of_pending_tasks" : 13686

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8eRJr6S%2B4k65tx6sPYMeDtiEuEhxJRicGtff35Gzz%3D%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Troubleshooting ES Resharding. Nature of immediate tasks and other questions

Alejandro De Lío
Hey guys, thanks for the answers.

Firstly, the index size can vary from index to index (as each index is associated with a particular business sector, each of wich might be more or less active). The max size is approximately 1 GB for the heaviest indexes.

Secondly, what do you think it would a sane ratio between shards and nodes (we have 5 primary shards per index)?

Background:
- the storage nodes have 24 GB RAM and 8 cores with 500 GB attached disks (x 12 nodes)
- we generate around 4.000 shards per day (around 400 indexes with default configuration of 5 primary shard per index and 1 replica per shard)
- one index contains only one document type

The reason we've chosen this "many indexes design" was, fundamentally, heterogeneous TTLs. We thought it would be better to drop a bunch of small indexes rather than scan a whole giant index (around 3GB per shard), then remove specific events and finally compact it.

Do you think it might be better to have just a few indexes and perform sporadic cleanup tasks, over partitioning information into lots of independent, small, flexible indexes? It should be taken into account that some of the current indexes have update-mapping operations frequently applied to it, as the document structure might have new fields.

Thanks for your time!
Ale



El lunes, 4 de mayo de 2015, 22:01:46 (UTC-3), Mark Walkom escribió:
The rationale of queuing is to allow for instances where temporary load on the cluster might otherwise reject a request.
There is no way to prioritise tasks over other tasks.

Though it looks like your problem is you are overloading your nodes. 32192 primary shards is a massive amount for only 12 nodes, you really need to reduce this pretty dramatically to alleviate the pressure.

On 5 May 2015 at 07:05, <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="_6uxOoQRXEIJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">tomas...@...> wrote:
Hi all,

We've been facing some trouble with our Elastic Search installation (v 1.5.1), mostly trying
to bring it back up. Some questions have come up.

This is our situation. We're seeing about 200 unassigned shards, which are not being 
reassigned automatically, which in turn leads our ES to move into a red status. In this state, 
queries are disabled, however we keep storing data.

What we see is that this generates refresh/update mapping tasks which are never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been running _cluster/reroute on primary
shards:

curl -XPOST 'localhost:9290/_cluster/reroute' -d '{
     "commands": [
        {
            "allocate": {
                "index": "'$INDEX'",
                "shard": '$SHARD',
                "node": "'$NODE'",
                "allow_primary": true
          }
        }
    ]
  }'

Commands such as _nodes/_local/{stats, settings, os, processes} will fail on the master node (hang indefinetly)

We monitor the pending tasks (see _cat/pending_tasks  and _cluster/health output below) and see that
the IMMEDIATE tasks are queued. 

However, we're wondering what's the rationale behind the queuing of this tasks:

Is there a round robin mechanism for the IMMEDIATE or is there some way of prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task preempt any other (e.g URGENT or HIGH?)

We've noticed that when queuing two IMMEDIATE tasks, the second one may timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single thread? If so, anyway to change that? 


Thanks in advance!
Tomás
--- 

_cat/pending_tasks 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0220771  2.1m IMMEDIATE cluster_reroute (api)                                                                                       
220891 29.8s IMMEDIATE cluster_reroute (api)                                                                                       
196772 10.1h HIGH      update-mapping [beer.raw.cl.business.2015-05-04][GDS_search_scans] / node [v5SxZ7CdRou13tzy-N1DJg], order [1109] 
220892 25.9s IMMEDIATE cluster_reroute (api)                                                                                       
196773 10.1h HIGH      update-mapping [beer.raw.pe.business.2015-05-04][GDS_search_scans] / node [BTYaSC3cT8K_3xHDQYoNXQ], order [419] 
196779 10.1h HIGH      update-mapping [beer.raw.ar.business.2015-05-04][prism_retrieve] / node [iDVZlJycRdeOa1PGB4Oi9Q], order [127] 
220893 25.7s IMMEDIATE cluster_reroute (api)                                                                                       
196787 10.1h HIGH      refresh-mapping [beer.raw.pt.business.2015-05-03][[GDS_search_scans]]                                       
196786 10.1h HIGH      refresh-mapping [beer.raw.ca.business.2015-05-03][[GDS_search_scans]]                                       
196774 10.1h HIGH      update-mapping [beer.raw.pe.business.2015-05-04][GDS_search_scans] / node [Kx-HMg4qQKqepJb1qjjS3A], order [151] 
196790 10.1h HIGH      refresh-mapping [beer.raw.ae.search.2015-05-03][[vito]]                                                     
196792 10.1h HIGH      refresh-mapping [beer.raw.tr.business.2015-05-03][[GDS_search_scans]]                                       
218944 35.5m URGENT    shard-started ([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K_3xHDQYoNXQ], relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after recovery (replica) from node [[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/<a href="http://10.70.163." target="_blank" rel="nofollow" onmousedown="this.href='http://10.70.163.';return true;" onclick="this.href='http://10.70.163.';return true;">10.70.163.240:9300]]{master=false}]]                                                                     
220894 25.7s IMMEDIATE cluster_reroute (api)                                                                                       
196850 10.1h HIGH      refresh-mapping [beer.raw.fi.business.2015-05-03][[GDS_search_scans]]                                       
196788 10.1h HIGH      refresh-mapping [beer.raw.il.business.2015-05-03][[GDS_search_scans]]                                       
196789 10.1h HIGH      refresh-mapping [beer.raw.nl.business.2015-05-03][[GDS_search_scans]]                                       



Health

_cluster/health?pretty=true"

  "cluster_name" : "beer-elastic",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 20,
  "number_of_data_nodes" : 12,
  "active_primary_shards" : 32192,
  "active_shards" : 64384,
  "relocating_shards" : 2,
  "initializing_shards" : 14,
  "unassigned_shards" : 182,
  "number_of_pending_tasks" : 13686

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="_6uxOoQRXEIJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Please update your bookmarks! We moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Troubleshooting ES Resharding. Nature of immediate tasks and other questions

Mark Walkom-2
If your largest index is only 1GB, then try reducing that to a single shard (with one replica) and see how the performance goes.
I'd trial that on a few indices before rolling it out across the board though.

Again, the core issue is that you have too many shards!


On 7 May 2015 at 00:34, Alejandro De Lío <[hidden email]> wrote:
Hey guys, thanks for the answers.

Firstly, the index size can vary from index to index (as each index is associated with a particular business sector, each of wich might be more or less active). The max size is approximately 1 GB for the heaviest indexes.

Secondly, what do you think it would a sane ratio between shards and nodes (we have 5 primary shards per index)?

Background:
- the storage nodes have 24 GB RAM and 8 cores with 500 GB attached disks (x 12 nodes)
- we generate around 4.000 shards per day (around 400 indexes with default configuration of 5 primary shard per index and 1 replica per shard)
- one index contains only one document type

The reason we've chosen this "many indexes design" was, fundamentally, heterogeneous TTLs. We thought it would be better to drop a bunch of small indexes rather than scan a whole giant index (around 3GB per shard), then remove specific events and finally compact it.

Do you think it might be better to have just a few indexes and perform sporadic cleanup tasks, over partitioning information into lots of independent, small, flexible indexes? It should be taken into account that some of the current indexes have update-mapping operations frequently applied to it, as the document structure might have new fields.

Thanks for your time!
Ale



El lunes, 4 de mayo de 2015, 22:01:46 (UTC-3), Mark Walkom escribió:
The rationale of queuing is to allow for instances where temporary load on the cluster might otherwise reject a request.
There is no way to prioritise tasks over other tasks.

Though it looks like your problem is you are overloading your nodes. 32192 primary shards is a massive amount for only 12 nodes, you really need to reduce this pretty dramatically to alleviate the pressure.

On 5 May 2015 at 07:05, <[hidden email]> wrote:
Hi all,

We've been facing some trouble with our Elastic Search installation (v 1.5.1), mostly trying
to bring it back up. Some questions have come up.

This is our situation. We're seeing about 200 unassigned shards, which are not being 
reassigned automatically, which in turn leads our ES to move into a red status. In this state, 
queries are disabled, however we keep storing data.

What we see is that this generates refresh/update mapping tasks which are never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been running _cluster/reroute on primary
shards:

curl -XPOST 'localhost:9290/_cluster/reroute' -d '{
     "commands": [
        {
            "allocate": {
                "index": "'$INDEX'",
                "shard": '$SHARD',
                "node": "'$NODE'",
                "allow_primary": true
          }
        }
    ]
  }'

Commands such as _nodes/_local/{stats, settings, os, processes} will fail on the master node (hang indefinetly)

We monitor the pending tasks (see _cat/pending_tasks  and _cluster/health output below) and see that
the IMMEDIATE tasks are queued. 

However, we're wondering what's the rationale behind the queuing of this tasks:

Is there a round robin mechanism for the IMMEDIATE or is there some way of prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task preempt any other (e.g URGENT or HIGH?)

We've noticed that when queuing two IMMEDIATE tasks, the second one may timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single thread? If so, anyway to change that? 


Thanks in advance!
Tomás
--- 

_cat/pending_tasks 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0220771  2.1m IMMEDIATE cluster_reroute (api)                                                                                       
220891 29.8s IMMEDIATE cluster_reroute (api)                                                                                       
196772 10.1h HIGH      update-mapping [beer.raw.cl.business.2015-05-04][GDS_search_scans] / node [v5SxZ7CdRou13tzy-N1DJg], order [1109] 
220892 25.9s IMMEDIATE cluster_reroute (api)                                                                                       
196773 10.1h HIGH      update-mapping [beer.raw.pe.business.2015-05-04][GDS_search_scans] / node [BTYaSC3cT8K_3xHDQYoNXQ], order [419] 
196779 10.1h HIGH      update-mapping [beer.raw.ar.business.2015-05-04][prism_retrieve] / node [iDVZlJycRdeOa1PGB4Oi9Q], order [127] 
220893 25.7s IMMEDIATE cluster_reroute (api)                                                                                       
196787 10.1h HIGH      refresh-mapping [beer.raw.pt.business.2015-05-03][[GDS_search_scans]]                                       
196786 10.1h HIGH      refresh-mapping [beer.raw.ca.business.2015-05-03][[GDS_search_scans]]                                       
196774 10.1h HIGH      update-mapping [beer.raw.pe.business.2015-05-04][GDS_search_scans] / node [Kx-HMg4qQKqepJb1qjjS3A], order [151] 
196790 10.1h HIGH      refresh-mapping [beer.raw.ae.search.2015-05-03][[vito]]                                                     
196792 10.1h HIGH      refresh-mapping [beer.raw.tr.business.2015-05-03][[GDS_search_scans]]                                       
218944 35.5m URGENT    shard-started ([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K_3xHDQYoNXQ], relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after recovery (replica) from node [[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/10.70.163.240:9300]]{master=false}]]                                                                     
220894 25.7s IMMEDIATE cluster_reroute (api)                                                                                       
196850 10.1h HIGH      refresh-mapping [beer.raw.fi.business.2015-05-03][[GDS_search_scans]]                                       
196788 10.1h HIGH      refresh-mapping [beer.raw.il.business.2015-05-03][[GDS_search_scans]]                                       
196789 10.1h HIGH      refresh-mapping [beer.raw.nl.business.2015-05-03][[GDS_search_scans]]                                       



Health

_cluster/health?pretty=true"

  "cluster_name" : "beer-elastic",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 20,
  "number_of_data_nodes" : 12,
  "active_primary_shards" : 32192,
  "active_shards" : 64384,
  "relocating_shards" : 2,
  "initializing_shards" : 14,
  "unassigned_shards" : 182,
  "number_of_pending_tasks" : 13686

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8dTdx6n8RfqMwgUX__%2B1_3FnMru%3DCGF4SPREy-FbGiFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.