Shards get stuck in RELOCATING state

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Shards get stuck in RELOCATING state

arta
This post was updated on .
Hi,
I see two shards are stuck in RELOCATING state, for more than 18 hours as of now.
curl localhost:9200/_status shows something like this:
        "7" : [ {
          "routing" : {
            "state" : "RELOCATING",
            "primary" : false,
            "node" : "fyNmNb0yRnuVq-xHLFNU-Q",
            "relocating_node" : "Gm7QcAG0Tom-kZFNfL-IGQ",
            "shard" : 7,
            "index" : "i42"
          },

and it does not change.

I don't see any errors in logs.
(indices.recovery log level is default)

indexing and search seem working, but not sure for the shards under relocation.
curl command such as
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.cluster_concurrent_rebalance" : 10}}'
does not come back. (does not even timeout)

What is the best way to make it proceed?

Thanks for your help.
Reply | Threaded
Open this post in threaded view
|

Re: Shards get stuck in RELOCATING state

Ivan Brusic
Quick question: do you have enough disk space on all the nodes?

-- 
Ivan

On Tue, Nov 20, 2012 at 10:35 AM, arta <[hidden email]> wrote:
Hi,
I see two shards are stuck in RELOCATING state, for more than 18 hours as of
now.
curl localhost:9200/_status shows something like this:
        "7" : [ {
          "routing" : {
            "state" : "RELOCATING",
            "primary" : false,
            "node" : "fyNmNb0yRnuVq-xHLFNU-Q",
            "relocating_node" : "Gm7QcAG0Tom-kZFNfL-IGQ",
            "shard" : 7,
            "index" : "i42"
          },

and it does not change.

I don't see any errors in logs.
(indices.recovery log level is default)

What is the best way to make it proceed?

Thanks for your help.




--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Shards-get-stuck-in-RELOCATING-state-tp4025778.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--



--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Shards get stuck in RELOCATING state

arta
Thanks Ivan. Yes I have enough diskspace.
Reply | Threaded
Open this post in threaded view
|

Re: Shards get stuck in RELOCATING state

arta
Additional notes:

indexing and search seem working, but not sure for the shards under relocation.
curl command such as
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.cluster_concurrent_rebalance" : 10}}'
does not come back. (does not even timeout)
Reply | Threaded
Open this post in threaded view
|

Re: Shards get stuck in RELOCATING state

Chris Male
Hi,

Are you able to give us some information about what version of ElasticSearch you're using and any information about your cluster layout / network layout.

On Wednesday, November 21, 2012 11:02:05 AM UTC+13, arta wrote:
Additional notes:

indexing and search seem working, but not sure for the shards under
relocation.
curl command such as
curl -XPUT localhost:9200/_cluster/settings -d
'{"transient":{"cluster.routing.allocation.cluster_concurrent_rebalance" :
10}}'
does not come back. (does not even timeout)




--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Shards-get-stuck-in-RELOCATING-state-tp4025778p4025783.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Shards get stuck in RELOCATING state

arta
Hi Chris, thanks for the reply.

I'm using 0.19.3.
I have 19 data nodes in one cluster, all are master eligible, they are in our private network.
Is this enough info for you?
Reply | Threaded
Open this post in threaded view
|

Re: Shards get stuck in RELOCATING state

Thomas Peuss
Hi!

Am Donnerstag, 22. November 2012 01:14:37 UTC+1 schrieb arta:
Hi Chris, thanks for the reply.

I'm using 0.19.3.
I have 19 data nodes in one cluster, all are master eligible, they are in
our private network.
Is this enough info for you?


We are running an 8 node cluster and I must say that it was not very stable before we installed 0.19.11 (we jumped from 0.19.4). Now we are running 0.19.11 and it is rock solid. We had similar problems with stuck shards.

CU
Thomas

--