Lost shards and cluster state stays red

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Lost shards and cluster state stays red

arta
I had a hardware trouble during ES was in yellow state.
Some shards that were in the troubled disk were gone.
It seems I lost some primary shards because it was in yellow state and some shards that were in the troubled node did not have its replica.
(my number_of_replicas is 1)

The shard reallocation has finished but the cluster state stays red.
{"cluster_name":"es-cluster1",
 "status":"red",
 "timed_out":false,
 "number_of_nodes":16,
 "number_of_data_nodes":16,
 "active_primary_shards":2469,
 "active_shards":4938,
 "relocating_shards":0,
 "initializing_shards":0,
 "unassigned_shards":118}

It should have 2528 primary shards but (2528 - 2469 = 59) shards are missing.

I'll reindex documents belonged to the lost shards, but is there any way to put the cluster back to yellow/green state, so that at the same time I can update index and search existing documents?

Thanks for your help.
I want to
Reply | Threaded
Open this post in threaded view
|

Re: Lost shards and cluster state stays red

Radu Gheorghe-2
Hello,

I'm not sure if I understood you question correctly, but what you can do is:
- delete the indices which have missing shards. Something like:
# curl -XDELETE localhost:9200/corrupted_index/
- reindex data belonging to those "incomplete" indices

Then your cluster state should be back to yellow/green again. Until
then, if you have indices that have missing shards but also allocated
shards, ES will still run your searches on the data you have. If
that's important to you, then you might prefer to do things like this:
- reindex data belonging to incomplete indices into new indices with
different names
- delete indices with missing shards
- add aliases[0] to the new indices with the old index names, so that
searches will run as before

[0] http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html

Best regards,
Radu
--
http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

On Sat, Nov 17, 2012 at 1:23 AM, arta <[hidden email]> wrote:

> I had a hardware trouble during ES was in yellow state.
> Some shards that were in the troubled disk were gone.
> It seems I lost some primary shards because it was in yellow state and some
> shards that were in the troubled node did not have its replica.
> (my number_of_replicas is 1)
>
> The shard reallocation has finished but the cluster state stays red.
> {"cluster_name":"es-cluster1",
>  "status":"red",
>  "timed_out":false,
>  "number_of_nodes":16,
>  "number_of_data_nodes":16,
>  "active_primary_shards":2469,
>  "active_shards":4938,
>  "relocating_shards":0,
>  "initializing_shards":0,
>  "unassigned_shards":118}
>
> It should have 2528 primary shards but (2528 - 2469 = 59) shards are
> missing.
>
> I'll reindex documents belonged to the lost shards, but is there any way to
> put the cluster back to yellow/green state, so that at the same time I can
> update index and search existing documents?
>
> Thanks for your help.
> I want to
>
>
>
> --
> View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Lost-shards-and-cluster-state-stays-red-tp4025616.html
> Sent from the ElasticSearch Users mailing list archive at Nabble.com.
>
> --
>

--


Reply | Threaded
Open this post in threaded view
|

Re: Lost shards and cluster state stays red

arta
Thank you Radu,
I don't want to delete indices, as they have millions of docs.

I found this topic:

http://elasticsearch-users.115913.n3.nabble.com/Recovering-after-shard-failure-td4018776.html

I created a dummy index, manually copied one of it's shards to the node where shards were lost, renamed shard directory names as they were, started elastic search at the node, and deleted the dummy index.
ES got back to green.
Docs in the indices remained (of course except for ones in the lost shards).

--