What is 'marked shard as started, but shard has not been created, mark shard as failed"?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

What is 'marked shard as started, but shard has not been created, mark shard as failed"?

Jeffrey Zhou-2
We're using Elasticsearch 1.2.2 in our cluster, and have 3 dedicated master nodes. Last night, two of these master nodes (ES-SEARCH-64 and ES-SEARCH-65) became primary master alternately in a period, and we found quite a few 'marked shard as started, but shard has not been created, mark shard as failed' in the logs, see an example below. Does anyone know what this means? and why it happened? 

[2014-09-10 06:58:15,567][DEBUG][action.admin.cluster.health] [ES-SEARCH-64-master] connection exception while trying to forward request to master node [[ES-SEARCH-65-master][LF0n3IxmSFGcIG6FoqTqCw][es-search-65][inet[/10.0.4.5:9300]]{data=false, AvailabilitySet=master, master=true}], scheduling a retry. Error: [org.elasticsearch.transport.SendRequestTransportException: [ES-SEARCH-65-master][inet[/10.0.4.5:9300]][cluster/health]; org.elasticsearch.transport.NodeNotConnectedException: [ES-SEARCH-65-master][inet[/10.0.4.5:9300]] Node not connected]
[2014-09-10 06:58:16,130][DEBUG][action.admin.cluster.health] [ES-SEARCH-64-master] connection exception while trying to forward request to master node [[ES-SEARCH-65-master][LF0n3IxmSFGcIG6FoqTqCw][es-search-65][inet[/10.0.4.5:9300]]{data=false, AvailabilitySet=master, master=true}], scheduling a retry. Error: [org.elasticsearch.transport.SendRequestTransportException: [ES-SEARCH-65-master][inet[/10.0.4.5:9300]][cluster/health]; org.elasticsearch.transport.NodeNotConnectedException: [ES-SEARCH-65-master][inet[/10.0.4.5:9300]] Node not connected]
[2014-09-10 06:58:17,045][DEBUG][action.admin.cluster.health] [ES-SEARCH-64-master] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-09-10 06:58:18,342][DEBUG][action.admin.cluster.health] [ES-SEARCH-64-master] connection exception while trying to forward request to master node [[ES-SEARCH-65-master][LF0n3IxmSFGcIG6FoqTqCw][es-search-65][inet[/10.0.4.5:9300]]{data=false, AvailabilitySet=master, master=true}], scheduling a retry. Error: [org.elasticsearch.transport.SendRequestTransportException: [ES-SEARCH-65-master][inet[/10.0.4.5:9300]][cluster/health]; org.elasticsearch.transport.NodeNotConnectedException: [ES-SEARCH-65-master][inet[/10.0.4.5:9300]] Node not connected]
[2014-09-10 06:58:18,734][DEBUG][action.admin.cluster.health] [ES-SEARCH-64-master] connection exception while trying to forward request to master node [[ES-SEARCH-65-master][LF0n3IxmSFGcIG6FoqTqCw][es-search-65][inet[/10.0.4.5:9300]]{data=false, AvailabilitySet=master, master=true}], scheduling a retry. Error: [org.elasticsearch.transport.SendRequestTransportException: [ES-SEARCH-65-master][inet[/10.0.4.5:9300]][cluster/health]; org.elasticsearch.transport.NodeNotConnectedException: [ES-SEARCH-65-master][inet[/10.0.4.5:9300]] Node not connected]
[2014-09-10 06:58:20,093][WARN ][cluster.action.shard     ] [ES-SEARCH-64-master] [doc-v5][3] received shard failed for [doc-v5][3], node[8-q2fmItSOyde6Yk2yUbWw], [R], s[STARTED], indexUUID [fw1ocDrpSTejbY_m3WQP6Q], reason [master [ES-SEARCH-64-master][_zIl76rXQVGzTKVs4LbsZQ][es-search-64][inet[/10.0.4.4:9300]]{data=false, AvailabilitySet=master, master=true} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 06:58:20,093][WARN ][cluster.action.shard     ] [ES-SEARCH-64-master] [pub-v5][2] received shard failed for [pub-v5][2], node[8-q2fmItSOyde6Yk2yUbWw], [P], s[STARTED], indexUUID [4QlLH670Q_Kd6rb1WXhgaQ], reason [master [ES-SEARCH-64-master][_zIl76rXQVGzTKVs4LbsZQ][es-search-64][inet[/10.0.4.4:9300]]{data=false, AvailabilitySet=master, master=true} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 06:58:20,109][WARN ][cluster.action.shard     ] [ES-SEARCH-64-master] [doc-v5][23] received shard failed for [doc-v5][23], node[_uM3zeD6QXaj3eWEzagADw], [R], s[STARTED], indexUUID [fw1ocDrpSTejbY_m3WQP6Q], reason [master [ES-SEARCH-64-master][_zIl76rXQVGzTKVs4LbsZQ][es-search-64][inet[/10.0.4.4:9300]]{data=false, AvailabilitySet=master, master=true} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 06:58:20,109][WARN ][cluster.action.shard     ] [ES-SEARCH-64-master] [ppub-v5][16] received shard failed for [ppub-v5][16], node[_uM3zeD6QXaj3eWEzagADw], [P], s[STARTED], indexUUID [4QlLH670Q_Kd6rb1WXhgaQ], reason [master [ES-SEARCH-64-master][_zIl76rXQVGzTKVs4LbsZQ][es-search-64][inet[/10.0.4.4:9300]]{data=false, AvailabilitySet=master, master=true} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 06:58:20,155][WARN ][cluster.action.shard     ] [ES-SEARCH-64-master] [doc-v5][20] received shard failed for [doc-v5][20], node[3fvBrmUcQmyhQ5XniKs7lg], [R], s[STARTED], indexUUID [fw1ocDrpSTejbY_m3WQP6Q], reason [master [ES-SEARCH-64-master][_zIl76rXQVGzTKVs4LbsZQ][es-search-64][inet[/10.0.4.4:9300]]{data=false, AvailabilitySet=master, master=true} marked shard as started, but shard has not been created, mark shard as failed]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f42e8767-6c80-46a7-bdb1-6e9251579428%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.