ES cluster become red

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

ES cluster become red

Moshe Recanati
Hi,
I wrote simple program that enters 1000 documents into clean ES cluster.
While querying the cluster during execution I'm getting Green health all the time.

C:\Users\mosher>curl -XGET http://mosher:9200/_cat/indices?pretty=true
green twitter2 1 0 1000 0 72.7kb 72.7kb

However after stop and start of ES - I'm getting Red status
C:\Users\mosher>curl -XGET http://mosher:9200/_cat/indices?pretty=true
red twitter2 1 0

As you can see I'm running with 1 shard and no replication.

My code is below.

Let me know what I'm doing wrong and how to fix it.

Thank you,
Moshe


// on startup
Client client = new TransportClient()
        .addTransportAddress(new InetSocketTransportAddress("mosher", 9300));

try
{
CreateIndexRequestBuilder createIndexRequestBuilder = client.admin().indices().prepareCreate("twitter2");
createIndexRequestBuilder.execute().actionGet();
}
catch (Exception e)
{
e.printStackTrace();
}
BulkRequestBuilder bulkRequest = client.prepareBulk();
int numOfDocs = 1000;
long startTime = System.currentTimeMillis();
System.out.println("Going to add " + numOfDocs);
// either use client#prepare, or use Requests# to directly build index/delete requests
long internalStartTime = System.currentTimeMillis();
for (int i = 0; i < numOfDocs; i++)
{
IndexRequestBuilder index =client.prepareIndex("twitter2", "tweet", "m1"+i);
index.setSource(jsonBuilder()
                .startObject()
                .field("user", "kimchy" +i)
                .field("postDate", new Date())
                .field("message", "trying out Elasticsearch"+i)
            .endObject());
bulkRequest.add(index);
if (i % 1000 == 0 )
{
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
bulkRequest = client.prepareBulk();
System.out.println ("processed 1000 records from " + (i-1000) + " until  " + i + " at " + (System.currentTimeMillis() - internalStartTime));
internalStartTime = System.currentTimeMillis();
}
}
       
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures()) 
{
BulkItemResponse item[] = bulkResponse.getItems();
for (int i = 0; i< item.length; i++)
{
if (item[i].isFailed())
{
System.out.println ("Error " + item[i].getFailureMessage());
}
}
 
}
System.out.println ("Finished entereing " + numOfDocs + " in " + (System.currentTimeMillis() - startTime));
client.close();

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1d805126-0f18-40b4-b1d9-95557a18d4da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ES cluster become red

Moshe Recanati
Update
After couple of seconds or minutes the cluster became green.
I assume this is after ES stabilized with data,

Thank you

On Sunday, November 9, 2014 3:06:46 PM UTC+2, Moshe Recanati wrote:
Hi,
I wrote simple program that enters 1000 documents into clean ES cluster.
While querying the cluster during execution I'm getting Green health all the time.

C:\Users\mosher>curl -XGET <a href="http://mosher:9200/_cat/indices?pretty=true" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmosher%3A9200%2F_cat%2Findices%3Fpretty%3Dtrue\46sa\75D\46sntz\0751\46usg\75AFQjCNGCB_MPDwS8EVJYvLf3cOWN2QoXEQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmosher%3A9200%2F_cat%2Findices%3Fpretty%3Dtrue\46sa\75D\46sntz\0751\46usg\75AFQjCNGCB_MPDwS8EVJYvLf3cOWN2QoXEQ';return true;">http://mosher:9200/_cat/indices?pretty=true
green twitter2 1 0 1000 0 72.7kb 72.7kb

However after stop and start of ES - I'm getting Red status
C:\Users\mosher>curl -XGET <a href="http://mosher:9200/_cat/indices?pretty=true" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmosher%3A9200%2F_cat%2Findices%3Fpretty%3Dtrue\46sa\75D\46sntz\0751\46usg\75AFQjCNGCB_MPDwS8EVJYvLf3cOWN2QoXEQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmosher%3A9200%2F_cat%2Findices%3Fpretty%3Dtrue\46sa\75D\46sntz\0751\46usg\75AFQjCNGCB_MPDwS8EVJYvLf3cOWN2QoXEQ';return true;">http://mosher:9200/_cat/indices?pretty=true
red twitter2 1 0

As you can see I'm running with 1 shard and no replication.

My code is below.

Let me know what I'm doing wrong and how to fix it.

Thank you,
Moshe


// on startup
Client client = new TransportClient()
        .addTransportAddress(new InetSocketTransportAddress("mosher", 9300));

try
{
CreateIndexRequestBuilder createIndexRequestBuilder = client.admin().indices().prepareCreate("twitter2");
createIndexRequestBuilder.execute().actionGet();
}
catch (Exception e)
{
e.printStackTrace();
}
BulkRequestBuilder bulkRequest = client.prepareBulk();
int numOfDocs = 1000;
long startTime = System.currentTimeMillis();
System.out.println("Going to add " + numOfDocs);
// either use client#prepare, or use Requests# to directly build index/delete requests
long internalStartTime = System.currentTimeMillis();
for (int i = 0; i < numOfDocs; i++)
{
IndexRequestBuilder index =client.prepareIndex("twitter2", "tweet", "m1"+i);
index.setSource(jsonBuilder()
                .startObject()
                .field("user", "kimchy" +i)
                .field("postDate", new Date())
                .field("message", "trying out Elasticsearch"+i)
            .endObject());
bulkRequest.add(index);
if (i % 1000 == 0 )
{
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
bulkRequest = client.prepareBulk();
System.out.println ("processed 1000 records from " + (i-1000) + " until  " + i + " at " + (System.currentTimeMillis() - internalStartTime));
internalStartTime = System.currentTimeMillis();
}
}
       
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures()) 
{
BulkItemResponse item[] = bulkResponse.getItems();
for (int i = 0; i< item.length; i++)
{
if (item[i].isFailed())
{
System.out.println ("Error " + item[i].getFailureMessage());
}
}
 
}
System.out.println ("Finished entereing " + numOfDocs + " in " + (System.currentTimeMillis() - startTime));
client.close();

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/81d44823-62c8-40a8-9782-4cc7ddc258f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ES cluster become red

InquiringMind
Moshe,

Exactly!

What you might wish to do is add a Wait for Yellow query before doing any queries, or a Wait for Green request before doing any updates. That way, you can deterministically wait for the appropriate status before continuing.

For example: Loop on the following until it succeeds, some timeout expires after repeatedly catching NoNodeAvailableException, or else some other serious exception is thrown:

client.admin().cluster().prepareHealth().setTimeout(timeout).setWaitForYellowStatus().execute().actionGet();

Hope this helps!

Brian

On Sunday, November 9, 2014 8:22:58 AM UTC-5, Moshe Recanati wrote:
Update
After couple of seconds or minutes the cluster became green.
I assume this is after ES stabilized with data,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ecd2f91b-c322-4df8-b411-d47f59f356a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.