Can I clarify how admin api works for creating an index?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Can I clarify how admin api works for creating an index?

zohar

We are testing to see if https://github.com/elasticsearch/elasticsearch/issues/#issue/726
is fixed.
Our test still fails - but differently now. Instead of it hanging, we
get an error.
"Failed to execute phase [query_fetch], total failure; shardFailures
{[_na_][testindex][0]: No active shards}"

We are creating the index thus:

CreateIndexRequest request = createIndexRequest(indexName);
request.settings("{number_of_shards : 1 }");
client.admin().indices().create(request).actionGet();

Is the index created & ready when "actionGet" method returns?

When we try and search this index immediately after creating it
(theres nothing in the index - we just expect zero results) we get the
above error.

Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

client.admin()
    .cluster()
    .health(
        clusterHealthRequest()
            .waitForYellowStatus() // we only have 1 shard - hence
wait for yellow
            .timeout(timeValueSeconds(60))
    )
    .actionGet();

Reply | Threaded
Open this post in threaded view
|

Re: Can I clarify how admin api works for creating an index?

Clinton Gormley
Hi Zohar

> Is the index created & ready when "actionGet" method returns?

As I understand it (and I stand under correction), a create index call
is async, so it is not ready when actionGet returns.


> Out of curiosity, we also added the following code, after creating the
> index.
> This makes it happy - but not sure if thats just because the call
> introduces sufficient delay - or whether its what we need to do.

Not a coincidence. Using a wait_for_status call is the correct thing to
do - exactly what it is designed for.

clint


Reply | Threaded
Open this post in threaded view
|

Re: Can I clarify how admin api works for creating an index?

kimchy
Administrator
There are three different things associated with creating an index. The first is adding it to the cluster metadata, then creating it on each relevant node, and finally, allocating shards to nodes that resulted from the index creation.

The create index API makes sure that the index provided is correct (analyzers are found, mappings are correct), waits for the *index* itself to be created on the relevant nodes, but does not wait for the shards to be allocated.

The cluster health API, with the wait_for..., allows to wait till shards are allocated. Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

-shay.banon

On Tuesday, March 15, 2011 at 5:32 PM, Clinton Gormley wrote:

Hi Zohar

Is the index created & ready when "actionGet" method returns?

As I understand it (and I stand under correction), a create index call
is async, so it is not ready when actionGet returns.


Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

Not a coincidence. Using a wait_for_status call is the correct thing to
do - exactly what it is designed for.

clint

Reply | Threaded
Open this post in threaded view
|

Re: Can I clarify how admin api works for creating an index?

mooky

>> The create index API makes sure that the index provided is correct
>> (analyzers are found, mappings are correct), waits for the *index* itself to
>> be created on the relevant nodes, but does not wait for the shards to be allocated.

What's the rationale for not waiting for the shards to be allocated?
(just for my education/understanding)

>> Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

Again, just for my education/understanding, why doesn't the search
wait?

-Nick



On Mar 15, 4:02 pm, Shay Banon <[hidden email]> wrote:

> There are three different things associated with creating an index. The first is adding it to the cluster metadata, then creating it on each relevant node, and finally, allocating shards to nodes that resulted from the index creation.
>
> The create index API makes sure that the index provided is correct (analyzers are found, mappings are correct), waits for the *index* itself to be created on the relevant nodes, but does not wait for the shards to be allocated.
>
> The cluster health API, with the wait_for..., allows to wait till shards are allocated. Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.
>
> -shay.banon
>
>
>
>
>
>
>
> On Tuesday, March 15, 2011 at 5:32 PM, Clinton Gormley wrote:
> > Hi Zohar
>
> > > Is the index created & ready when "actionGet" method returns?
>
> > As I understand it (and I stand under correction), a create index call
> > is async, so it is not ready when actionGet returns.
>
> > > Out of curiosity, we also added the following code, after creating the
> > > index.
> > > This makes it happy - but not sure if thats just because the call
> > > introduces sufficient delay - or whether its what we need to do.
>
> > Not a coincidence. Using a wait_for_status call is the correct thing to
> > do - exactly what it is designed for.
>
> > clint
Reply | Threaded
Open this post in threaded view
|

Re: Can I clarify how admin api works for creating an index?

kimchy
Administrator
No specific reason except for the complexity involved in implementing it, and the fact that it mainly when doing testing.

On Tuesday, March 15, 2011 at 7:46 PM, Mooky wrote:


The create index API makes sure that the index provided is correct
(analyzers are found, mappings are correct), waits for the *index* itself to
be created on the relevant nodes, but does not wait for the shards to be allocated.

What's the rationale for not waiting for the shards to be allocated?
(just for my education/understanding)

Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

Again, just for my education/understanding, why doesn't the search
wait?

-Nick



On Mar 15, 4:02 pm, Shay Banon <shay.ba...@elasticsearch.com> wrote:
There are three different things associated with creating an index. The first is adding it to the cluster metadata, then creating it on each relevant node, and finally, allocating shards to nodes that resulted from the index creation.

The create index API makes sure that the index provided is correct (analyzers are found, mappings are correct), waits for the *index* itself to be created on the relevant nodes, but does not wait for the shards to be allocated.

The cluster health API, with the wait_for..., allows to wait till shards are allocated. Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

-shay.banon







On Tuesday, March 15, 2011 at 5:32 PM, Clinton Gormley wrote:
Hi Zohar

Is the index created & ready when "actionGet" method returns?

As I understand it (and I stand under correction), a create index call
is async, so it is not ready when actionGet returns.

Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

Not a coincidence. Using a wait_for_status call is the correct thing to
do - exactly what it is designed for.

clint

Reply | Threaded
Open this post in threaded view
|

Re: Can I clarify how admin api works for creating an index?

kimchy
Administrator
Also, you might actually want to get search response when not all shards are available.

On Tuesday, March 15, 2011 at 9:25 PM, Shay Banon wrote:

No specific reason except for the complexity involved in implementing it, and the fact that it mainly when doing testing.

On Tuesday, March 15, 2011 at 7:46 PM, Mooky wrote:


The create index API makes sure that the index provided is correct
(analyzers are found, mappings are correct), waits for the *index* itself to
be created on the relevant nodes, but does not wait for the shards to be allocated.

What's the rationale for not waiting for the shards to be allocated?
(just for my education/understanding)

Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

Again, just for my education/understanding, why doesn't the search
wait?

-Nick



On Mar 15, 4:02 pm, Shay Banon <shay.ba...@elasticsearch.com> wrote:
There are three different things associated with creating an index. The first is adding it to the cluster metadata, then creating it on each relevant node, and finally, allocating shards to nodes that resulted from the index creation.

The create index API makes sure that the index provided is correct (analyzers are found, mappings are correct), waits for the *index* itself to be created on the relevant nodes, but does not wait for the shards to be allocated.

The cluster health API, with the wait_for..., allows to wait till shards are allocated. Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

-shay.banon







On Tuesday, March 15, 2011 at 5:32 PM, Clinton Gormley wrote:
Hi Zohar

Is the index created & ready when "actionGet" method returns?

As I understand it (and I stand under correction), a create index call
is async, so it is not ready when actionGet returns.

Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

Not a coincidence. Using a wait_for_status call is the correct thing to
do - exactly what it is designed for.

clint


Reply | Threaded
Open this post in threaded view
|

Re: Can I clarify how admin api works for creating an index?

mooky
Ah, yep, true.

On 15 March 2011 19:28, Shay Banon <[hidden email]> wrote:
> Also, you might actually want to get search response when not all shards are
> available.
Reply | Threaded
Open this post in threaded view
|

Re: Can I clarify how admin api works for creating an index?

mooky
In reply to this post by kimchy
Fair enough. Thanks.

Yeah, I think in our situation it showed up as a test that kept
failing intermittently.... just a timing thing (initially it was
hanging)

-Nick

On 15 March 2011 19:25, Shay Banon <[hidden email]> wrote:

> No specific reason except for the complexity involved in implementing it,
> and the fact that it mainly when doing testing.
>
> On Tuesday, March 15, 2011 at 7:46 PM, Mooky wrote:
>
> The create index API makes sure that the index provided is correct
> (analyzers are found, mappings are correct), waits for the *index* itself to
> be created on the relevant nodes, but does not wait for the shards to be
> allocated.
>
> What's the rationale for not waiting for the shards to be allocated?
> (just for my education/understanding)
>
> Some APIs wait (till a timeout) if a shard is not allocated yet (like the
> index API), some don't, like search.
>
> Again, just for my education/understanding, why doesn't the search
> wait?
>
> -Nick
>
>
>
> On Mar 15, 4:02 pm, Shay Banon <[hidden email]> wrote:
>
> There are three different things associated with creating an index. The
> first is adding it to the cluster metadata, then creating it on each
> relevant node, and finally, allocating shards to nodes that resulted from
> the index creation.
>
> The create index API makes sure that the index provided is correct
> (analyzers are found, mappings are correct), waits for the *index* itself to
> be created on the relevant nodes, but does not wait for the shards to be
> allocated.
>
> The cluster health API, with the wait_for..., allows to wait till shards are
> allocated. Some APIs wait (till a timeout) if a shard is not allocated yet
> (like the index API), some don't, like search.
>
> -shay.banon
>
>
>
>
>
>
>
> On Tuesday, March 15, 2011 at 5:32 PM, Clinton Gormley wrote:
>
> Hi Zohar
>
> Is the index created & ready when "actionGet" method returns?
>
> As I understand it (and I stand under correction), a create index call
> is async, so it is not ready when actionGet returns.
>
> Out of curiosity, we also added the following code, after creating the
> index.
> This makes it happy - but not sure if thats just because the call
> introduces sufficient delay - or whether its what we need to do.
>
> Not a coincidence. Using a wait_for_status call is the correct thing to
> do - exactly what it is designed for.
>
> clint
>
>