>30s to create an index

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

>30s to create an index

Filirom1
Hi guys, 

On a 4 nodes cluster a new index creation takes more than 30 seconds. 
Is it something usual ?

$ time curl -XPUT http://localhost:9023/test/user/kimchy -d '{
  "name" : "Shay Banon"
}'
{"ok":true,"_index":"test","_type":"user","_id":"kimchy","_version":1}
real 0m32.836s
user 0m0.002s
sys 0m0.005s

I enabled the cluster.service logs to see what happens. I put the logs on a gist https://gist.github.com/Filirom1/4754342

Cheers
Romain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: >30s to create an index

Drew Raines-2
Filirom1 wrote:

> On a 4 nodes cluster a new index creation takes more than 30 seconds.
> Is it something usual ?
>
> $ time curl -XPUT http://localhost:9023/test/user/kimchy -d '{
>   "name" : "Shay Banon"
> }'
> {"ok":true,"_index":"test","_type":"user","_id":"kimchy","_version":1}
> real 0m32.836s
> user 0m0.002s
> sys 0m0.005s
>
> I enabled the cluster.service logs to see what happens. I put the logs on a
> gist https://gist.github.com/Filirom1/4754342

It looks like you've created, or are creating, a bunch of unique
indices and types as you index.  Can we see the indexing code?
Perhaps you're doing something inefficiently we can help you think
through.

-Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: >30s to create an index

Filirom1
When I did the curl command to create a new index, the cluster was doing nothing else.
The indexing code is the curl command.

This 4 nodes cluster is not used in production. It just for test purpose, so I can try what you want on it.


2013/2/11 Drew Raines <[hidden email]>
Filirom1 wrote:

> On a 4 nodes cluster a new index creation takes more than 30 seconds.
> Is it something usual ?
>
> $ time curl -XPUT http://localhost:9023/test/user/kimchy -d '{
>   "name" : "Shay Banon"
> }'
> {"ok":true,"_index":"test","_type":"user","_id":"kimchy","_version":1}
> real 0m32.836s
> user 0m0.002s
> sys 0m0.005s
>
> I enabled the cluster.service logs to see what happens. I put the logs on a
> gist https://gist.github.com/Filirom1/4754342

It looks like you've created, or are creating, a bunch of unique
indices and types as you index.  Can we see the indexing code?
Perhaps you're doing something inefficiently we can help you think
through.

-Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: >30s to create an index

egaumer
In reply to this post by Filirom1
Are there any other indices in the cluster? Types? 

It looks like you have a lot of mappings that are continuously getting refreshed.



On Monday, February 11, 2013 8:46:03 AM UTC-5, Filirom1 wrote:
Hi guys, 

On a 4 nodes cluster a new index creation takes more than 30 seconds. 
Is it something usual ?

  "name" : "Shay Banon"
}'
{"ok":true,"_index":"test","_type":"user","_id":"kimchy","_version":1}
real 0m32.836s
user 0m0.002s
sys 0m0.005s

I enabled the cluster.service logs to see what happens. I put the logs on a gist https://gist.github.com/Filirom1/4754342

Cheers
Romain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: >30s to create an index

Filirom1
There are 51 indices in the cluster, and for each indices there is 6 shards and 1 replica.

Here is the number of types per index : 
[ 6,
  9,
  6,
  1,
  15,
  2,
  3,
  31,
  1,
  3,
  4,
  2,
  0,
  3,
  5,
  5,
  86,
  4,
  1,
  50,
  1,
  0,
  1,
  3,
  21,
  3,
  8,
  51,
  6,
  3,
  2,
  27,
  2,
  318,
  3,
  711,
  2,
  4,
  21,
  11,
  3,
  2,
  2,
  4,
  5,
  3,
  1,
  3,
  8,
  9,
  1 ]

The download the _mapping, it weights 5.2Mo




2013/2/11 egaumer <[hidden email]>
Are there any other indices in the cluster? Types? 

It looks like you have a lot of mappings that are continuously getting refreshed.



On Monday, February 11, 2013 8:46:03 AM UTC-5, Filirom1 wrote:
Hi guys, 

On a 4 nodes cluster a new index creation takes more than 30 seconds. 
Is it something usual ?

  "name" : "Shay Banon"
}'
{"ok":true,"_index":"test","_type":"user","_id":"kimchy","_version":1}
real 0m32.836s
user 0m0.002s
sys 0m0.005s

I enabled the cluster.service logs to see what happens. I put the logs on a gist https://gist.github.com/Filirom1/4754342

Cheers
Romain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: >30s to create an index

joergprante@gmail.com
As said, you created many indices, each with many types, with many
shards, and even more dynamic fields that are refreshing dynamically.

Remember: each new field mapping is a cluster upgrade. Such an operation
is costly as it propagates throughout the whole cluster.

30 seconds are very fast according to the some thousands of cluster
state field mapping upgrade operations you have provoked.

The index creation time scales over the number of nodes, indices, types,
shards, and fields. Use less and you will observe faster creation times.

Best regards,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: >30s to create an index

Filirom1
Thanks a lot for the explanation !


2013/2/12 Jörg Prante <[hidden email]>
As said, you created many indices, each with many types, with many shards, and even more dynamic fields that are refreshing dynamically.

Remember: each new field mapping is a cluster upgrade. Such an operation is costly as it propagates throughout the whole cluster.

30 seconds are very fast according to the some thousands of cluster state field mapping upgrade operations you have provoked.

The index creation time scales over the number of nodes, indices, types, shards, and fields. Use less and you will observe faster creation times.

Best regards,

Jörg


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: >30s to create an index

kimchy
Administrator
I would still love to see the cluster state if you provide, would like to simulate it on our end and see if we can improve things. Can you gist the result of /_cluster/state on the cluster with all the indices?

On Feb 12, 2013, at 2:02 PM, Romain <[hidden email]> wrote:

Thanks a lot for the explanation !


2013/2/12 Jörg Prante <[hidden email]>
As said, you created many indices, each with many types, with many shards, and even more dynamic fields that are refreshing dynamically.

Remember: each new field mapping is a cluster upgrade. Such an operation is costly as it propagates throughout the whole cluster.

30 seconds are very fast according to the some thousands of cluster state field mapping upgrade operations you have provoked.

The index creation time scales over the number of nodes, indices, types, shards, and fields. Use less and you will observe faster creation times.

Best regards,

Jörg


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: >30s to create an index

Filirom1
I would prefer not to publish the mapping on internet. I sent it to you by email.

Thank you
Romain


2013/2/12 <[hidden email]>
I would still love to see the cluster state if you provide, would like to simulate it on our end and see if we can improve things. Can you gist the result of /_cluster/state on the cluster with all the indices?

On Feb 12, 2013, at 2:02 PM, Romain <[hidden email]> wrote:

Thanks a lot for the explanation !


2013/2/12 Jörg Prante <[hidden email]>
As said, you created many indices, each with many types, with many shards, and even more dynamic fields that are refreshing dynamically.

Remember: each new field mapping is a cluster upgrade. Such an operation is costly as it propagates throughout the whole cluster.

30 seconds are very fast according to the some thousands of cluster state field mapping upgrade operations you have provoked.

The index creation time scales over the number of nodes, indices, types, shards, and fields. Use less and you will observe faster creation times.

Best regards,

Jörg


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.