Number of shards in 4 node Cluster

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Number of shards in 4 node Cluster

John S
Hi All,

Is there any best practices of having on the number of shards for a cluster? I have a 4 node cluster and used shards of 20.

During any node failure or other events i doubts since the shards number is high, replication to new node is taking more time...

Is there any metrics or formula to be done for number or shards?

Regards
John

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6e51f1e4-8938-4196-84a9-007705869b6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Number of shards in 4 node Cluster

Mark Walkom-2
What sort of data do you have, time based or static? If it's the former then going with any arbitrary number is less of a problem as you can change this the next roll over period. If it's static then 4 would be a good start.

There aren't any metrics around this, other than *not* creating a large number to start with, as each shard is a lucene instance and does take resources.

On 17 March 2015 at 11:00, John S <[hidden email]> wrote:
Hi All,

Is there any best practices of having on the number of shards for a cluster? I have a 4 node cluster and used shards of 20.

During any node failure or other events i doubts since the shards number is high, replication to new node is taking more time...

Is there any metrics or formula to be done for number or shards?

Regards
John

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6e51f1e4-8938-4196-84a9-007705869b6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_B9mxm9xnJtzoSc-tj1G-MoZ7vdQ-ye%2B7woLfj7aRHJw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Number of shards in 4 node Cluster

Andrew Selden-2
In reply to this post by John S
I typically suggest to start with the default of 5 shards. A single shard can hold several tens of gigabytes. Certainly in your case it seems like 20 shards is overkill for a 4 node cluster. 


On Mar 17, 2015, at 11:00 AM, John S <[hidden email]> wrote:

Hi All,

Is there any best practices of having on the number of shards for a cluster? I have a 4 node cluster and used shards of 20.

During any node failure or other events i doubts since the shards number is high, replication to new node is taking more time...

Is there any metrics or formula to be done for number or shards?

Regards
John

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6e51f1e4-8938-4196-84a9-007705869b6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/DD2AA858-ABD4-49F5-9F9C-D73C01F615CE%40elastic.co.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Number of shards in 4 node Cluster

Georgi Ivanov
In reply to this post by John S
My rules is : 1 primary shard per server.

Also make some estimation how big will be the single index/shard 

I think it is not good if single shard exceed 10 GB, although there is no exact limit.


Georgi 

On Tuesday, March 17, 2015 at 7:00:23 PM UTC+1, John S wrote:
Hi All,

Is there any best practices of having on the number of shards for a cluster? I have a 4 node cluster and used shards of 20.

During any node failure or other events i doubts since the shards number is high, replication to new node is taking more time...

Is there any metrics or formula to be done for number or shards?

Regards
John

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Number of shards in 4 node Cluster

Mark Walkom-2
We recommend shards no larger than 50GB, but as you mention there is no exact limit.

On 18 March 2015 at 04:09, Georgi Ivanov <[hidden email]> wrote:
My rules is : 1 primary shard per server.

Also make some estimation how big will be the single index/shard 

I think it is not good if single shard exceed 10 GB, although there is no exact limit.


Georgi 

On Tuesday, March 17, 2015 at 7:00:23 PM UTC+1, John S wrote:
Hi All,

Is there any best practices of having on the number of shards for a cluster? I have a 4 node cluster and used shards of 20.

During any node failure or other events i doubts since the shards number is high, replication to new node is taking more time...

Is there any metrics or formula to be done for number or shards?

Regards
John

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-iUMdUXTS31uHuJ8FyXJy7vNqCrP_gw0tSs1xNFkzz%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Number of shards in 4 node Cluster

Andrew Selden-2
In reply to this post by Georgi Ivanov
As Mark said, there is no hard limit on how big a single shard can be, but just so it’s clear, 10GB is actually quite small for a single shard. It’s not at all uncommon for me to see shards with upwards of 60 GB or more. 


On Mar 18, 2015, at 4:09 AM, Georgi Ivanov <[hidden email]> wrote:

My rules is : 1 primary shard per server.

Also make some estimation how big will be the single index/shard 

I think it is not good if single shard exceed 10 GB, although there is no exact limit.


Georgi 

On Tuesday, March 17, 2015 at 7:00:23 PM UTC+1, John S wrote:
Hi All,

Is there any best practices of having on the number of shards for a cluster? I have a 4 node cluster and used shards of 20.

During any node failure or other events i doubts since the shards number is high, replication to new node is taking more time...

Is there any metrics or formula to be done for number or shards?

Regards
John

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/39389779-986A-4933-89BD-7B842B168EA0%40elastic.co.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Number of shards in 4 node Cluster

Andrej
In reply to this post by Mark Walkom-2
Hi Mark,

may I ask what the reason for this recommendation is?

Thanks,
Andrej

Am Mittwoch, 18. März 2015 17:50:09 UTC+1 schrieb Mark Walkom:
We recommend shards no larger than 50GB, but as you mention there is no exact limit.

On 18 March 2015 at 04:09, Georgi Ivanov <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="nBq59YXwynQJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">georgi....@...> wrote:
My rules is : 1 primary shard per server.

Also make some estimation how big will be the single index/shard 

I think it is not good if single shard exceed 10 GB, although there is no exact limit.


Georgi 

On Tuesday, March 17, 2015 at 7:00:23 PM UTC+1, John S wrote:
Hi All,

Is there any best practices of having on the number of shards for a cluster? I have a 4 node cluster and used shards of 20.

During any node failure or other events i doubts since the shards number is high, replication to new node is taking more time...

Is there any metrics or formula to be done for number or shards?

Regards
John

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="nBq59YXwynQJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/385c959b-0132-4d9d-95cd-b4b08fca9c94%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Number of shards in 4 node Cluster

Mark Walkom-2
Part of it is based on knowledge picked up from our customers, part of it is that once you have to start shifting files larger than this around (during reallocation or recovery) it can take excessive time.

There is also a ~2 billion hard limit for documents in a single shard, which is a lucene limit, so you reduce your exposure there too.

On 18 March 2015 at 10:29, Andrej Rosenheinrich <[hidden email]> wrote:
Hi Mark,

may I ask what the reason for this recommendation is?

Thanks,
Andrej

Am Mittwoch, 18. März 2015 17:50:09 UTC+1 schrieb Mark Walkom:
We recommend shards no larger than 50GB, but as you mention there is no exact limit.

On 18 March 2015 at 04:09, Georgi Ivanov <[hidden email]> wrote:
My rules is : 1 primary shard per server.

Also make some estimation how big will be the single index/shard 

I think it is not good if single shard exceed 10 GB, although there is no exact limit.


Georgi 

On Tuesday, March 17, 2015 at 7:00:23 PM UTC+1, John S wrote:
Hi All,

Is there any best practices of having on the number of shards for a cluster? I have a 4 node cluster and used shards of 20.

During any node failure or other events i doubts since the shards number is high, replication to new node is taking more time...

Is there any metrics or formula to be done for number or shards?

Regards
John

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/385c959b-0132-4d9d-95cd-b4b08fca9c94%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8ajDqiyn04JXqH9RcBU8vaOLKZkEdFRYnNWFiV0ihw9Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.