Rebalancing a cluster

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Rebalancing a cluster

James Bardin
I have a cluster with one index which is much more heavily used than the rest. My problem is keeping the busy index distributed across all nodes. For example, do to recent changes in the the cluster, I now have 1 node with 80 percent of the hot primary shards, and two nodes doing almost nothing.

I been using "i.r.a.include.name" to force elasticsearch to move shards around, but since it always wants to evenly distribute the number of shards, the load often ends up skewed. This can work, but it take a few pushes to get the shards to fall nicely, and isn't a maintainable solution (also if anyone knows how to *unset* an index setting, I'd like to get rid of the "index.routing.allocation" settings when I'm done, so they don't need to be updated if the cluster gets resized).

So, is there a better way to balance a cluster?
Should I just double or tripple the shard count, so they're more likely to get evenly distributed?

Thanks,
-jim

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Rebalancing a cluster

Mohammady Mahdy
Did you find any solution for this?

On Thursday, August 23, 2012 1:21:03 AM UTC+4, James Bardin wrote:
I have a cluster with one index which is much more heavily used than the rest. My problem is keeping the busy index distributed across all nodes. For example, do to recent changes in the the cluster, I now have 1 node with 80 percent of the hot primary shards, and two nodes doing almost nothing.

I been using "i.r.a.include.name" to force elasticsearch to move shards around, but since it always wants to evenly distribute the number of shards, the load often ends up skewed. This can work, but it take a few pushes to get the shards to fall nicely, and isn't a maintainable solution (also if anyone knows how to *unset* an index setting, I'd like to get rid of the "index.routing.allocation" settings when I'm done, so they don't need to be updated if the cluster gets resized).

So, is there a better way to balance a cluster?
Should I just double or tripple the shard count, so they're more likely to get evenly distributed?

Thanks,
-jim

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Rebalancing a cluster

ferhatsb
In reply to this post by James Bardin
Hi,

Maybe you can set "index.routing.allocation.total_shards_per_node" for that nasty index.
http://www.elasticsearch.org/guide/reference/index-modules/allocation.html

Regards,
Ferhat

On Thursday, August 23, 2012 12:21:03 AM UTC+3, James Bardin wrote:
I have a cluster with one index which is much more heavily used than the rest. My problem is keeping the busy index distributed across all nodes. For example, do to recent changes in the the cluster, I now have 1 node with 80 percent of the hot primary shards, and two nodes doing almost nothing.

I been using "i.r.a.include.name" to force elasticsearch to move shards around, but since it always wants to evenly distribute the number of shards, the load often ends up skewed. This can work, but it take a few pushes to get the shards to fall nicely, and isn't a maintainable solution (also if anyone knows how to *unset* an index setting, I'd like to get rid of the "index.routing.allocation" settings when I'm done, so they don't need to be updated if the cluster gets resized).

So, is there a better way to balance a cluster?
Should I just double or tripple the shard count, so they're more likely to get evenly distributed?

Thanks,
-jim

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Rebalancing a cluster

Mohammady Mahdy
Oh, very nice.

On Wed, Jan 16, 2013 at 5:56 PM, ferhatsb <[hidden email]> wrote:
Hi,

Maybe you can set "index.routing.allocation.total_shards_per_node" for that nasty index.
http://www.elasticsearch.org/guide/reference/index-modules/allocation.html

Regards,
Ferhat

On Thursday, August 23, 2012 12:21:03 AM UTC+3, James Bardin wrote:
I have a cluster with one index which is much more heavily used than the rest. My problem is keeping the busy index distributed across all nodes. For example, do to recent changes in the the cluster, I now have 1 node with 80 percent of the hot primary shards, and two nodes doing almost nothing.

I been using "i.r.a.include.name" to force elasticsearch to move shards around, but since it always wants to evenly distribute the number of shards, the load often ends up skewed. This can work, but it take a few pushes to get the shards to fall nicely, and isn't a maintainable solution (also if anyone knows how to *unset* an index setting, I'd like to get rid of the "index.routing.allocation" settings when I'm done, so they don't need to be updated if the cluster gets resized).

So, is there a better way to balance a cluster?
Should I just double or tripple the shard count, so they're more likely to get evenly distributed?

Thanks,
-jim

--
 
 

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Rebalancing a cluster

James Bardin
In reply to this post by ferhatsb

On Wed, Jan 16, 2013 at 8:56 AM, ferhatsb <[hidden email]> wrote:
Maybe you can set "index.routing.allocation.total_shards_per_node" for that nasty index.
http://www.elasticsearch.org/guide/reference/index-modules/allocation.html

I wish the elasticsearch docs tagged features in the docs with their release. I don't think this was in the stable version when I first asked, but I'm not sure now.

I ended up having to schedule a full cluster shutdown to clear this up. 
Also, last time I checked, there still is no way to remove index settings without restarting, but many of them now treat an empty string as if it were unset.

-jim

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Rebalancing a cluster

ppearcy
If you're not sure, search google/github will typically bring you to the issue that added the feature:
https://github.com/elasticsearch/elasticsearch/issues/1650

Not ideal, but it works. 

On Wednesday, January 16, 2013 9:19:47 AM UTC-7, James Bardin wrote:

On Wed, Jan 16, 2013 at 8:56 AM, ferhatsb <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="OfVE0xh6SmQJ">ferhat...@...> wrote:
Maybe you can set "index.routing.allocation.total_shards_per_node" for that nasty index.
http://www.elasticsearch.org/guide/reference/index-modules/allocation.html

I wish the elasticsearch docs tagged features in the docs with their release. I don't think this was in the stable version when I first asked, but I'm not sure now.

I ended up having to schedule a full cluster shutdown to clear this up. 
Also, last time I checked, there still is no way to remove index settings without restarting, but many of them now treat an empty string as if it were unset.

-jim

--