Need help with shard balancing

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Need help with shard balancing

Craig Brown-2
We have a 4-node cluster with 2 indexes each with 1 replica. We are currently indexing all documents against a single index, while the other one is empty. One of the nodes it doing very little work because it has mostly empty shards.

I need a way to have ES distribute the shards from each index evenly across all 4 nodes. I know by default, ES considers all shards equally. I've looked into some of the ES config settings around shard balancing, but it's not obvious how to make this happen.

Any pointers appreciated, thanks!

  -Craig

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Need help with shard balancing

Clinton Gormley-2
Hi Craig

On Wed, 2013-02-06 at 08:26 -0800, Craig Brown wrote:
> We have a 4-node cluster with 2 indexes each with 1 replica. We are
> currently indexing all documents against a single index, while the
> other one is empty. One of the nodes it doing very little work because
> it has mostly empty shards.
>
> I need a way to have ES distribute the shards from each index evenly
> across all 4 nodes. I know by default, ES considers all shards
> equally. I've looked into some of the ES config settings around shard
> balancing, but it's not obvious how to make this happen.

If you're using v0.19.10 or higher, then you can use the cluster reroute
API to move your shards around:

http://www.elasticsearch.org/guide/reference/api/admin-cluster-reroute.html

If not, and you're not using the other index, try just closing and
reopening it.  If that doesn't work, you may have to delete it

clint

>
> Any pointers appreciated, thanks!
>
>   -Craig
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Need help with shard balancing

ppearcy
The best way to do this is via index.routing.allocation.total_shards_per_node:
http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html

I have a process that looks at the number of data nodes I have and the number of shards for an index and automatically sets this to force a fairer balancing. Would be a nice feature to have in the core elasticsearch, though.

Best Regards,
Paul

On Wednesday, February 6, 2013 9:45:02 AM UTC-7, Clinton Gormley wrote:
Hi Craig

On Wed, 2013-02-06 at 08:26 -0800, Craig Brown wrote:
> We have a 4-node cluster with 2 indexes each with 1 replica. We are
> currently indexing all documents against a single index, while the
> other one is empty. One of the nodes it doing very little work because
> it has mostly empty shards.
>
> I need a way to have ES distribute the shards from each index evenly
> across all 4 nodes. I know by default, ES considers all shards
> equally. I've looked into some of the ES config settings around shard
> balancing, but it's not obvious how to make this happen.

If you're using v0.19.10 or higher, then you can use the cluster reroute
API to move your shards around:

http://www.elasticsearch.org/guide/reference/api/admin-cluster-reroute.html

If not, and you're not using the other index, try just closing and
reopening it.  If that doesn't work, you may have to delete it

clint

>
> Any pointers appreciated, thanks!
>
>   -Craig
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="4gFN4xhvrigJ">elasticsearc...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Need help with shard balancing

Craig Brown
Thanks. Yes, its a 19.9 cluster.

  - Craig


On Wed, Feb 6, 2013 at 10:03 AM, ppearcy <[hidden email]> wrote:
The best way to do this is via index.routing.allocation.total_shards_per_node:

I have a process that looks at the number of data nodes I have and the number of shards for an index and automatically sets this to force a fairer balancing. Would be a nice feature to have in the core elasticsearch, though.

Best Regards,
Paul


On Wednesday, February 6, 2013 9:45:02 AM UTC-7, Clinton Gormley wrote:
Hi Craig

On Wed, 2013-02-06 at 08:26 -0800, Craig Brown wrote:
> We have a 4-node cluster with 2 indexes each with 1 replica. We are
> currently indexing all documents against a single index, while the
> other one is empty. One of the nodes it doing very little work because
> it has mostly empty shards.
>
> I need a way to have ES distribute the shards from each index evenly
> across all 4 nodes. I know by default, ES considers all shards
> equally. I've looked into some of the ES config settings around shard
> balancing, but it's not obvious how to make this happen.

If you're using v0.19.10 or higher, then you can use the cluster reroute
API to move your shards around:

http://www.elasticsearch.org/guide/reference/api/admin-cluster-reroute.html

If not, and you're not using the other index, try just closing and
reopening it.  If that doesn't work, you may have to delete it

clint

>
> Any pointers appreciated, thanks!
>
>   -Craig
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearc...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
… 
CRAIG BROWN
Search Architect
youwho, Inc.

www.youwho.com

 M: 801.680.7300

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.