Feeds/Shard based on machines hardware capacity

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Feeds/Shard based on machines hardware capacity

vineeth mohan
Hi ,

Is there any provision in elasticsearch where it automatically decides on increasing or decreasing percentage of feed in a particular shard due to its hardware capability.
I would love to see a feature where it allocates more feeds to a shard whose machine has bigger capability and lesser feeds whose allocated machines have lesser hardware capability.

It would be even great if i can tell manually which all shards should have what percentage of total feeds.
Is there any configuration which will help in any of the above.

Thanks
           Vineeth

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Feeds/Shard based on machines hardware capacity

Alexander Reelsen-2
Hey,

have you taken a look at shard allocation? 

It's not automatic (there is no reliable mechanism deciding for elasticsearch that this node is extraordinary powerful. Judging from its hardware might be a mistake as many other services might run on the node, you have to jump in here) mechanism, but you can do it by grouping nodes for example.


--Alex


On Thu, May 30, 2013 at 7:09 AM, Vineeth Mohan <[hidden email]> wrote:
Hi ,

Is there any provision in elasticsearch where it automatically decides on increasing or decreasing percentage of feed in a particular shard due to its hardware capability.
I would love to see a feature where it allocates more feeds to a shard whose machine has bigger capability and lesser feeds whose allocated machines have lesser hardware capability.

It would be even great if i can tell manually which all shards should have what percentage of total feeds.
Is there any configuration which will help in any of the above.

Thanks
           Vineeth

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Feeds/Shard based on machines hardware capacity

vineeth mohan
Hello Alex ,

Using shard allocation , i might be able to move a shard from a week machine to a string machine.
But this wont fully exploit the capability of the entire network.

What i am looking for is to do the following -
                             Suppose there are 4 shards on 4 different machines. I want ES to put 40% of the total data in shard1 , 10% in shard2 , 30% in shard3 and 20% in shard4.

Is this possible in ES ?
If not , wont it be a good idea to post a feature requirement based on this.

Thanks
              Vineeth

On Thu, May 30, 2013 at 12:25 PM, Alexander Reelsen <[hidden email]> wrote:
Hey,

have you taken a look at shard allocation? 

It's not automatic (there is no reliable mechanism deciding for elasticsearch that this node is extraordinary powerful. Judging from its hardware might be a mistake as many other services might run on the node, you have to jump in here) mechanism, but you can do it by grouping nodes for example.


--Alex


On Thu, May 30, 2013 at 7:09 AM, Vineeth Mohan <[hidden email]> wrote:
Hi ,

Is there any provision in elasticsearch where it automatically decides on increasing or decreasing percentage of feed in a particular shard due to its hardware capability.
I would love to see a feature where it allocates more feeds to a shard whose machine has bigger capability and lesser feeds whose allocated machines have lesser hardware capability.

It would be even great if i can tell manually which all shards should have what percentage of total feeds.
Is there any configuration which will help in any of the above.

Thanks

           Vineeth

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Feeds/Shard based on machines hardware capacity

dadoonet
I don't think it's doable easily.
I mean that you will need to control that on your end and change a little bit the routing feature.

Because, when you want to do a GET, ES is going directly to the right shard using this routing.
How to find the relevant shard if it's not based on id or routing key?

-- 
David Pilato | Technical Advocate | Elasticsearch.com



Le 30 mai 2013 à 11:16, Vineeth Mohan <[hidden email]> a écrit :

Hello Alex ,

Using shard allocation , i might be able to move a shard from a week machine to a string machine.
But this wont fully exploit the capability of the entire network.

What i am looking for is to do the following -
                             Suppose there are 4 shards on 4 different machines. I want ES to put 40% of the total data in shard1 , 10% in shard2 , 30% in shard3 and 20% in shard4.

Is this possible in ES ?
If not , wont it be a good idea to post a feature requirement based on this.

Thanks
              Vineeth

On Thu, May 30, 2013 at 12:25 PM, Alexander Reelsen <[hidden email]> wrote:
Hey,

have you taken a look at shard allocation? 

It's not automatic (there is no reliable mechanism deciding for elasticsearch that this node is extraordinary powerful. Judging from its hardware might be a mistake as many other services might run on the node, you have to jump in here) mechanism, but you can do it by grouping nodes for example.


--Alex


On Thu, May 30, 2013 at 7:09 AM, Vineeth Mohan <[hidden email]> wrote:
Hi ,

Is there any provision in elasticsearch where it automatically decides on increasing or decreasing percentage of feed in a particular shard due to its hardware capability.
I would love to see a feature where it allocates more feeds to a shard whose machine has bigger capability and lesser feeds whose allocated machines have lesser hardware capability.

It would be even great if i can tell manually which all shards should have what percentage of total feeds.
Is there any configuration which will help in any of the above.

Thanks

           Vineeth

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 


smime.p7s (1K) Download Attachment