index partition strategy

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

index partition strategy

Valerio
Hello,
can you provide some technical details about how indexes are
partitioned on nodes ?

Thanks,
Valerio
Reply | Threaded
Open this post in threaded view
|

Re: index partition strategy

kimchy
Administrator
Sure. Each index is broken down into shards, each shard can have 0 or more replicas. Each index can have a different number of shards (and replicas) exposed through the create index API. When a node comes up, shards are allocated to it either by relocating them from existing nodes, or simply creating them if they were not previously allocated. It is ensured that a specific shard and its replica will not be created on the same node (later versions will provide more control to not allocated them on the same rack and so on).

In terms of clustering model, there is a single master in the cluster (automatically elected), which controls the shards allocation.

Is this what you were after?
-shay.banon

On Thu, Mar 4, 2010 at 2:31 PM, Valerio Schiavoni <[hidden email]> wrote:
Hello,
can you provide some technical details about how indexes are
partitioned on nodes ?

Thanks,
Valerio

Reply | Threaded
Open this post in threaded view
|

Re: index partition strategy

Valerio
Hello Shay,
thanks for the quick reply, which got me even more interested.

I am particularly interested in the allocation strategy of shrads. 

Which is the strategy to choose a node responsible for a shard ?
How shrad replicas are placed ? 
Do you react in case the number of replicas for a given shrad fall 

thanks,
valerio

On Thu, Mar 4, 2010 at 1:59 PM, Shay Banon <[hidden email]> wrote:
Sure. Each index is broken down into shards, each shard can have 0 or more replicas. Each index can have a different number of shards (and replicas) exposed through the create index API. When a node comes up, shards are allocated to it either by relocating them from existing nodes, or simply creating them if they were not previously allocated. It is ensured that a specific shard and its replica will not be created on the same node (later versions will provide more control to not allocated them on the same rack and so on).

In terms of clustering model, there is a single master in the cluster (automatically elected), which controls the shards allocation.

Reply | Threaded
Open this post in threaded view
|

Re: index partition strategy

kimchy
Administrator
Hey,

   Shards are allocated in an even manner between nodes. Specific shard replicas are only placed on nodes that have no other instance of that shard running. If a node fails, then its is detected and the shards allocated on it are reallocated to other nodes.

-shay.banon

On Thu, Mar 4, 2010 at 3:47 PM, Valerio Schiavoni <[hidden email]> wrote:
Hello Shay,
thanks for the quick reply, which got me even more interested.

I am particularly interested in the allocation strategy of shrads. 

Which is the strategy to choose a node responsible for a shard ?
How shrad replicas are placed ? 
Do you react in case the number of replicas for a given shrad fall 

thanks,
valerio


On Thu, Mar 4, 2010 at 1:59 PM, Shay Banon <[hidden email]> wrote:
Sure. Each index is broken down into shards, each shard can have 0 or more replicas. Each index can have a different number of shards (and replicas) exposed through the create index API. When a node comes up, shards are allocated to it either by relocating them from existing nodes, or simply creating them if they were not previously allocated. It is ensured that a specific shard and its replica will not be created on the same node (later versions will provide more control to not allocated them on the same rack and so on).

In terms of clustering model, there is a single master in the cluster (automatically elected), which controls the shards allocation.