Changing the number of shards and replicas of an existing index

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Changing the number of shards and replicas of an existing index

Bruno Dumon
Hi,

Do you have any plans or thoughts on allowing to change
(increase/decrease) the number of shards and replicas of an existing
index? It seems like currently these can only be specified at index
creation time.

--
Bruno Dumon
Outerthought
http://outerthought.org/
Reply | Threaded
Open this post in threaded view
|

Re: Changing the number of shards and replicas of an existing index

kimchy
Administrator
Yes, certainly, though the effort involved in supporting it is quite big (its quite complex when putting search engine into the picture, compared with the very simple models like dynamo, for example).

But, most cases you won't necessarily need it because of the "multi-index" support elasticsearch provides. For example, lets take a twitter example. We can decide to create an index per user, or create an index per day. The ability to perform *all* operations in elasticsearch across more than one index is intentional to support that.

This gives you added benefit. For example, if you have an index per user in the twitter example, you can search on your friends and their friends (and *score* differently based on the relationship you have with them) quite easily.

-shay.banon

On Wed, Feb 24, 2010 at 5:38 PM, Bruno Dumon <[hidden email]> wrote:
Hi,

Do you have any plans or thoughts on allowing to change
(increase/decrease) the number of shards and replicas of an existing
index? It seems like currently these can only be specified at index
creation time.

--
Bruno Dumon
Outerthought
http://outerthought.org/

Reply | Threaded
Open this post in threaded view
|

Re: Changing the number of shards and replicas of an existing index

kimchy
Administrator
One more thing I forgot to mention. Even with one index, and the out of the box default of 5 shards with 1 replica each, you can scale easily up to 10 machines (just for the index, of course you can create more indices and scale even more) which gives you hugh amount of storage (10s of terra) and great search scalability. Also, changing the number of replicas dynamically should be much simpler. If search scalability is what you are after.

-shay.banon

On Wed, Feb 24, 2010 at 8:27 PM, Shay Banon <[hidden email]> wrote:
Yes, certainly, though the effort involved in supporting it is quite big (its quite complex when putting search engine into the picture, compared with the very simple models like dynamo, for example).

But, most cases you won't necessarily need it because of the "multi-index" support elasticsearch provides. For example, lets take a twitter example. We can decide to create an index per user, or create an index per day. The ability to perform *all* operations in elasticsearch across more than one index is intentional to support that.

This gives you added benefit. For example, if you have an index per user in the twitter example, you can search on your friends and their friends (and *score* differently based on the relationship you have with them) quite easily.

-shay.banon


On Wed, Feb 24, 2010 at 5:38 PM, Bruno Dumon <[hidden email]> wrote:
Hi,

Do you have any plans or thoughts on allowing to change
(increase/decrease) the number of shards and replicas of an existing
index? It seems like currently these can only be specified at index
creation time.

--
Bruno Dumon
Outerthought
http://outerthought.org/


Reply | Threaded
Open this post in threaded view
|

Re: Changing the number of shards and replicas of an existing index

Bruno Dumon
Thanks for the clarification.

Using multiple indices a a way to scale is an interesting thought, I
had not considered it.

Changing the number of shards probably needs less flexibility than
changing the number of replicas, though it would be nice if there was
some way to grow, even if it would be just by doubling the number of
current shards (i.e. splitting each shard into two).

But anyway, the current flexibility and power of ElasticSearch is
already quite impressive!

On Wed, Feb 24, 2010 at 7:37 PM, Shay Banon
<[hidden email]> wrote:

> One more thing I forgot to mention. Even with one index, and the out of the
> box default of 5 shards with 1 replica each, you can scale easily up to 10
> machines (just for the index, of course you can create more indices and
> scale even more) which gives you hugh amount of storage (10s of terra) and
> great search scalability. Also, changing the number of replicas dynamically
> should be much simpler. If search scalability is what you are after.
> -shay.banon
>
> On Wed, Feb 24, 2010 at 8:27 PM, Shay Banon <[hidden email]>
> wrote:
>>
>> Yes, certainly, though the effort involved in supporting it is quite big
>> (its quite complex when putting search engine into the picture, compared
>> with the very simple models like dynamo, for example).
>> But, most cases you won't necessarily need it because of the "multi-index"
>> support elasticsearch provides. For example, lets take a twitter example. We
>> can decide to create an index per user, or create an index per day. The
>> ability to perform *all* operations in elasticsearch across more than one
>> index is intentional to support that.
>> This gives you added benefit. For example, if you have an index per user
>> in the twitter example, you can search on your friends and their friends
>> (and *score* differently based on the relationship you have with them) quite
>> easily.
>> -shay.banon
>>
>> On Wed, Feb 24, 2010 at 5:38 PM, Bruno Dumon <[hidden email]>
>> wrote:
>>>
>>> Hi,
>>>
>>> Do you have any plans or thoughts on allowing to change
>>> (increase/decrease) the number of shards and replicas of an existing
>>> index? It seems like currently these can only be specified at index
>>> creation time.
>>>

--
Bruno Dumon
Outerthought
http://outerthought.org/