How to migrate index without disturbing client?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to migrate index without disturbing client?

mehmet
Hello,

I am running a cluster which contains many indexes. In the near future I will have new machines so that I will need to migrate my indexes to those machines. I have thought some scenarios but it will not be possible to do so. Let me explain what I thought and why it was not possible;

- I wanted to put the new machines into existing cluster and then I thought I could use exclude._ip option to exclude existing machines. So that old indexes would have migrated. 

This seemed possible for me but I can not stop working on the index, users should continue making searches on the indexes. So can you please tell me how I can migrate the data without disturbing clients ?

Thank you

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to migrate index without disturbing client?

Nikolas Everett
What you just described should work fine. exclude._ip will move the shards off of the nodes you exclude but queries and updates can proceed while this is happening because the data is still on the old nodes. The updates will make their way to the new copies via a transaction log reply mechanism.

Depending on how the clients connect you might need to shift their connection uris to the new servers and that might require restarting them - depending on how your clients work.

On Fri, May 29, 2015 at 8:38 AM, mehmet özer <[hidden email]> wrote:
Hello,

I am running a cluster which contains many indexes. In the near future I will have new machines so that I will need to migrate my indexes to those machines. I have thought some scenarios but it will not be possible to do so. Let me explain what I thought and why it was not possible;

- I wanted to put the new machines into existing cluster and then I thought I could use exclude._ip option to exclude existing machines. So that old indexes would have migrated. 

This seemed possible for me but I can not stop working on the index, users should continue making searches on the indexes. So can you please tell me how I can migrate the data without disturbing clients ?

Thank you

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1mYXppwguwnPCfyMCdG4jx0AmQBCMoqcxDE%2Bb9U1_nmQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to migrate index without disturbing client?

mehmet
Thanks for your answer. I didnt know the exclude doesnt disable the research. But according to the restart,

- I cannot restart the cluster because I have 9 tera of data so it can take one day to restart all cluster. In addition to that old machines and new machines will not know each other( I mean unicast variable of the elasticsearch.yml, the old machines will know just old ones). Do you think it can cause a problem ?

Thanks 

 

29 Mayıs 2015 Cuma 14:45:40 UTC+2 tarihinde Nikolas Everett yazdı:
What you just described should work fine. exclude._ip will move the shards off of the nodes you exclude but queries and updates can proceed while this is happening because the data is still on the old nodes. The updates will make their way to the new copies via a transaction log reply mechanism.

Depending on how the clients connect you might need to shift their connection uris to the new servers and that might require restarting them - depending on how your clients work.

On Fri, May 29, 2015 at 8:38 AM, mehmet özer <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="689eGBrUBYwJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">mehmeto...@...> wrote:
Hello,

I am running a cluster which contains many indexes. In the near future I will have new machines so that I will need to migrate my indexes to those machines. I have thought some scenarios but it will not be possible to do so. Let me explain what I thought and why it was not possible;

- I wanted to put the new machines into existing cluster and then I thought I could use exclude._ip option to exclude existing machines. So that old indexes would have migrated. 

This seemed possible for me but I can not stop working on the index, users should continue making searches on the indexes. So can you please tell me how I can migrate the data without disturbing clients ?

Thank you

--
Please update your bookmarks! We have moved to <a href="https://discuss.elastic.co/" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;">https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="689eGBrUBYwJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to migrate index without disturbing client?

Nikolas Everett
You certainly will have to think of a way to add new cluster masters - but you can still do it with unicast discovery I think.

Yeah - exclude doesn't disable searches - it just causes elasticsearch to move the shards using it normal shard moving mechanism. Its really just adding a routing rule.

You shouldn't have to restart the whole cluster - but you will have to convince your clients to look at the new nodes.

The last time I did a full restart of the Elasticsearch cluster was about a year ago when I upgraded from 0.90.something to 1.1.something and I expect the next full restart to be the upgrade to 2.0 when its released. In the mean time we've:
1. Done about twenty rolling restarts to pick up new code
2. Replaced all the hard drives in all the nodes - two at a time - moving data off of the nodes we were restarting using excluse._ip
3. Changed mount options on the elasticsearch data partition using the same technique
4. More than doubled the number of nodes in the cluster.

In our case the application connects using linux virtual server so all we had to do is teach the application to retry on connection errors for when lvs round-robin-ed it to a down Elasticsearch node and teach our lvs pool maintainer to depool elasticsearch nodes that are down.

I'm not an expert on the Java client so I don't know what you'd have to do to update and java applications to the new nodes. If you are using a proxy like nginx you'd just have to change where it points.

Nik

On Fri, May 29, 2015 at 8:51 AM, mehmet özer <[hidden email]> wrote:
Thanks for your answer. I didnt know the exclude doesnt disable the research. But according to the restart,

- I cannot restart the cluster because I have 9 tera of data so it can take one day to restart all cluster. In addition to that old machines and new machines will not know each other( I mean unicast variable of the elasticsearch.yml, the old machines will know just old ones). Do you think it can cause a problem ?

Thanks 

 

29 Mayıs 2015 Cuma 14:45:40 UTC+2 tarihinde Nikolas Everett yazdı:
What you just described should work fine. exclude._ip will move the shards off of the nodes you exclude but queries and updates can proceed while this is happening because the data is still on the old nodes. The updates will make their way to the new copies via a transaction log reply mechanism.

Depending on how the clients connect you might need to shift their connection uris to the new servers and that might require restarting them - depending on how your clients work.

On Fri, May 29, 2015 at 8:38 AM, mehmet özer <[hidden email]> wrote:
Hello,

I am running a cluster which contains many indexes. In the near future I will have new machines so that I will need to migrate my indexes to those machines. I have thought some scenarios but it will not be possible to do so. Let me explain what I thought and why it was not possible;

- I wanted to put the new machines into existing cluster and then I thought I could use exclude._ip option to exclude existing machines. So that old indexes would have migrated. 

This seemed possible for me but I can not stop working on the index, users should continue making searches on the indexes. So can you please tell me how I can migrate the data without disturbing clients ?

Thank you

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0JOF3%2BAQ4Y0Pfsp8yWheec6WwnrKEuqjCPN7rPHqH24g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to migrate index without disturbing client?

mehmet
Thats great. Thanks for your help !

29 Mayıs 2015 Cuma 15:09:21 UTC+2 tarihinde Nikolas Everett yazdı:
You certainly will have to think of a way to add new cluster masters - but you can still do it with unicast discovery I think.

Yeah - exclude doesn't disable searches - it just causes elasticsearch to move the shards using it normal shard moving mechanism. Its really just adding a routing rule.

You shouldn't have to restart the whole cluster - but you will have to convince your clients to look at the new nodes.

The last time I did a full restart of the Elasticsearch cluster was about a year ago when I upgraded from 0.90.something to 1.1.something and I expect the next full restart to be the upgrade to 2.0 when its released. In the mean time we've:
1. Done about twenty rolling restarts to pick up new code
2. Replaced all the hard drives in all the nodes - two at a time - moving data off of the nodes we were restarting using excluse._ip
3. Changed mount options on the elasticsearch data partition using the same technique
4. More than doubled the number of nodes in the cluster.

In our case the application connects using linux virtual server so all we had to do is teach the application to retry on connection errors for when lvs round-robin-ed it to a down Elasticsearch node and teach our lvs pool maintainer to depool elasticsearch nodes that are down.

I'm not an expert on the Java client so I don't know what you'd have to do to update and java applications to the new nodes. If you are using a proxy like nginx you'd just have to change where it points.

Nik

On Fri, May 29, 2015 at 8:51 AM, mehmet özer <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="H_pCjkI3Y8oJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">mehmeto...@...> wrote:
Thanks for your answer. I didnt know the exclude doesnt disable the research. But according to the restart,

- I cannot restart the cluster because I have 9 tera of data so it can take one day to restart all cluster. In addition to that old machines and new machines will not know each other( I mean unicast variable of the elasticsearch.yml, the old machines will know just old ones). Do you think it can cause a problem ?

Thanks 

 

29 Mayıs 2015 Cuma 14:45:40 UTC+2 tarihinde Nikolas Everett yazdı:
What you just described should work fine. exclude._ip will move the shards off of the nodes you exclude but queries and updates can proceed while this is happening because the data is still on the old nodes. The updates will make their way to the new copies via a transaction log reply mechanism.

Depending on how the clients connect you might need to shift their connection uris to the new servers and that might require restarting them - depending on how your clients work.

On Fri, May 29, 2015 at 8:38 AM, mehmet özer <[hidden email]> wrote:
Hello,

I am running a cluster which contains many indexes. In the near future I will have new machines so that I will need to migrate my indexes to those machines. I have thought some scenarios but it will not be possible to do so. Let me explain what I thought and why it was not possible;

- I wanted to put the new machines into existing cluster and then I thought I could use exclude._ip option to exclude existing machines. So that old indexes would have migrated. 

This seemed possible for me but I can not stop working on the index, users should continue making searches on the indexes. So can you please tell me how I can migrate the data without disturbing clients ?

Thank you

--
Please update your bookmarks! We have moved to <a href="https://discuss.elastic.co/" rel="nofollow" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;">https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to <a href="https://discuss.elastic.co/" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;">https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="H_pCjkI3Y8oJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0f797fb5-e8ef-4a34-9513-6ae891435976%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.