1.3.2 snapshot file system question

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

1.3.2 snapshot file system question

bitsofinfo.g
I have a cluster (1.3.2) of 10 data nodes and 5 master nodes.

I want to take a snapshot of one index. 

I'd like to configure a new "fs" snapshot "mybackupdir" where the "location" is ONLY accessible from the node (master node) I am issuing the snapshot creation PUT against. 

Next, if I issue a snapshot PUT for "mybackupdir/backup1" against the master node where that location is indeed accessible, will this work? Does the node that gets the snapshot request pull all the shard data from the data nodes over to itself and write them to the snapshot dir on disk? Or does each data-node responsible for each shard attempt to write to that same location? (thereby requiring that the snapshot "location" be accessible by all 15 nodes...)

I ask this because I have a cluster that spans two data-centers and they don't all have access to a globally available NFS share where I could have a common mount path for the snapshots root

thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1b2274d3-304d-4470-8cda-f9462c831aad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.3.2 snapshot file system question

Mark Walkom-2
It won't work, the snapshot is run against any node that has shards of the index and doesn't funnel data back to the node you ran the command on.

On 6 January 2015 at 02:40, <[hidden email]> wrote:
I have a cluster (1.3.2) of 10 data nodes and 5 master nodes.

I want to take a snapshot of one index. 

I'd like to configure a new "fs" snapshot "mybackupdir" where the "location" is ONLY accessible from the node (master node) I am issuing the snapshot creation PUT against. 

Next, if I issue a snapshot PUT for "mybackupdir/backup1" against the master node where that location is indeed accessible, will this work? Does the node that gets the snapshot request pull all the shard data from the data nodes over to itself and write them to the snapshot dir on disk? Or does each data-node responsible for each shard attempt to write to that same location? (thereby requiring that the snapshot "location" be accessible by all 15 nodes...)

I ask this because I have a cluster that spans two data-centers and they don't all have access to a globally available NFS share where I could have a common mount path for the snapshots root

thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1b2274d3-304d-4470-8cda-f9462c831aad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8h2LV0OSWmEwYZ_SAF7t41k4G00_p6_9XVw3SVLk5n_w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: 1.3.2 snapshot file system question

bitsofinfo.g
in case anyone else comes across this, I ended up making this to assist w/ my issue of aggregating fs snapshots 

https://github.com/bitsofinfo/elasticsearch-snapshot-manager

On Monday, January 5, 2015 at 1:08:05 PM UTC-7, Mark Walkom wrote:
It won't work, the snapshot is run against any node that has shards of the index and doesn't funnel data back to the node you ran the command on.

On 6 January 2015 at 02:40, <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Yy1BNLulDMJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">bitsof...@...> wrote:
I have a cluster (1.3.2) of 10 data nodes and 5 master nodes.

I want to take a snapshot of one index. 

I'd like to configure a new "fs" snapshot "mybackupdir" where the "location" is ONLY accessible from the node (master node) I am issuing the snapshot creation PUT against. 

Next, if I issue a snapshot PUT for "mybackupdir/backup1" against the master node where that location is indeed accessible, will this work? Does the node that gets the snapshot request pull all the shard data from the data nodes over to itself and write them to the snapshot dir on disk? Or does each data-node responsible for each shard attempt to write to that same location? (thereby requiring that the snapshot "location" be accessible by all 15 nodes...)

I ask this because I have a cluster that spans two data-centers and they don't all have access to a globally available NFS share where I could have a common mount path for the snapshots root

thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Yy1BNLulDMJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/1b2274d3-304d-4470-8cda-f9462c831aad%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/1b2274d3-304d-4470-8cda-f9462c831aad%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/1b2274d3-304d-4470-8cda-f9462c831aad%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/1b2274d3-304d-4470-8cda-f9462c831aad%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/359543f4-89c7-4194-8ac1-d6c26fdbdffa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.