Snapshot files are created every time even no change in index.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Snapshot files are created every time even no change in index.

shoebalig
 I have a cron-job runs daily. There are chances that there is no modification or changes found in original index. however cron-job create snapshot file again on its scheduled time. Due to which number of snapshot files is too large. which leads restoration of index, takes a lot of time.
Is there anyway I should create snapshot file only when there is a difference in index data?
What if I retain only last 5 or last 10 snapshot? Will this cause any data lose anyway? And on restore whole index will be restored?

Please suggest.
Reply | Threaded
Open this post in threaded view
|

Re: Snapshot files are created every time even no change in index.

Mark Walkom-2
Snapshots are taken on the shard level, so if a shard changes due to things like merging, then it needs to record this.
You can delete snapshots, ES will just hold onto the shards it needs.

PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions!

On 28 May 2015 at 21:13, shoebalig <[hidden email]> wrote:
 I have a cron-job runs daily. There are chances that there is no
modification or changes found in original index. however cron-job create
snapshot file again on its scheduled time. Due to which number of snapshot
files is too large. which leads restoration of index, takes a lot of time.
Is there anyway I should create snapshot file only when there is a
difference in index data?
What if I retain only last 5 or last 10 snapshot? Will this cause any data
lose anyway? And on restore whole index will be restored?

Please suggest.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Snapshot-files-are-created-every-time-even-no-change-in-index-tp4074965.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1432811614768-4074965.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9kFwpxBaDZt0C_HzUG_U5zHpTWgGBqfReEzrB0WK1umg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Snapshot files are created every time even no change in index.

Allan Mitchell
Mark

This seems counter intuitive. Incremental to me and technologies with which I have worked before means that in order to restore to a known state you need  A+B+C..... In temporal order.  Consider this example

DELETE /test

POST /test
{
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "mytype" : {
            "properties" : {
                "myfield" : { "type" : "string","analyzer": "standard" }
            }
        }
    }
}


POST /test/mytype/1
{
    "myfield": "Doc 1"
}

Snapshot to snapshot1

POST /test/mytype/2
{
    "myfield": "Doc 2"
}


Snapshot to snapshot2


POST /test/mytype/3
{
    "myfield": "Doc 3"
}


Snapshot to  snapshot3

Delete the test index in ES

Now go ahead and delete snapshot2 

Restore the snapshot3 backup

POST /_snapshot/FILE_SYSTEM_ES/snapshot3/_restore

All 3 documents are in the index

Delete the index in ES again

Delete snapshot1
delete the index in ES
Restore the snapshot3 backup

All 3 documents are in the index


Even if the snapshot only takes the incremental changes it would seem the previous snapshot is merged with the latest allowing the deletions you see above.  To get to my latest position I only need the latest snapshot.  If I want to get back to a previous state then I only need the relevant snapshot.

I dare say this is just nomenclature but incremental to me does not do what the incremental in ES portrays

SQL Server Example.

If I take a FULL backup on Sunday and Incrementals on each night of the week.  If I delete everything before last night and try to restore only the latest incremental I would be stuck.

If I delete snapshots in reverse order then the document count in the index decreases as expected.

I cannot find any documentation anywhere on this process.



Allan



On 28 May 2015, at 23:07, Mark Walkom <[hidden email]> wrote:

Snapshots are taken on the shard level, so if a shard changes due to things like merging, then it needs to record this.
You can delete snapshots, ES will just hold onto the shards it needs.

PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions!

On 28 May 2015 at 21:13, shoebalig <[hidden email]> wrote:
 I have a cron-job runs daily. There are chances that there is no
modification or changes found in original index. however cron-job create
snapshot file again on its scheduled time. Due to which number of snapshot
files is too large. which leads restoration of index, takes a lot of time.
Is there anyway I should create snapshot file only when there is a
difference in index data?
What if I retain only last 5 or last 10 snapshot? Will this cause any data
lose anyway? And on restore whole index will be restored?

Please suggest.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Snapshot-files-are-created-every-time-even-no-change-in-index-tp4074965.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1432811614768-4074965.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9kFwpxBaDZt0C_HzUG_U5zHpTWgGBqfReEzrB0WK1umg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAECdJzBpOYsTwB9F6w6nNWRp_nCHmQfZy0fbeFy484Q9969Zog%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.