Snapshots don't get compressed

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Snapshots don't get compressed

"Tomcsányi, Domonkos"
Hi Everyone,

I just started experimenting with the cool snapshot feature of ES (using
ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository on a
mounted NFS storage, using only the default options (compression turned
on). I checked it using curl:

user@myserver:~# curl -XGET
'http://IP_ADDRESS:9200/_snapshot/logBack?pretty'
{
   "logBack" : {
     "type" : "fs",
     "settings" : {
       "compress" : "true",
       "location" : "/es_snapshots"
     }
   }
}

So, after that I used curator to create a snapshot of some older
indices. The process finished after some minutes, so I decided to have a
look at the files it created. It turned out that the snapshot's files
take up exactly as much space as the indices did originally while they
were in the cluster, so no compression happened at all. This is kind of
a problem for me, because I assumed that compression will greatly reduce
the size of the indices I put in a snapshot. So is there anything I'm
doing wrong?

Thank you,
Domonkos

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Snapshots don't get compressed

"Tomcsányi, Domonkos"
I tried using tar on the indices folder, it resulted in a 1.5G tarball
(compared to the 2.8G size of the folder), so I really think something
is wrong here.

2014.09.18. 15:36 keltezéssel, "Tomcsányi, Domonkos" wrote:

> Hi Everyone,
>
> I just started experimenting with the cool snapshot feature of ES
> (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new
> repository on a mounted NFS storage, using only the default options
> (compression turned on). I checked it using curl:
>
> user@myserver:~# curl -XGET
> 'http://IP_ADDRESS:9200/_snapshot/logBack?pretty'
> {
>   "logBack" : {
>     "type" : "fs",
>     "settings" : {
>       "compress" : "true",
>       "location" : "/es_snapshots"
>     }
>   }
> }
>
> So, after that I used curator to create a snapshot of some older
> indices. The process finished after some minutes, so I decided to have
> a look at the files it created. It turned out that the snapshot's
> files take up exactly as much space as the indices did originally
> while they were in the cluster, so no compression happened at all.
> This is kind of a problem for me, because I assumed that compression
> will greatly reduce the size of the indices I put in a snapshot. So is
> there anything I'm doing wrong?
>
> Thank you,
> Domonkos
>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541AE449.8000508%40modit.hu.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Snapshots don't get compressed

dadoonet
In reply to this post by "Tomcsányi, Domonkos"
Only metadata are compressed.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos" <[hidden email]> a écrit :
>
> Hi Everyone,
>
> I just started experimenting with the cool snapshot feature of ES (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository on a mounted NFS storage, using only the default options (compression turned on). I checked it using curl:
>
> user@myserver:~# curl -XGET 'http://IP_ADDRESS:9200/_snapshot/logBack?pretty'
> {
>  "logBack" : {
>    "type" : "fs",
>    "settings" : {
>      "compress" : "true",
>      "location" : "/es_snapshots"
>    }
>  }
> }
>
> So, after that I used curator to create a snapshot of some older indices. The process finished after some minutes, so I decided to have a look at the files it created. It turned out that the snapshot's files take up exactly as much space as the indices did originally while they were in the cluster, so no compression happened at all. This is kind of a problem for me, because I assumed that compression will greatly reduce the size of the indices I put in a snapshot. So is there anything I'm doing wrong?
>
> Thank you,
> Domonkos
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8D533203-2C4F-4DA2-89F1-7B5F4842D5E0%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Snapshots don't get compressed

"Tomcsányi, Domonkos"
Thank you for the answer, but may I know why? What is the reason behind
this?

thanks,
Domonkos

2014.09.18. 16:15 keltezéssel, David Pilato írta:

> Only metadata are compressed.
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>> Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos" <[hidden email]> a écrit :
>>
>> Hi Everyone,
>>
>> I just started experimenting with the cool snapshot feature of ES (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository on a mounted NFS storage, using only the default options (compression turned on). I checked it using curl:
>>
>> user@myserver:~# curl -XGET 'http://IP_ADDRESS:9200/_snapshot/logBack?pretty'
>> {
>>   "logBack" : {
>>     "type" : "fs",
>>     "settings" : {
>>       "compress" : "true",
>>       "location" : "/es_snapshots"
>>     }
>>   }
>> }
>>
>> So, after that I used curator to create a snapshot of some older indices. The process finished after some minutes, so I decided to have a look at the files it created. It turned out that the snapshot's files take up exactly as much space as the indices did originally while they were in the cluster, so no compression happened at all. This is kind of a problem for me, because I assumed that compression will greatly reduce the size of the indices I put in a snapshot. So is there anything I'm doing wrong?
>>
>> Thank you,
>> Domonkos
>>
>> --
>> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
>> To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu.
>> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Snapshots don't get compressed

dadoonet
I don't know. I think this could happen in the future but unsure though.
May be Igor could answer this?



-- 
David Pilato | Technical Advocate | Elasticsearch.com


Le 18 septembre 2014 à 16:35:14, Tomcsányi, Domonkos ([hidden email]) a écrit:

Thank you for the answer, but may I know why? What is the reason behind
this?

thanks,
Domonkos

2014.09.18. 16:15 keltezéssel, David Pilato írta:

> Only metadata are compressed.
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>> Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos" <[hidden email]> a écrit :
>>
>> Hi Everyone,
>>
>> I just started experimenting with the cool snapshot feature of ES (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository on a mounted NFS storage, using only the default options (compression turned on). I checked it using curl:
>>
>> user@myserver:~# curl -XGET 'http://IP_ADDRESS:9200/_snapshot/logBack?pretty'
>> {
>> "logBack" : {
>> "type" : "fs",
>> "settings" : {
>> "compress" : "true",
>> "location" : "/es_snapshots"
>> }
>> }
>> }
>>
>> So, after that I used curator to create a snapshot of some older indices. The process finished after some minutes, so I decided to have a look at the files it created. It turned out that the snapshot's files take up exactly as much space as the indices did originally while they were in the cluster, so no compression happened at all. This is kind of a problem for me, because I assumed that compression will greatly reduce the size of the indices I put in a snapshot. So is there anything I'm doing wrong?
>>
>> Thank you,
>> Domonkos
>>
>> --
>> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
>> To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu.
>> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Snapshots don't get compressed

"Tomcsányi, Domonkos"
thank you again, I'll keep an eye on this issue.

Domonkos

2014.09.18. 16:38 keltezéssel, David Pilato írta:
I don't know. I think this could happen in the future but unsure though.
May be Igor could answer this?



-- 
David Pilato | Technical Advocate | Elasticsearch.com


Le 18 septembre 2014 à 16:35:14, Tomcsányi, Domonkos ([hidden email]) a écrit:

Thank you for the answer, but may I know why? What is the reason behind
this?

thanks,
Domonkos

2014.09.18. 16:15 keltezéssel, David Pilato írta:
> Only metadata are compressed.
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>> Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos" [hidden email] a écrit :
>>
>> Hi Everyone,
>>
>> I just started experimenting with the cool snapshot feature of ES (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository on a mounted NFS storage, using only the default options (compression turned on). I checked it using curl:
>>
>> user@myserver:~# curl -XGET 'http://IP_ADDRESS:9200/_snapshot/logBack?pretty'
>> {
>> "logBack" : {
>> "type" : "fs",
>> "settings" : {
>> "compress" : "true",
>> "location" : "/es_snapshots"
>> }
>> }
>> }
>>
>> So, after that I used curator to create a snapshot of some older indices. The process finished after some minutes, so I decided to have a look at the files it created. It turned out that the snapshot's files take up exactly as much space as the indices did originally while they were in the cluster, so no compression happened at all. This is kind of a problem for me, because I assumed that compression will greatly reduce the size of the indices I put in a snapshot. So is there anything I'm doing wrong?
>>
>> Thank you,
>> Domonkos
>>
>> --
>> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
>> To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu.
>> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541AF1F4.20502%40modit.hu.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Snapshots don't get compressed

Igor Motov-3
There were two reasons for not enabling compression on data files. First of all, the way "chunking" in snapshot/restore API was implemented didn't allow simple implementation of compression on data files. Moreover, the data files are already compressed to a certain degree. In my tests I was getting about 20% compressions rates on index data with recent versions of Elasticsearch (we have a limitation that we can compress only one file at a time). So, difficulties with implementation together with limit benefits made this feature not very compelling. After recent refactoring of the storage code, it's now much easier to add this feature in if it makes sense. However, I am really curious how you've got such great compression rates. Which version of Elasticsearch were these indices created with? Did you upgrade from older version of elasticsearch recently and most of your data files are created with older versions of Lucene?

On Thursday, September 18, 2014 10:53:58 AM UTC-4, Domonkos Tomcsanyi wrote:
thank you again, I'll keep an eye on this issue.

Domonkos

2014.09.18. 16:38 keltezéssel, David Pilato írta:
I don't know. I think this could happen in the future but unsure though.
May be Igor could answer this?

Here is a related doc PR: <a href="https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fpull%2F7654&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNFe2yi4QlvBIRxBw9tDL3IOdwzsIQ" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fpull%2F7654\46sa\75D\46sntz\0751\46usg\75AFQjCNFe2yi4QlvBIRxBw9tDL3IOdwzsIQ';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fpull%2F7654\46sa\75D\46sntz\0751\46usg\75AFQjCNFe2yi4QlvBIRxBw9tDL3IOdwzsIQ';return true;">https://github.com/elasticsearch/elasticsearch/pull/7654


-- 
David Pilato | Technical Advocate | Elasticsearch.com
<a href="https://twitter.com/dadoonet" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;">@dadoonet | <a href="https://twitter.com/elasticsearchfr" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;">@elasticsearchfr


Le 18 septembre 2014 à 16:35:14, Tomcsányi, Domonkos ([hidden email]) a écrit:

Thank you for the answer, but may I know why? What is the reason behind
this?

thanks,
Domonkos

2014.09.18. 16:15 keltezéssel, David Pilato írta:
> Only metadata are compressed.
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>> Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos" [hidden email] a écrit :
>>
>> Hi Everyone,
>>
>> I just started experimenting with the cool snapshot feature of ES (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository on a mounted NFS storage, using only the default options (compression turned on). I checked it using curl:
>>
>> user@myserver:~# curl -XGET '<a href="http://IP_ADDRESS:9200/_snapshot/logBack?pretty" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2FIP_ADDRESS%3A9200%2F_snapshot%2FlogBack%3Fpretty\46sa\75D\46sntz\0751\46usg\75AFQjCNFxXhp8QkGHibPfaxM9F0RrCxio_g';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2FIP_ADDRESS%3A9200%2F_snapshot%2FlogBack%3Fpretty\46sa\75D\46sntz\0751\46usg\75AFQjCNFxXhp8QkGHibPfaxM9F0RrCxio_g';return true;">http://IP_ADDRESS:9200/_snapshot/logBack?pretty'
>> {
>> "logBack" : {
>> "type" : "fs",
>> "settings" : {
>> "compress" : "true",
>> "location" : "/es_snapshots"
>> }
>> }
>> }
>>
>> So, after that I used curator to create a snapshot of some older indices. The process finished after some minutes, so I decided to have a look at the files it created. It turned out that the snapshot's files take up exactly as much space as the indices did originally while they were in the cluster, so no compression happened at all. This is kind of a problem for me, because I assumed that compression will greatly reduce the size of the indices I put in a snapshot. So is there anything I'm doing wrong?
>>
>> Thank you,
>> Domonkos
>>
>> --
>> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
>> To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu';return true;">https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu.
>> For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu';return true;">https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c8712a76-6eed-4a48-8a1e-95062e2f1e7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Snapshots don't get compressed

"Tomcsányi, Domonkos"
Yes, I updated from 0.90 (meaning some of my data was created with 0.90, then some with 1.0) to the latest version of elasticsearch. Is there a way to compress older data?

Thanks!


2014.09.19. 19:42 keltezéssel, Igor Motov írta:
There were two reasons for not enabling compression on data files. First of all, the way "chunking" in snapshot/restore API was implemented didn't allow simple implementation of compression on data files. Moreover, the data files are already compressed to a certain degree. In my tests I was getting about 20% compressions rates on index data with recent versions of Elasticsearch (we have a limitation that we can compress only one file at a time). So, difficulties with implementation together with limit benefits made this feature not very compelling. After recent refactoring of the storage code, it's now much easier to add this feature in if it makes sense. However, I am really curious how you've got such great compression rates. Which version of Elasticsearch were these indices created with? Did you upgrade from older version of elasticsearch recently and most of your data files are created with older versions of Lucene?

On Thursday, September 18, 2014 10:53:58 AM UTC-4, Domonkos Tomcsanyi wrote:
thank you again, I'll keep an eye on this issue.

Domonkos

2014.09.18. 16:38 keltezéssel, David Pilato írta:
I don't know. I think this could happen in the future but unsure though.
May be Igor could answer this?

Here is a related doc PR: <a moz-do-not-send="true" href="https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fpull%2F7654&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNFe2yi4QlvBIRxBw9tDL3IOdwzsIQ" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fpull%2F7654\46sa\75D\46sntz\0751\46usg\75AFQjCNFe2yi4QlvBIRxBw9tDL3IOdwzsIQ';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fpull%2F7654\46sa\75D\46sntz\0751\46usg\75AFQjCNFe2yi4QlvBIRxBw9tDL3IOdwzsIQ';return true;">https://github.com/elasticsearch/elasticsearch/pull/7654


-- 
David Pilato | Technical Advocate | Elasticsearch.com
<a moz-do-not-send="true" href="https://twitter.com/dadoonet" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;">@dadoonet | <a moz-do-not-send="true" href="https://twitter.com/elasticsearchfr" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;">@elasticsearchfr


Le 18 septembre 2014 à 16:35:14, Tomcsányi, Domonkos ([hidden email]) a écrit:

Thank you for the answer, but may I know why? What is the reason behind
this?

thanks,
Domonkos

2014.09.18. 16:15 keltezéssel, David Pilato írta:
> Only metadata are compressed.
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>> Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos" [hidden email] a écrit :
>>
>> Hi Everyone,
>>
>> I just started experimenting with the cool snapshot feature of ES (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository on a mounted NFS storage, using only the default options (compression turned on). I checked it using curl:
>>
>> user@myserver:~# curl -XGET '<a moz-do-not-send="true" href="http://IP_ADDRESS:9200/_snapshot/logBack?pretty" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2FIP_ADDRESS%3A9200%2F_snapshot%2FlogBack%3Fpretty\46sa\75D\46sntz\0751\46usg\75AFQjCNFxXhp8QkGHibPfaxM9F0RrCxio_g';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2FIP_ADDRESS%3A9200%2F_snapshot%2FlogBack%3Fpretty\46sa\75D\46sntz\0751\46usg\75AFQjCNFxXhp8QkGHibPfaxM9F0RrCxio_g';return true;">http://IP_ADDRESS:9200/_snapshot/logBack?pretty'
>> {
>> "logBack" : {
>> "type" : "fs",
>> "settings" : {
>> "compress" : "true",
>> "location" : "/es_snapshots"
>> }
>> }
>> }
>>
>> So, after that I used curator to create a snapshot of some older indices. The process finished after some minutes, so I decided to have a look at the files it created. It turned out that the snapshot's files take up exactly as much space as the indices did originally while they were in the cluster, so no compression happened at all. This is kind of a problem for me, because I assumed that compression will greatly reduce the size of the indices I put in a snapshot. So is there anything I'm doing wrong?
>>
>> Thank you,
>> Domonkos
>>
>> --
>> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
>> To view this discussion on the web visit <a moz-do-not-send="true" href="https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu';return true;">https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu.
>> For more options, visit <a moz-do-not-send="true" href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit <a moz-do-not-send="true" href="https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu';return true;">https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu.
For more options, visit <a moz-do-not-send="true" href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit <a moz-do-not-send="true" href="https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local.
For more options, visit <a moz-do-not-send="true" href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c8712a76-6eed-4a48-8a1e-95062e2f1e7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5429458F.7060704%40modit.hu.
For more options, visit https://groups.google.com/d/optout.