Storage type in Elasticsearch

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Storage type in Elasticsearch

Prashant Agrawal
Hi All,

We are using ES for indexing bulk data which may go in terms of TB.
So just looking for which file system would be more suitable in terms of efficiency, performance, indexing time , searching time out of EXT4 , XFS or ZFS?

~Prashant
Reply | Threaded
Open this post in threaded view
|

Re: Storage type in Elasticsearch

Mark Walkom
You'll get the best performance from SSDs (obviously), but ES compresses data natively so you won't get much out of doing the same on the FS level.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email]
web: www.campaignmonitor.com


On 23 June 2014 22:45, Prashant Agrawal <[hidden email]> wrote:
Hi All,

We are using ES for indexing bulk data which may go in terms of TB.
So just looking for which file system would be more suitable in terms of
efficiency, performance, indexing time , searching time out of EXT4 , XFS or
ZFS?

~Prashant



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Storage-type-in-Elasticsearch-tp4058250.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1403527528571-4058250.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Y9%2BP2DduQxRXbxkoJ3QnvdgDm3iv%2B8MTDWT4CZkZsVHQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Storage type in Elasticsearch

Patrick Proniewski
On 24 juin 2014, at 00:27, Mark Walkom wrote:

> but ES compresses data natively so you won't get much out of doing the same on the FS level

Using a gzip compressed FS I got 2.27x compress ratio on my ES data. It's very bad for performance but depending on your needs it can be useful (storage of very old indices for example). ES reads and writes 4k blocks, so it's compression is made on 4k chunks. A bigger block size would yield to better compress ratio.

Patrick

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/686A2E78-B053-48A5-8E9F-357ABB7715DE%40patpro.net.
For more options, visit https://groups.google.com/d/optout.