To Raid or not to Raid

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

To Raid or not to Raid

Elvar Böðvarsson
When running Elasticsearch on physical hardware you have it create replicas to make sure no node is a single point of failure. From everyone's experiance should I use Hardware Raid as well, or is it not needed?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

Nikolas Everett

Striping raid is viable for 2 or 3 disks because of the redundancy.  Software raid works fine for me. Hardware raid enables battery backed write behind but I don't know how important that is with ssds. Either way, we go 2xSSDs per server with os in mirrored raid and data striped.

Depending on your data you may want spinning disks instead, then battery backed writes are probably a bigger win.

On Dec 12, 2014 7:32 AM, "Elvar Böðvarsson" <[hidden email]> wrote:
When running Elasticsearch on physical hardware you have it create replicas to make sure no node is a single point of failure. From everyone's experiance should I use Hardware Raid as well, or is it not needed?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2k6KMPF%3Dw%3DzzAEg1bxCNp1Aa-7ft_BAT4wa6dZ%2Bfrupg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

Elvar Böðvarsson
This would be for log storage, plan is to go with HDD's for the long term storage and SSD's for short term storage. Total storage will be maybe 8tb total, using replica's would bring that to 16tb and if using raid1 or raid10 would bring total raw storage up to 32tb



On Friday, December 12, 2014 1:30:32 PM UTC, Nikolas Everett wrote:

Striping raid is viable for 2 or 3 disks because of the redundancy.  Software raid works fine for me. Hardware raid enables battery backed write behind but I don't know how important that is with ssds. Either way, we go 2xSSDs per server with os in mirrored raid and data striped.

Depending on your data you may want spinning disks instead, then battery backed writes are probably a bigger win.

On Dec 12, 2014 7:32 AM, "Elvar Böðvarsson" <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="eMY1OOB1nqUJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elv...@...> wrote:
When running Elasticsearch on physical hardware you have it create replicas to make sure no node is a single point of failure. From everyone's experiance should I use Hardware Raid as well, or is it not needed?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="eMY1OOB1nqUJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13b36ab6-7fb6-4f6f-b11d-79069aed74ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

Mark Walkom-2
In reply to this post by Nikolas Everett
If you can architect around the loss of a node and subsequent recovery, then I reckon it's worth testing the notion of not running RAID.

On 12 December 2014 at 14:30, Nikolas Everett <[hidden email]> wrote:

Striping raid is viable for 2 or 3 disks because of the redundancy.  Software raid works fine for me. Hardware raid enables battery backed write behind but I don't know how important that is with ssds. Either way, we go 2xSSDs per server with os in mirrored raid and data striped.

Depending on your data you may want spinning disks instead, then battery backed writes are probably a bigger win.

On Dec 12, 2014 7:32 AM, "Elvar Böðvarsson" <[hidden email]> wrote:
When running Elasticsearch on physical hardware you have it create replicas to make sure no node is a single point of failure. From everyone's experiance should I use Hardware Raid as well, or is it not needed?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2k6KMPF%3Dw%3DzzAEg1bxCNp1Aa-7ft_BAT4wa6dZ%2Bfrupg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8pf%3DxopyPw9tGs_GXDUZ25y7o_b%2BU5KujXU%2BJJbN012w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

Mark Walkom-2
In reply to this post by Elvar Böðvarsson
It's a risk thing, you need to be comfy with the risk of losing one disk and all that it entails.
If you can mitigate that through a process and you are happy with the remaining risk, then :)

On 12 December 2014 at 16:13, Elvar Böðvarsson <[hidden email]> wrote:
This would be for log storage, plan is to go with HDD's for the long term storage and SSD's for short term storage. Total storage will be maybe 8tb total, using replica's would bring that to 16tb and if using raid1 or raid10 would bring total raw storage up to 32tb



On Friday, December 12, 2014 1:30:32 PM UTC, Nikolas Everett wrote:

Striping raid is viable for 2 or 3 disks because of the redundancy.  Software raid works fine for me. Hardware raid enables battery backed write behind but I don't know how important that is with ssds. Either way, we go 2xSSDs per server with os in mirrored raid and data striped.

Depending on your data you may want spinning disks instead, then battery backed writes are probably a bigger win.

On Dec 12, 2014 7:32 AM, "Elvar Böðvarsson" <[hidden email]> wrote:
When running Elasticsearch on physical hardware you have it create replicas to make sure no node is a single point of failure. From everyone's experiance should I use Hardware Raid as well, or is it not needed?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13b36ab6-7fb6-4f6f-b11d-79069aed74ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9_ULd9j7jFVJaMYvLz3NNwYsqCAwHsfUxkxCwg1oV3-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

Jack Park-2
I cannot put my finger on it, but I think I recall that someone in this group once said that ES is IO intensive, that RAID would slow things down, arguing in favor of redundant servers over RAID. Does that still make sense?

On Fri, Dec 12, 2014 at 8:00 AM, Mark Walkom <[hidden email]> wrote:
It's a risk thing, you need to be comfy with the risk of losing one disk and all that it entails.
If you can mitigate that through a process and you are happy with the remaining risk, then :)

On 12 December 2014 at 16:13, Elvar Böðvarsson <[hidden email]> wrote:
This would be for log storage, plan is to go with HDD's for the long term storage and SSD's for short term storage. Total storage will be maybe 8tb total, using replica's would bring that to 16tb and if using raid1 or raid10 would bring total raw storage up to 32tb



On Friday, December 12, 2014 1:30:32 PM UTC, Nikolas Everett wrote:

Striping raid is viable for 2 or 3 disks because of the redundancy.  Software raid works fine for me. Hardware raid enables battery backed write behind but I don't know how important that is with ssds. Either way, we go 2xSSDs per server with os in mirrored raid and data striped.

Depending on your data you may want spinning disks instead, then battery backed writes are probably a bigger win.

On Dec 12, 2014 7:32 AM, "Elvar Böðvarsson" <[hidden email]> wrote:
When running Elasticsearch on physical hardware you have it create replicas to make sure no node is a single point of failure. From everyone's experiance should I use Hardware Raid as well, or is it not needed?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13b36ab6-7fb6-4f6f-b11d-79069aed74ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9_ULd9j7jFVJaMYvLz3NNwYsqCAwHsfUxkxCwg1oV3-A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwLhZNmSQVJmOKY4WtL-d003J0_%3DYamGh7i7uhT%3Df_W6A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

Elvar Böðvarsson
Just went through these slides

https://speakerdeck.com/bhaskarvk/scaling-elasticsearch-washington-dc-meetup

First of all, thats one HUGE cluster

Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok, maybe then to be safe go with 2x replicas, goes well with having 3x nodes


On Friday, December 12, 2014 4:18:01 PM UTC, Jack Park wrote:
I cannot put my finger on it, but I think I recall that someone in this group once said that ES is IO intensive, that RAID would slow things down, arguing in favor of redundant servers over RAID. Does that still make sense?

On Fri, Dec 12, 2014 at 8:00 AM, Mark Walkom <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="LnIv9VXHIIgJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">markw...@...> wrote:
It's a risk thing, you need to be comfy with the risk of losing one disk and all that it entails.
If you can mitigate that through a process and you are happy with the remaining risk, then :)

On 12 December 2014 at 16:13, Elvar Böðvarsson <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="LnIv9VXHIIgJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elv...@...> wrote:
This would be for log storage, plan is to go with HDD's for the long term storage and SSD's for short term storage. Total storage will be maybe 8tb total, using replica's would bring that to 16tb and if using raid1 or raid10 would bring total raw storage up to 32tb



On Friday, December 12, 2014 1:30:32 PM UTC, Nikolas Everett wrote:

Striping raid is viable for 2 or 3 disks because of the redundancy.  Software raid works fine for me. Hardware raid enables battery backed write behind but I don't know how important that is with ssds. Either way, we go 2xSSDs per server with os in mirrored raid and data striped.

Depending on your data you may want spinning disks instead, then battery backed writes are probably a bigger win.

On Dec 12, 2014 7:32 AM, "Elvar Böðvarsson" <[hidden email]> wrote:
When running Elasticsearch on physical hardware you have it create replicas to make sure no node is a single point of failure. From everyone's experiance should I use Hardware Raid as well, or is it not needed?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LnIv9VXHIIgJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/13b36ab6-7fb6-4f6f-b11d-79069aed74ad%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/13b36ab6-7fb6-4f6f-b11d-79069aed74ad%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/13b36ab6-7fb6-4f6f-b11d-79069aed74ad%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/13b36ab6-7fb6-4f6f-b11d-79069aed74ad%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LnIv9VXHIIgJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9_ULd9j7jFVJaMYvLz3NNwYsqCAwHsfUxkxCwg1oV3-A%40mail.gmail.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9_ULd9j7jFVJaMYvLz3NNwYsqCAwHsfUxkxCwg1oV3-A%40mail.gmail.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9_ULd9j7jFVJaMYvLz3NNwYsqCAwHsfUxkxCwg1oV3-A%40mail.gmail.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9_ULd9j7jFVJaMYvLz3NNwYsqCAwHsfUxkxCwg1oV3-A%40mail.gmail.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02040a16-9718-4fc8-be28-4ff781de5c40%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

joergprante@gmail.com
The statement is related to performance and I can't agree with it. You can easily build a RAID 0 system which has massive I/O throughput performance and is superior to JBOD, because RAID striping does not slow things down, it is as always as much as fast than a single drive and in most RAID levels it is much faster. 

In the past, RAID was invented for mirroring cheap and error-prone spindle disk arrays, while mirrors increase costs but decrease fault probability.

With Elasticsearch, the decision is if you still want to handle disk faults by drive redundancy (RAID) and all other hardware faults like power outages by server downtime. This is just a matter of organization and of cost. I would suggest from my experience: take control over your complete hardware setup, equip your systems with expensive SAS2 (or even better) controllers with RAID 0 to reduce cost and maximize performance, and handle all kind of hardware faults by server downtime, because ES replica level > 0 allows that.

There is also a simplification of SAN/NAS in the statement but that is a different discussion. Never use SAN/NAS for ES local gateway.

Jörg

On Fri, Dec 12, 2014 at 7:28 PM, Elvar Böðvarsson <[hidden email]> wrote:

Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok, maybe then to be safe go with 2x replicas, goes well with having 3x nodes

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE9g%2BJFNQdZYH1%3D3pz-b%2Bx0j9cc3M5dLV9rB4gL_SWvWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

Elvar Böðvarsson
If you have a node that has 4x disks as JBOD and you configure Elasticsearch to use all of them, so it will write to as like its Raid0. 

How does Elasticsearch handle a failure of one disk?
Will the whole node go down or will Elasticsearch continue to function just with lower total available storage? (and then recreate the shards that went down)



On Saturday, December 13, 2014 3:48:55 PM UTC, Jörg Prante wrote:
The statement is related to performance and I can't agree with it. You can easily build a RAID 0 system which has massive I/O throughput performance and is superior to JBOD, because RAID striping does not slow things down, it is as always as much as fast than a single drive and in most RAID levels it is much faster. 

In the past, RAID was invented for mirroring cheap and error-prone spindle disk arrays, while mirrors increase costs but decrease fault probability.

With Elasticsearch, the decision is if you still want to handle disk faults by drive redundancy (RAID) and all other hardware faults like power outages by server downtime. This is just a matter of organization and of cost. I would suggest from my experience: take control over your complete hardware setup, equip your systems with expensive SAS2 (or even better) controllers with RAID 0 to reduce cost and maximize performance, and handle all kind of hardware faults by server downtime, because ES replica level > 0 allows that.

There is also a simplification of SAN/NAS in the statement but that is a different discussion. Never use SAN/NAS for ES local gateway.

Jörg

On Fri, Dec 12, 2014 at 7:28 PM, Elvar Böðvarsson <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="uoJBKhbkvwsJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elv...@...> wrote:

Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok, maybe then to be safe go with 2x replicas, goes well with having 3x nodes

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fad86579-2072-438d-94da-80219e200b67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

Mark Walkom-2

Unfortunately you lose all data on the node as ES will stripe segments across the disks/mount points.


On 15 December 2014 at 11:45, Elvar Böðvarsson <[hidden email]> wrote:
If you have a node that has 4x disks as JBOD and you configure Elasticsearch to use all of them, so it will write to as like its Raid0. 

How does Elasticsearch handle a failure of one disk?
Will the whole node go down or will Elasticsearch continue to function just with lower total available storage? (and then recreate the shards that went down)



On Saturday, December 13, 2014 3:48:55 PM UTC, Jörg Prante wrote:
The statement is related to performance and I can't agree with it. You can easily build a RAID 0 system which has massive I/O throughput performance and is superior to JBOD, because RAID striping does not slow things down, it is as always as much as fast than a single drive and in most RAID levels it is much faster. 

In the past, RAID was invented for mirroring cheap and error-prone spindle disk arrays, while mirrors increase costs but decrease fault probability.

With Elasticsearch, the decision is if you still want to handle disk faults by drive redundancy (RAID) and all other hardware faults like power outages by server downtime. This is just a matter of organization and of cost. I would suggest from my experience: take control over your complete hardware setup, equip your systems with expensive SAS2 (or even better) controllers with RAID 0 to reduce cost and maximize performance, and handle all kind of hardware faults by server downtime, because ES replica level > 0 allows that.

There is also a simplification of SAN/NAS in the statement but that is a different discussion. Never use SAN/NAS for ES local gateway.

Jörg

On Fri, Dec 12, 2014 at 7:28 PM, Elvar Böðvarsson <[hidden email]> wrote:

Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok, maybe then to be safe go with 2x replicas, goes well with having 3x nodes

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fad86579-2072-438d-94da-80219e200b67%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_fAiiMnXq5dfGJBuTFxFGt182hVVv_qYKYiqC6%2BQfhUQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

Elvar Böðvarsson
That is quite scary

Lets say I have 3x nodes each with 4x disks, a total of 12x disks. Two disk fails would mean a data loss if its set to 1 replicas, with 2 replicas it would mean no data loss but with the cluster availability of n+1 nodes it would take down the whole cluster if one disk failsin two hosts.

Maybe It would be smart to split the elasticsearch processes or run two VM's per host.

https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/

If I did it like this then each process would have access to 2x disks meaning if one disk fails only two disks will be unavailable to the cluster instead of four.



On Monday, December 15, 2014 12:05:18 PM UTC, Mark Walkom wrote:

Unfortunately you lose all data on the node as ES will stripe segments across the disks/mount points.


On 15 December 2014 at 11:45, Elvar Böðvarsson <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ssWPniJ0VowJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elv...@...> wrote:
If you have a node that has 4x disks as JBOD and you configure Elasticsearch to use all of them, so it will write to as like its Raid0. 

How does Elasticsearch handle a failure of one disk?
Will the whole node go down or will Elasticsearch continue to function just with lower total available storage? (and then recreate the shards that went down)



On Saturday, December 13, 2014 3:48:55 PM UTC, Jörg Prante wrote:
The statement is related to performance and I can't agree with it. You can easily build a RAID 0 system which has massive I/O throughput performance and is superior to JBOD, because RAID striping does not slow things down, it is as always as much as fast than a single drive and in most RAID levels it is much faster. 

In the past, RAID was invented for mirroring cheap and error-prone spindle disk arrays, while mirrors increase costs but decrease fault probability.

With Elasticsearch, the decision is if you still want to handle disk faults by drive redundancy (RAID) and all other hardware faults like power outages by server downtime. This is just a matter of organization and of cost. I would suggest from my experience: take control over your complete hardware setup, equip your systems with expensive SAS2 (or even better) controllers with RAID 0 to reduce cost and maximize performance, and handle all kind of hardware faults by server downtime, because ES replica level > 0 allows that.

There is also a simplification of SAN/NAS in the statement but that is a different discussion. Never use SAN/NAS for ES local gateway.

Jörg

On Fri, Dec 12, 2014 at 7:28 PM, Elvar Böðvarsson <[hidden email]> wrote:

Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok, maybe then to be safe go with 2x replicas, goes well with having 3x nodes

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ssWPniJ0VowJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/fad86579-2072-438d-94da-80219e200b67%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/fad86579-2072-438d-94da-80219e200b67%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/fad86579-2072-438d-94da-80219e200b67%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/fad86579-2072-438d-94da-80219e200b67%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7e2260da-7082-4957-8302-57bf2a912b7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: To Raid or not to Raid

Bhaskar Karambelkar
In reply to this post by joergprante@gmail.com
Hi Jörg,
I'm the author of those slides, and that statement, even when taken out of context starts with  "Prefer", 
I don't think I need to explain what prefer means, but just in case ...
Using JBOD will be your safest bet as opposed to using something like RAID / SAN/ NAS unless you really know what you're doing.
I never said DON'T EVER use RAID or even SAN|NAS, just "prefer" JBOD.

I do agree with your assessment of RAID 0 below, but do remember that, that one statement was taken out of context, for full context I suggest you go through the whole slide deck and better yet the whole talk whose video was posted on elasticsearch site. I even made a point about some of my recommendations not being applicable to cloud deployments etc.

As to your point about simplification of NAS|SAN, that's the whole point of presenting to a wide audience, one simplifies things such that they can be applied to majority of the cases, and not concentrate on esoteric deployments :). As to local gateway, that's the only one ES recommends now, the  shared FS, HDFS, S3 gateways were long deprecated.

FWIW I fully agree with your statement on taking control over complete hardware setup, heck there's a full slide in there dedicated to this point, titled 'Know your platform'.

At the end of the day, there's no single silver bullet, everyone will have to evaluate what works best for their situation, what worked for us may not work well for others. It would be indeed very naive to take my slides as laws, they are more or less pointers worth exploring. Some may work for you some won't. They worked fairly well for us.

I might sound a bit defensive here, but hey we did build that cluster and we're nearing a Trillion documents in it, so I guess we must be doing something right :). 


Bhaskar


On Saturday, December 13, 2014 at 10:48:55 AM UTC-5, Jörg Prante wrote:
The statement is related to performance and I can't agree with it. You can easily build a RAID 0 system which has massive I/O throughput performance and is superior to JBOD, because RAID striping does not slow things down, it is as always as much as fast than a single drive and in most RAID levels it is much faster. 

In the past, RAID was invented for mirroring cheap and error-prone spindle disk arrays, while mirrors increase costs but decrease fault probability.

With Elasticsearch, the decision is if you still want to handle disk faults by drive redundancy (RAID) and all other hardware faults like power outages by server downtime. This is just a matter of organization and of cost. I would suggest from my experience: take control over your complete hardware setup, equip your systems with expensive SAS2 (or even better) controllers with RAID 0 to reduce cost and maximize performance, and handle all kind of hardware faults by server downtime, because ES replica level > 0 allows that.

There is also a simplification of SAN/NAS in the statement but that is a different discussion. Never use SAN/NAS for ES local gateway.

Jörg

On Fri, Dec 12, 2014 at 7:28 PM, Elvar Böðvarsson <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="uoJBKhbkvwsJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elv...@...> wrote:

Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok, maybe then to be safe go with 2x replicas, goes well with having 3x nodes

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d0fa5e9c-2658-4fef-a9ad-ea83873a8f28%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.