Creating and storing ES indices on S3

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Creating and storing ES indices on S3

Lavesh Gupta
Hi Everyone,

I went through the group and came across various replies by David Pilato, Kimchy and other people which stated to use EBS volumes or local disks for storing ES indices and S3 for periodic snapshots. Wanted to know the reason why using S3 will be a bad choice for creating and storing indices on S3 (even if I mount S3 to EC2 instance).

My main points for storing and creating indices on S3 are:

1) I want to avoid cost of using both EBS and S3. (EBS for storing indices data and S3 for snapshots). If I only use S3 then due to replication feature I do not have to worry about my data loss.
2) Using S3 I will not have to worry about failure of nodes as all data is in centralized placed being replicated.
3) I can use AWS auto scaling feature to much advantage as I can spawn a new EC2 machine when demand rises and then all the indexing and querying can be done through a central data source on S3. 

If somebody can suggest how I can achieve creating and storing of indices on S3 it will be a great help.

Thanking You

This message contains confidential information and is intended only for the individual to whom it is addressed. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and permanently delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, late or incomplete, or could contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version from the sender. Druva, www.druva.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/77357d50-d938-4c4b-9a4b-98001e793798%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Creating and storing ES indices on S3

Mark Walkom-2
It might be possible, but not really recommended as I'd imagine it'd be pretty slow, which would impact your performance of ES.


On 4 May 2015 at 23:56, Lavesh Gupta <[hidden email]> wrote:
Hi Everyone,

I went through the group and came across various replies by David Pilato, Kimchy and other people which stated to use EBS volumes or local disks for storing ES indices and S3 for periodic snapshots. Wanted to know the reason why using S3 will be a bad choice for creating and storing indices on S3 (even if I mount S3 to EC2 instance).

My main points for storing and creating indices on S3 are:

1) I want to avoid cost of using both EBS and S3. (EBS for storing indices data and S3 for snapshots). If I only use S3 then due to replication feature I do not have to worry about my data loss.
2) Using S3 I will not have to worry about failure of nodes as all data is in centralized placed being replicated.
3) I can use AWS auto scaling feature to much advantage as I can spawn a new EC2 machine when demand rises and then all the indexing and querying can be done through a central data source on S3. 

If somebody can suggest how I can achieve creating and storing of indices on S3 it will be a great help.

Thanking You

This message contains confidential information and is intended only for the individual to whom it is addressed. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and permanently delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, late or incomplete, or could contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version from the sender. Druva, www.druva.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/77357d50-d938-4c4b-9a4b-98001e793798%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8UV_%3DEBTLFdvDev4%2BMpVaOUwc7Yw6wBKve3TdO1CnpzQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.