Elasticsearch cluster on AWS. Article.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Elasticsearch cluster on AWS. Article.

Pavel P
Hi everyone,

Below you can find one big article, summing up all my experience of building the cluster on AWS.
When I started I had no information at all, but I found the needed pieces in different places, including this user group.

With your help I succeeded, and want to share the knowledge, that newcomers would find everything in one place.


Hope it would help someone!

Regards,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/37fa850d-2b0f-4495-9765-24b1f4019a19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch cluster on AWS. Article.

Mark Walkom
Nice and indepth, thanks for posting.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email]
web: www.campaignmonitor.com


On 15 August 2014 03:13, Pavel P <[hidden email]> wrote:
Hi everyone,

Below you can find one big article, summing up all my experience of building the cluster on AWS.
When I started I had no information at all, but I found the needed pieces in different places, including this user group.

With your help I succeeded, and want to share the knowledge, that newcomers would find everything in one place.


Hope it would help someone!

Regards,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/37fa850d-2b0f-4495-9765-24b1f4019a19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aJT3i_0T_6R%3DL%2B2NC0SQZ_Xet8kUi%2B37nhdvpF7-eybg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch cluster on AWS. Article.

Andrej
In reply to this post by Pavel P
Nice collection, well presented, thanks!

One note, you can probably restrict your permissions even further, instead of ReadOnly I use just the following 5 rules (may be even that is too much) :

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances", 
        "ec2:DescribeRegions",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeTags"
      ],
      "Resource": "*"
	}
  ]
}

Greets,
Andrej


Am Donnerstag, 14. August 2014 19:13:34 UTC+2 schrieb Pavel P:
Hi everyone,

Below you can find one big article, summing up all my experience of building the cluster on AWS.
When I started I had no information at all, but I found the needed pieces in different places, including this user group.

With your help I succeeded, and want to share the knowledge, that newcomers would find everything in one place.

<a href="http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F13%2Felasticsearch-cluster-on-aws-part-1-preparing-environment%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGmn3RLSv5f5vPGf1BJS8sauen8Tg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F13%2Felasticsearch-cluster-on-aws-part-1-preparing-environment%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGmn3RLSv5f5vPGf1BJS8sauen8Tg';return true;">Elasticsearch cluster on AWS. Part 1 - preparing the environment.
<a href="http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F14%2Felasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGYgA8Z7BcCCjsvDgYIrolEm0lwAg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F14%2Felasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGYgA8Z7BcCCjsvDgYIrolEm0lwAg';return true;">Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch.

Hope it would help someone!

Regards,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3a76bdfb-1d95-4b32-8769-116641011c1b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch cluster on AWS. Article.

Pavel P
Thanks Andrej,

Would have it in mind!


On Fri, Aug 15, 2014 at 2:25 PM, Andrej Rosenheinrich <[hidden email]> wrote:
Nice collection, well presented, thanks!

One note, you can probably restrict your permissions even further, instead of ReadOnly I use just the following 5 rules (may be even that is too much) :

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances", 
        "ec2:DescribeRegions",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeTags"
      ],
      "Resource": "*"
	}
  ]
}

Greets,
Andrej


Am Donnerstag, 14. August 2014 19:13:34 UTC+2 schrieb Pavel P:
Hi everyone,

Below you can find one big article, summing up all my experience of building the cluster on AWS.
When I started I had no information at all, but I found the needed pieces in different places, including this user group.

With your help I succeeded, and want to share the knowledge, that newcomers would find everything in one place.


Hope it would help someone!

Regards,

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/NU2pktgTkDc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3a76bdfb-1d95-4b32-8769-116641011c1b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Pavel Polyakov

Software Engineer - PHP team

E-mail: [hidden email]
Skype: 
pavel.polyakov.x1


Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If you are not the intended recipient or if you have received this e-mail by error please notify the sender immediately and destroy this e-mail. Any unauthorized review, copying, disclosure or distribution of the material in this e-mail is strictly forbidden. The contents of this e-mail is legally binding only if it is confirmed by letter or fax. The sending of e-mails to us does not have any period-protecting effect. Thank you for your cooperation.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFVUaqO3DzJQgqrhn9Vs7e29BwjVQZFk5R7QT2Kn4FSPCHxw%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch cluster on AWS. Article.

David Severski
In reply to this post by Pavel P
Thanks for collecting this information together! A couple points for tweaking:

1) Instead of hard coding the IAM credentials into the file, associate the instances with an IAM role. cloud-aws will use those automatically and AWS will handle key rotation for you.
2) You are launching all the instances into the same availability zone. That greatly reduces the ability of the cluster to tolerate an AWS outage. Stick each of your three nodes in a different availability zone and you'll be much better off.
3) EC2-Classic is deprecated. Demonstrating use of VPC would be helpful.
4) I encourage AWS hosts _not_ to be named. Users should plan for hosts to come and go. This means no-unique host names and hard coded IPs. AWS is ephemeral infrastructure and ES, as a cluster app, is very happy playing in this space.

and the big one...

5) Your security group looks to open ES to the world. DON'T DO THIS! There's been a tremendous amount of angst recently from ES clusters getting owned via open tcp/9200 and these security groups look to open your cluster to the entire internet. There's no need for that. cloud-aws will work with private IPs just fine.

David

On Thursday, August 14, 2014 10:13:34 AM UTC-7, Pavel P wrote:
Hi everyone,

Below you can find one big article, summing up all my experience of building the cluster on AWS.
When I started I had no information at all, but I found the needed pieces in different places, including this user group.

With your help I succeeded, and want to share the knowledge, that newcomers would find everything in one place.

<a href="http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F13%2Felasticsearch-cluster-on-aws-part-1-preparing-environment%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGmn3RLSv5f5vPGf1BJS8sauen8Tg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F13%2Felasticsearch-cluster-on-aws-part-1-preparing-environment%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGmn3RLSv5f5vPGf1BJS8sauen8Tg';return true;">Elasticsearch cluster on AWS. Part 1 - preparing the environment.
<a href="http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F14%2Felasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGYgA8Z7BcCCjsvDgYIrolEm0lwAg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F14%2Felasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGYgA8Z7BcCCjsvDgYIrolEm0lwAg';return true;">Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch.

Hope it would help someone!

Regards,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch cluster on AWS. Article.

Pavel P
@David Severski

Thanks for the input.

I've actually encountered the issue, when my security group was closed for the world and the private IPs were not stated in the security group rules. I had no idea why the could-aws does not connect to the hosts, because the public IPs were there.
I've stated that issue in the article.

However I agree with you, the cluster should not be available from the world.


On Fri, Aug 15, 2014 at 3:32 PM, David Severski <[hidden email]> wrote:
Thanks for collecting this information together! A couple points for tweaking:

1) Instead of hard coding the IAM credentials into the file, associate the instances with an IAM role. cloud-aws will use those automatically and AWS will handle key rotation for you.
2) You are launching all the instances into the same availability zone. That greatly reduces the ability of the cluster to tolerate an AWS outage. Stick each of your three nodes in a different availability zone and you'll be much better off.
3) EC2-Classic is deprecated. Demonstrating use of VPC would be helpful.
4) I encourage AWS hosts _not_ to be named. Users should plan for hosts to come and go. This means no-unique host names and hard coded IPs. AWS is ephemeral infrastructure and ES, as a cluster app, is very happy playing in this space.

and the big one...

5) Your security group looks to open ES to the world. DON'T DO THIS! There's been a tremendous amount of angst recently from ES clusters getting owned via open tcp/9200 and these security groups look to open your cluster to the entire internet. There's no need for that. cloud-aws will work with private IPs just fine.

David


On Thursday, August 14, 2014 10:13:34 AM UTC-7, Pavel P wrote:
Hi everyone,

Below you can find one big article, summing up all my experience of building the cluster on AWS.
When I started I had no information at all, but I found the needed pieces in different places, including this user group.

With your help I succeeded, and want to share the knowledge, that newcomers would find everything in one place.


Hope it would help someone!

Regards,

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/NU2pktgTkDc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Pavel Polyakov

Software Engineer - PHP team

E-mail: [hidden email]
Skype: 
pavel.polyakov.x1


Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If you are not the intended recipient or if you have received this e-mail by error please notify the sender immediately and destroy this e-mail. Any unauthorized review, copying, disclosure or distribution of the material in this e-mail is strictly forbidden. The contents of this e-mail is legally binding only if it is confirmed by letter or fax. The sending of e-mails to us does not have any period-protecting effect. Thank you for your cooperation.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFVUaqOWpEd5k9%3Dn5H2yu9kxjSpSJkO05jc%2BjxZNZbnm5AFN%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch cluster on AWS. Article.

Pavel P
@David Severski

One more question

3) EC2-Classic is deprecated. Demonstrating use of VPC would be helpful.

What do you mean by that?


On Fri, Aug 15, 2014 at 3:39 PM, Pavel P <[hidden email]> wrote:
@David Severski

Thanks for the input.

I've actually encountered the issue, when my security group was closed for the world and the private IPs were not stated in the security group rules. I had no idea why the could-aws does not connect to the hosts, because the public IPs were there.
I've stated that issue in the article.

However I agree with you, the cluster should not be available from the world.


On Fri, Aug 15, 2014 at 3:32 PM, David Severski <[hidden email]> wrote:
Thanks for collecting this information together! A couple points for tweaking:

1) Instead of hard coding the IAM credentials into the file, associate the instances with an IAM role. cloud-aws will use those automatically and AWS will handle key rotation for you.
2) You are launching all the instances into the same availability zone. That greatly reduces the ability of the cluster to tolerate an AWS outage. Stick each of your three nodes in a different availability zone and you'll be much better off.
3) EC2-Classic is deprecated. Demonstrating use of VPC would be helpful.
4) I encourage AWS hosts _not_ to be named. Users should plan for hosts to come and go. This means no-unique host names and hard coded IPs. AWS is ephemeral infrastructure and ES, as a cluster app, is very happy playing in this space.

and the big one...

5) Your security group looks to open ES to the world. DON'T DO THIS! There's been a tremendous amount of angst recently from ES clusters getting owned via open tcp/9200 and these security groups look to open your cluster to the entire internet. There's no need for that. cloud-aws will work with private IPs just fine.

David


On Thursday, August 14, 2014 10:13:34 AM UTC-7, Pavel P wrote:
Hi everyone,

Below you can find one big article, summing up all my experience of building the cluster on AWS.
When I started I had no information at all, but I found the needed pieces in different places, including this user group.

With your help I succeeded, and want to share the knowledge, that newcomers would find everything in one place.


Hope it would help someone!

Regards,

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/NU2pktgTkDc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Pavel Polyakov

Software Engineer - PHP team

E-mail: [hidden email]
Skype: 
pavel.polyakov.x1


Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If you are not the intended recipient or if you have received this e-mail by error please notify the sender immediately and destroy this e-mail. Any unauthorized review, copying, disclosure or distribution of the material in this e-mail is strictly forbidden. The contents of this e-mail is legally binding only if it is confirmed by letter or fax. The sending of e-mails to us does not have any period-protecting effect. Thank you for your cooperation.




--

Pavel Polyakov

Software Engineer - PHP team

E-mail: [hidden email]
Skype: 
pavel.polyakov.x1


Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If you are not the intended recipient or if you have received this e-mail by error please notify the sender immediately and destroy this e-mail. Any unauthorized review, copying, disclosure or distribution of the material in this e-mail is strictly forbidden. The contents of this e-mail is legally binding only if it is confirmed by letter or fax. The sending of e-mails to us does not have any period-protecting effect. Thank you for your cooperation.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFVUaqP2M6hSMTeT8Zpzj%3DZUo05W1J%3D3ZZ8VOGWSG4h0R9eeWQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch cluster on AWS. Article.

Andrej
In reply to this post by David Severski
David, you are of course right with 2), but one thing to concider is that you pay for incoming and outgoing traffic between different availability zones.

Am Freitag, 15. August 2014 14:32:54 UTC+2 schrieb David Severski:
Thanks for collecting this information together! A couple points for tweaking:

1) Instead of hard coding the IAM credentials into the file, associate the instances with an IAM role. cloud-aws will use those automatically and AWS will handle key rotation for you.
2) You are launching all the instances into the same availability zone. That greatly reduces the ability of the cluster to tolerate an AWS outage. Stick each of your three nodes in a different availability zone and you'll be much better off.
3) EC2-Classic is deprecated. Demonstrating use of VPC would be helpful.
4) I encourage AWS hosts _not_ to be named. Users should plan for hosts to come and go. This means no-unique host names and hard coded IPs. AWS is ephemeral infrastructure and ES, as a cluster app, is very happy playing in this space.

and the big one...

5) Your security group looks to open ES to the world. DON'T DO THIS! There's been a tremendous amount of angst recently from ES clusters getting owned via open tcp/9200 and these security groups look to open your cluster to the entire internet. There's no need for that. cloud-aws will work with private IPs just fine.

David

On Thursday, August 14, 2014 10:13:34 AM UTC-7, Pavel P wrote:
Hi everyone,

Below you can find one big article, summing up all my experience of building the cluster on AWS.
When I started I had no information at all, but I found the needed pieces in different places, including this user group.

With your help I succeeded, and want to share the knowledge, that newcomers would find everything in one place.

<a href="http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F13%2Felasticsearch-cluster-on-aws-part-1-preparing-environment%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGmn3RLSv5f5vPGf1BJS8sauen8Tg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F13%2Felasticsearch-cluster-on-aws-part-1-preparing-environment%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGmn3RLSv5f5vPGf1BJS8sauen8Tg';return true;">Elasticsearch cluster on AWS. Part 1 - preparing the environment.
<a href="http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F14%2Felasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGYgA8Z7BcCCjsvDgYIrolEm0lwAg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F14%2Felasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGYgA8Z7BcCCjsvDgYIrolEm0lwAg';return true;">Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch.

Hope it would help someone!

Regards,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aa245f74-5ae1-46ad-b01d-415735345356%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch cluster on AWS. Article.

David Severski
Sort of. If you use private IPs on both side the rate for transfer between availability zones (not regions) is 0.00 per GB. Zero cents. Just another reason not to use public IPs on your ES instances.

David

On Friday, August 15, 2014 7:09:52 AM UTC-7, Andrej Rosenheinrich wrote:
David, you are of course right with 2), but one thing to concider is that you pay for incoming and outgoing traffic between different availability zones.

Am Freitag, 15. August 2014 14:32:54 UTC+2 schrieb David Severski:
Thanks for collecting this information together! A couple points for tweaking:

1) Instead of hard coding the IAM credentials into the file, associate the instances with an IAM role. cloud-aws will use those automatically and AWS will handle key rotation for you.
2) You are launching all the instances into the same availability zone. That greatly reduces the ability of the cluster to tolerate an AWS outage. Stick each of your three nodes in a different availability zone and you'll be much better off.
3) EC2-Classic is deprecated. Demonstrating use of VPC would be helpful.
4) I encourage AWS hosts _not_ to be named. Users should plan for hosts to come and go. This means no-unique host names and hard coded IPs. AWS is ephemeral infrastructure and ES, as a cluster app, is very happy playing in this space.

and the big one...

5) Your security group looks to open ES to the world. DON'T DO THIS! There's been a tremendous amount of angst recently from ES clusters getting owned via open tcp/9200 and these security groups look to open your cluster to the entire internet. There's no need for that. cloud-aws will work with private IPs just fine.

David

On Thursday, August 14, 2014 10:13:34 AM UTC-7, Pavel P wrote:
Hi everyone,

Below you can find one big article, summing up all my experience of building the cluster on AWS.
When I started I had no information at all, but I found the needed pieces in different places, including this user group.

With your help I succeeded, and want to share the knowledge, that newcomers would find everything in one place.

<a href="http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F13%2Felasticsearch-cluster-on-aws-part-1-preparing-environment%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGmn3RLSv5f5vPGf1BJS8sauen8Tg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F13%2Felasticsearch-cluster-on-aws-part-1-preparing-environment%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGmn3RLSv5f5vPGf1BJS8sauen8Tg';return true;">Elasticsearch cluster on AWS. Part 1 - preparing the environment.
<a href="http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F14%2Felasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGYgA8Z7BcCCjsvDgYIrolEm0lwAg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpavelpolyakov.com%2F2014%2F08%2F14%2Felasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGYgA8Z7BcCCjsvDgYIrolEm0lwAg';return true;">Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch.

Hope it would help someone!

Regards,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/518c2feb-8c7c-4bef-9a1c-659fea2c1211%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.