Problem Configuring AWS S3 for Backups

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem Configuring AWS S3 for Backups

IronMike
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/89c4ac87-6f88-4b1a-a87e-c4482600b21c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

Glen Smith
According to
https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

dadoonet
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/022D907F-1D1E-44D3-A38B-3C6827CFF96C%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

IronMike
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ObRJW2ZEApwJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">gl...@...> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;">https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ObRJW2ZEApwJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5b056cb9-e71b-4317-b8f5-416385f731c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

Glen Smith
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;">https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/32558546-8352-4d51-90e7-94e96fd11bf7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

IronMike
Unfortunately, I tried with and without the region setting, no difference.

On Tuesday, July 1, 2014 7:43:21 PM UTC-4, Glen Smith wrote:
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;">https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/56de92a0-6c58-43b5-b4cc-df7c613ba4e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

IronMike
I installed latest ES version too "1.2.1", still getting same error
{
   "error": "RemoteTransportException[[node 2][inet[/10.211.154.24:9300]][cluster/repository/put]]; nested: RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}

Any ideas? I would appreciate some feedback on how to figure out this problem because I would like to backup our index to S3.

On Wednesday, July 2, 2014 3:36:58 PM UTC-4, sabdalla80 wrote:
Unfortunately, I tried with and without the region setting, no difference.

On Tuesday, July 1, 2014 7:43:21 PM UTC-4, Glen Smith wrote:
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;">https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/acc97d08-94c8-4459-b80d-2a32d229e109%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

Ross Simpson
That specific exception (com.amazonaws.AmazonClientException) is thrown by the AWS client libraries, and it means the library couldn't find your AWS credentials.  I'm not sure why, as the details in your original post look correct.

FWIW, S3 snapshots are working well for me.  Here's my setup:
ES 1.1.1
AWS cloud plugin 2.1.0

elasticsearch.yml:
cloud.aws.access_key: ...
cloud
.aws.secret_key: ......

Repo registration:
$ curl -XPUT 'http://localhost:9200/_snapshot/es-backups' -d '{"type":"s3","settings":{"compress":"true","base_path":"prod_backups","region":"us-east","bucket":"..."}}'
{"acknowledged":true}

In your latest post, it looks like you're running the command on a remote ES host (10.211.154.24).  Does that specific host have the AWS credentials in its ES config?  Snapshotting will require that all nodes in the cluster have the AWS credentials, because they will each be writing to S3.

Are there any relevant entries in the ES logs from startup?  



On Friday, 4 July 2014 04:25:44 UTC+10, sabdalla80 wrote:
I installed latest ES version too "1.2.1", still getting same error
{
   "error": "RemoteTransportException[[node 2][inet[/10.211.154.24:9300]][cluster/repository/put]]; nested: RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}

Any ideas? I would appreciate some feedback on how to figure out this problem because I would like to backup our index to S3.

On Wednesday, July 2, 2014 3:36:58 PM UTC-4, sabdalla80 wrote:
Unfortunately, I tried with and without the region setting, no difference.

On Tuesday, July 1, 2014 7:43:21 PM UTC-4, Glen Smith wrote:
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;">https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

dadoonet
Agreed. Could you share your elasticsearch.yml file without touching anything but only replacing Key/secret?

Keep the formating.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 4 juil. 2014 à 06:51, Ross Simpson <[hidden email]> a écrit :

That specific exception (com.amazonaws.AmazonClientException) is thrown by the AWS client libraries, and it means the library couldn't find your AWS credentials.  I'm not sure why, as the details in your original post look correct.

FWIW, S3 snapshots are working well for me.  Here's my setup:
ES 1.1.1
AWS cloud plugin 2.1.0

elasticsearch.yml:
cloud.aws.access_key: ...
cloud
.aws.secret_key: ......

Repo registration:
$ curl -XPUT 'http://localhost:9200/_snapshot/es-backups' -d '{"type":"s3","settings":{"compress":"true","base_path":"prod_backups","region":"us-east","bucket":"..."}}'
{"acknowledged":true}

In your latest post, it looks like you're running the command on a remote ES host (10.211.154.24).  Does that specific host have the AWS credentials in its ES config?  Snapshotting will require that all nodes in the cluster have the AWS credentials, because they will each be writing to S3.

Are there any relevant entries in the ES logs from startup?  



On Friday, 4 July 2014 04:25:44 UTC+10, sabdalla80 wrote:
I installed latest ES version too "1.2.1", still getting same error
{
   "error": "RemoteTransportException[[node 2][inet[/10.211.154.24:9300]][cluster/repository/put]]; nested: RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}

Any ideas? I would appreciate some feedback on how to figure out this problem because I would like to backup our index to S3.

On Wednesday, July 2, 2014 3:36:58 PM UTC-4, sabdalla80 wrote:
Unfortunately, I tried with and without the region setting, no difference.

On Tuesday, July 1, 2014 7:43:21 PM UTC-4, Glen Smith wrote:
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;">https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7414DF53-3844-4C76-A9B0-63D16EB32980%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

IronMike
My cluster has two instances, I have the same setup on both instances. To answer Ross's question, I use the curl command to register the repository from the instance(s) itself. Do I need to do anything else on the instance(s) as far as AWS credentials?

Here is what I have:
Enter ##################### Elasticsearch Configuration Example #####################


# This file contains an overview of various configuration settings,

# targeted at operations staff. Application developers should

# consult the guide at <http://elasticsearch.org/guide>.

#

# The installation procedure is covered at

# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

#

# Elasticsearch comes with reasonable defaults for most settings,

# so you can try it out without bothering with configuration.

#

# Most of the time, these defaults are just fine for running a production

# cluster. If you're fine-tuning your cluster, or wondering about the

# effect of certain configuration option, please _do ask_ on the

# mailing list or IRC channel [http://elasticsearch.org/community].


# Any element in the configuration can be replaced with environment variables

# by placing them in ${...} notation. For example:

#

# node.rack: ${RACK_ENV_VAR}


# For information on supported formats and syntax for the config file, see

# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>



################################### Cluster ###################################


# Cluster name identifies your cluster for auto-discovery. If you're running

# multiple clusters on the same network, make sure you're using unique names.

#

 cluster.name: rexCluster

#################################### Node #####################################


# Node names are generated dynamically on startup, so you're relieved

# from configuring them manually. You can tie this node to a specific name:

#

 node.name: "node 1"


# Every node can be configured to allow or deny being eligible as the master,

# and to allow or deny to store the data.

#

# Allow this node to be eligible as a master node (enabled by default):

#

 node.master: true

#

# Allow this node to store data (enabled by default):

#

# node.data: true


# You can exploit these settings to design advanced cluster topologies.

#

# 1. You want this node to never become a master node, only to hold data.

#    This will be the "workhorse" of your cluster.

#

# node.master: false

# node.data: true

#

# 2. You want this node to only serve as a master: to not store any data and

#    to have free resources. This will be the "coordinator" of your cluster.

#

# node.master: true

# node.data: false

#

# 3. You want this node to be neither master nor data node, but

#    to act as a "search load balancer" (fetching data from nodes,

#    aggregating results, etc.)

#

# node.master: false

# node.data: false


# Use the Cluster Health API [http://localhost:9200/_cluster/health], the

# Node Info API [http://localhost:9200/_nodes] or GUI tools

# such as <http://www.elasticsearch.org/overview/marvel/>,

# <http://github.com/karmi/elasticsearch-paramedic>,

# <http://github.com/lukas-vlcek/bigdesk> and

# <http://mobz.github.com/elasticsearch-head> to inspect the cluster state.


# A node can have generic attributes associated with it, which can later be used

# for customized shard allocation filtering, or allocation awareness. An attribute

# Path to directory containing configuration (this file and logging.yml):

#

 path.conf: /etc/elasticsearch


# Path to directory where to store index data allocated for this node.

#

# path.data: /path/to/data

#

# Can optionally include more than one location, causing data to be striped across

# the locations (a la RAID 0) on a file level, favouring locations with most free

# space on creation. For example:

#

 path.data: /opt/cores/elasticsearch/data


# Path to temporary files:

#

 path.work: /opt/cores/elasticsearch/work


# Path to log files:

#

 path.logs: /opt/cores/elasticsearch/logs


# Path to where plugins are installed:

#

# path.plugins: /path/to/plugins

################################### Memory ####################################


# Elasticsearch performs poorly when JVM starts swapping: you should ensure that

# it _never_ swaps.

#

# Set this property to true to lock the memory:

#

 bootstrap.mlockall: true

################################## Discovery ##################################


# Discovery infrastructure ensures nodes can be found within a cluster

# and master node is elected. Multicast discovery is the default.


# Set to ensure a node sees N other master eligible nodes to be considered

# operational within the cluster. Its recommended to set it to a higher value

# than 1 when running more than 2 nodes in the cluster.

#

 discovery.zen.minimum_master_nodes: 2


# Set the time to wait for ping responses from other nodes when discovering.

# Set this option to a higher value on a slow or congested network

# to minimize discovery failures:

#

 discovery.zen.ping.timeout: 3s


# For more information, see

# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>


# Unicast discovery allows to explicitly control which nodes will be used

# to discover the cluster. It can be used when multicast is not present,

# or to restrict the cluster communication-wise.

#

# 1. Disable multicast discovery (enabled by default):

#

 discovery.zen.ping.multicast.enabled: false

#

# 2. Configure an initial list of master nodes in the cluster

#    to perform discovery when new nodes (master or data) are started:

#

 discovery.zen.ping.unicast.hosts: ["IP1","IP2"]


# EC2 discovery allows to use AWS EC2 API in order to perform discovery.

#

# You have to install the cloud-aws plugin for enabling the EC2 discovery.

#

# For more information, see

# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>

#

# See <http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>

# for a step-by-step tutorial.

cloud:


     aws:

         access_key: XXXXX

         secret_key: YYYYYYYYYYY

    discovery:

        type: ec2




On Friday, July 4, 2014 2:27:48 AM UTC-4, David Pilato wrote:
Agreed. Could you share your elasticsearch.yml file without touching anything but only replacing Key/secret?

Keep the formating.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 4 juil. 2014 à 06:51, Ross Simpson <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="weuDypyRnfQJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">simp...@...> a écrit :

That specific exception (com.amazonaws.AmazonClientException) is thrown by the AWS client libraries, and it means the library couldn't find your AWS credentials.  I'm not sure why, as the details in your original post look correct.

FWIW, S3 snapshots are working well for me.  Here's my setup:
ES 1.1.1
AWS cloud plugin 2.1.0

elasticsearch.yml:
cloud.aws.access_key: ...
cloud
.aws.secret_key: ......

Repo registration:
$ curl -XPUT '<a href="http://localhost:9200/_snapshot/es-backups'" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;">http://localhost:9200/_snapshot/es-backups' -d '{"type":"s3","settings":{"compress":"true","base_path":"prod_backups","region":"us-east","bucket":"..."}}'
{"acknowledged":true}

In your latest post, it looks like you're running the command on a remote ES host (10.211.154.24).  Does that specific host have the AWS credentials in its ES config?  Snapshotting will require that all nodes in the cluster have the AWS credentials, because they will each be writing to S3.

Are there any relevant entries in the ES logs from startup?  



On Friday, 4 July 2014 04:25:44 UTC+10, sabdalla80 wrote:
I installed latest ES version too "1.2.1", still getting same error
{
   "error": "RemoteTransportException[[node 2][inet[/10.211.154.24:9300]][cluster/repository/put]]; nested: RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}

Any ideas? I would appreciate some feedback on how to figure out this problem because I would like to backup our index to S3.

On Wednesday, July 2, 2014 3:36:58 PM UTC-4, sabdalla80 wrote:
Unfortunately, I tried with and without the region setting, no difference.

On Tuesday, July 1, 2014 7:43:21 PM UTC-4, Glen Smith wrote:
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;">https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="weuDypyRnfQJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ec5d719f-514b-4224-9138-0a61370f2870%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

dadoonet
Try to add some spaces before cloud, so like this…


  cloud:


     aws:

         access_key: XXXXX

         secret_key: YYYYYYYYYYY

    discovery:

        type: ec2


-- 
David Pilato | Technical Advocate | Elasticsearch.com


Le 4 juillet 2014 à 16:47:03, sabdalla80 ([hidden email]) a écrit:

My cluster has two instances, I have the same setup on both instances. To answer Ross's question, I use the curl command to register the repository from the instance(s) itself. Do I need to do anything else on the instance(s) as far as AWS credentials?

Here is what I have:
Enter ##################### Elasticsearch Configuration Example #####################


# This file contains an overview of various configuration settings,

# targeted at operations staff. Application developers should

# consult the guide at <http://elasticsearch.org/guide>.

#

# The installation procedure is covered at

# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

#

# Elasticsearch comes with reasonable defaults for most settings,

# so you can try it out without bothering with configuration.

#

# Most of the time, these defaults are just fine for running a production

# cluster. If you're fine-tuning your cluster, or wondering about the

# effect of certain configuration option, please _do ask_ on the

# mailing list or IRC channel [http://elasticsearch.org/community].


# Any element in the configuration can be replaced with environment variables

# by placing them in ${...} notation. For example:

#

# node.rack: ${RACK_ENV_VAR}


# For information on supported formats and syntax for the config file, see

# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>



################################### Cluster ###################################


# Cluster name identifies your cluster for auto-discovery. If you're running

# multiple clusters on the same network, make sure you're using unique names.

#

 cluster.name: rexCluster

#################################### Node #####################################


# Node names are generated dynamically on startup, so you're relieved

# from configuring them manually. You can tie this node to a specific name:

#

 node.name: "node 1"


# Every node can be configured to allow or deny being eligible as the master,

# and to allow or deny to store the data.

#

# Allow this node to be eligible as a master node (enabled by default):

#

 node.master: true

#

# Allow this node to store data (enabled by default):

#

# node.data: true


# You can exploit these settings to design advanced cluster topologies.

#

# 1. You want this node to never become a master node, only to hold data.

#    This will be the "workhorse" of your cluster.

#

# node.master: false

# node.data: true

#

# 2. You want this node to only serve as a master: to not store any data and

#    to have free resources. This will be the "coordinator" of your cluster.

#

# node.master: true

# node.data: false

#

# 3. You want this node to be neither master nor data node, but

#    to act as a "search load balancer" (fetching data from nodes,

#    aggregating results, etc.)

#

# node.master: false

# node.data: false


# Use the Cluster Health API [http://localhost:9200/_cluster/health], the

# Node Info API [http://localhost:9200/_nodes] or GUI tools

# such as <http://www.elasticsearch.org/overview/marvel/>,

# <http://github.com/karmi/elasticsearch-paramedic>,

# <http://github.com/lukas-vlcek/bigdesk> and

# <http://mobz.github.com/elasticsearch-head> to inspect the cluster state.


# A node can have generic attributes associated with it, which can later be used

# for customized shard allocation filtering, or allocation awareness. An attribute

# Path to directory containing configuration (this file and logging.yml):

#

 path.conf: /etc/elasticsearch


# Path to directory where to store index data allocated for this node.

#

# path.data: /path/to/data

#

# Can optionally include more than one location, causing data to be striped across

# the locations (a la RAID 0) on a file level, favouring locations with most free

# space on creation. For example:

#

 path.data: /opt/cores/elasticsearch/data


# Path to temporary files:

#

 path.work: /opt/cores/elasticsearch/work


# Path to log files:

#

 path.logs: /opt/cores/elasticsearch/logs


# Path to where plugins are installed:

#

# path.plugins: /path/to/plugins

################################### Memory ####################################


# Elasticsearch performs poorly when JVM starts swapping: you should ensure that

# it _never_ swaps.

#

# Set this property to true to lock the memory:

#

 bootstrap.mlockall: true

################################## Discovery ##################################


# Discovery infrastructure ensures nodes can be found within a cluster

# and master node is elected. Multicast discovery is the default.


# Set to ensure a node sees N other master eligible nodes to be considered

# operational within the cluster. Its recommended to set it to a higher value

# than 1 when running more than 2 nodes in the cluster.

#

 discovery.zen.minimum_master_nodes: 2


# Set the time to wait for ping responses from other nodes when discovering.

# Set this option to a higher value on a slow or congested network

# to minimize discovery failures:

#

 discovery.zen.ping.timeout: 3s


# For more information, see

# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>


# Unicast discovery allows to explicitly control which nodes will be used

# to discover the cluster. It can be used when multicast is not present,

# or to restrict the cluster communication-wise.

#

# 1. Disable multicast discovery (enabled by default):

#

 discovery.zen.ping.multicast.enabled: false

#

# 2. Configure an initial list of master nodes in the cluster

#    to perform discovery when new nodes (master or data) are started:

#

 discovery.zen.ping.unicast.hosts: ["IP1","IP2"]


# EC2 discovery allows to use AWS EC2 API in order to perform discovery.

#

# You have to install the cloud-aws plugin for enabling the EC2 discovery.

#

# For more information, see

# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>

#

# See <http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>

# for a step-by-step tutorial.

cloud:


     aws:

         access_key: XXXXX

         secret_key: YYYYYYYYYYY

    discovery:

        type: ec2




On Friday, July 4, 2014 2:27:48 AM UTC-4, David Pilato wrote:
Agreed. Could you share your elasticsearch.yml file without touching anything but only replacing Key/secret?

Keep the formating.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 4 juil. 2014 à 06:51, Ross Simpson <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="weuDypyRnfQJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">simp...@...> a écrit :

That specific exception (com.amazonaws.AmazonClientException) is thrown by the AWS client libraries, and it means the library couldn't find your AWS credentials.  I'm not sure why, as the details in your original post look correct.

FWIW, S3 snapshots are working well for me.  Here's my setup:
ES 1.1.1
AWS cloud plugin 2.1.0

elasticsearch.yml:
cloud.aws.access_key: ...
cloud
.aws.secret_key: ......

Repo registration:
$ curl -XPUT '<a href="http://localhost:9200/_snapshot/es-backups'" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;">http://localhost:9200/_snapshot/es-backups' -d '{"type":"s3","settings":{"compress":"true","base_path":"prod_backups","region":"us-east","bucket":"..."}}'
{"acknowledged":true}

In your latest post, it looks like you're running the command on a remote ES host (10.211.154.24).  Does that specific host have the AWS credentials in its ES config?  Snapshotting will require that all nodes in the cluster have the AWS credentials, because they will each be writing to S3.

Are there any relevant entries in the ES logs from startup?  



On Friday, 4 July 2014 04:25:44 UTC+10, sabdalla80 wrote:
I installed latest ES version too "1.2.1", still getting same error
{
   "error": "RemoteTransportException[[node 2][inet[/10.211.154.24:9300]][cluster/repository/put]]; nested: RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}

Any ideas? I would appreciate some feedback on how to figure out this problem because I would like to backup our index to S3.

On Wednesday, July 2, 2014 3:36:58 PM UTC-4, sabdalla80 wrote:
Unfortunately, I tried with and without the region setting, no difference.

On Tuesday, July 1, 2014 7:43:21 PM UTC-4, Glen Smith wrote:
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;"> https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;"> https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="weuDypyRnfQJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;"> https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ec5d719f-514b-4224-9138-0a61370f2870%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53b6c241.436c6125.2fae%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

IronMike
In reply to this post by IronMike
I also meant to mention that I commented out the credentials in the yml file, I was still getting same message, meaning that credentials aren't probably getting read for whatever reason.

On Friday, July 4, 2014 10:46:58 AM UTC-4, sabdalla80 wrote:
My cluster has two instances, I have the same setup on both instances. To answer Ross's question, I use the curl command to register the repository from the instance(s) itself. Do I need to do anything else on the instance(s) as far as AWS credentials?

Here is what I have:
Enter ##################### Elasticsearch Configuration Example #####################


# This file contains an overview of various configuration settings,

# targeted at operations staff. Application developers should

# consult the guide at <<a href="http://elasticsearch.org/guide" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide\46sa\75D\46sntz\0751\46usg\75AFQjCNHGVzOZXgIUDzcAhujxiAQ65nxA2Q';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide\46sa\75D\46sntz\0751\46usg\75AFQjCNHGVzOZXgIUDzcAhujxiAQ65nxA2Q';return true;">http://elasticsearch.org/guide>.

#

# The installation procedure is covered at

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup.html\46sa\75D\46sntz\0751\46usg\75AFQjCNE8aLcBxYP2ZX6HT0KcrhSMb8lCuA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup.html\46sa\75D\46sntz\0751\46usg\75AFQjCNE8aLcBxYP2ZX6HT0KcrhSMb8lCuA';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

#

# Elasticsearch comes with reasonable defaults for most settings,

# so you can try it out without bothering with configuration.

#

# Most of the time, these defaults are just fine for running a production

# cluster. If you're fine-tuning your cluster, or wondering about the

# effect of certain configuration option, please _do ask_ on the

# mailing list or IRC channel [<a href="http://elasticsearch.org/community" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fcommunity\46sa\75D\46sntz\0751\46usg\75AFQjCNGkn9EDVM8wAXyGlGk_y3ruMtKGtQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fcommunity\46sa\75D\46sntz\0751\46usg\75AFQjCNGkn9EDVM8wAXyGlGk_y3ruMtKGtQ';return true;">http://elasticsearch.org/community].


# Any element in the configuration can be replaced with environment variables

# by placing them in ${...} notation. For example:

#

# node.rack: ${RACK_ENV_VAR}


# For information on supported formats and syntax for the config file, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-configuration.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHM-VEpNYJoEoG_96VPkPm2lXp_Cg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-configuration.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHM-VEpNYJoEoG_96VPkPm2lXp_Cg';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>



################################### Cluster ###################################


# Cluster name identifies your cluster for auto-discovery. If you're running

# multiple clusters on the same network, make sure you're using unique names.

#

 <a href="http://cluster.name" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fcluster.name\46sa\75D\46sntz\0751\46usg\75AFQjCNGbdss-_rGgAY2He8KM6ZUy2TdG8g';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fcluster.name\46sa\75D\46sntz\0751\46usg\75AFQjCNGbdss-_rGgAY2He8KM6ZUy2TdG8g';return true;">cluster.name: rexCluster

#################################### Node #####################################


# Node names are generated dynamically on startup, so you're relieved

# from configuring them manually. You can tie this node to a specific name:

#

 <a href="http://node.name" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fnode.name\46sa\75D\46sntz\0751\46usg\75AFQjCNE8Cff_rbggu0_UXL6VsgDVF5NFCg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fnode.name\46sa\75D\46sntz\0751\46usg\75AFQjCNE8Cff_rbggu0_UXL6VsgDVF5NFCg';return true;">node.name: "node 1"


# Every node can be configured to allow or deny being eligible as the master,

# and to allow or deny to store the data.

#

# Allow this node to be eligible as a master node (enabled by default):

#

 node.master: true

#

# Allow this node to store data (enabled by default):

#

# node.data: true


# You can exploit these settings to design advanced cluster topologies.

#

# 1. You want this node to never become a master node, only to hold data.

#    This will be the "workhorse" of your cluster.

#

# node.master: false

# node.data: true

#

# 2. You want this node to only serve as a master: to not store any data and

#    to have free resources. This will be the "coordinator" of your cluster.

#

# node.master: true

# node.data: false

#

# 3. You want this node to be neither master nor data node, but

#    to act as a "search load balancer" (fetching data from nodes,

#    aggregating results, etc.)

#

# node.master: false

# node.data: false


# Use the Cluster Health API [<a href="http://localhost:9200/_cluster/health" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_cluster%2Fhealth\46sa\75D\46sntz\0751\46usg\75AFQjCNGisf40DOD4uGeoIFwv68oiHJ0Nbg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_cluster%2Fhealth\46sa\75D\46sntz\0751\46usg\75AFQjCNGisf40DOD4uGeoIFwv68oiHJ0Nbg';return true;">http://localhost:9200/_cluster/health], the

# Node Info API [<a href="http://localhost:9200/_nodes" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_nodes\46sa\75D\46sntz\0751\46usg\75AFQjCNE99WYZY_Q1F7dQUvZcolOKUcd8Cw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_nodes\46sa\75D\46sntz\0751\46usg\75AFQjCNE99WYZY_Q1F7dQUvZcolOKUcd8Cw';return true;">http://localhost:9200/_nodes] or GUI tools

# such as <<a href="http://www.elasticsearch.org/overview/marvel/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Foverview%2Fmarvel%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFI3F2C99cAAKzrktL3jJ_ItyH5HQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Foverview%2Fmarvel%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFI3F2C99cAAKzrktL3jJ_ItyH5HQ';return true;">http://www.elasticsearch.org/overview/marvel/>,

# <<a href="http://github.com/karmi/elasticsearch-paramedic" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Fkarmi%2Felasticsearch-paramedic\46sa\75D\46sntz\0751\46usg\75AFQjCNFS5RH6734c8UlxtdcuA8CTomooXg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Fkarmi%2Felasticsearch-paramedic\46sa\75D\46sntz\0751\46usg\75AFQjCNFS5RH6734c8UlxtdcuA8CTomooXg';return true;">http://github.com/karmi/elasticsearch-paramedic>,

# <<a href="http://github.com/lukas-vlcek/bigdesk" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Flukas-vlcek%2Fbigdesk\46sa\75D\46sntz\0751\46usg\75AFQjCNES4Fnkp8Sklz8ywE8hEYD1YlcjZw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Flukas-vlcek%2Fbigdesk\46sa\75D\46sntz\0751\46usg\75AFQjCNES4Fnkp8Sklz8ywE8hEYD1YlcjZw';return true;">http://github.com/lukas-vlcek/bigdesk> and

# <<a href="http://mobz.github.com/elasticsearch-head" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmobz.github.com%2Felasticsearch-head\46sa\75D\46sntz\0751\46usg\75AFQjCNE1MKGqn-0PzflEARleUHZ-JtmtUw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmobz.github.com%2Felasticsearch-head\46sa\75D\46sntz\0751\46usg\75AFQjCNE1MKGqn-0PzflEARleUHZ-JtmtUw';return true;">http://mobz.github.com/elasticsearch-head> to inspect the cluster state.


# A node can have generic attributes associated with it, which can later be used

# for customized shard allocation filtering, or allocation awareness. An attribute

# Path to directory containing configuration (this file and logging.yml):

#

 path.conf: /etc/elasticsearch


# Path to directory where to store index data allocated for this node.

#

# path.data: /path/to/data

#

# Can optionally include more than one location, causing data to be striped across

# the locations (a la RAID 0) on a file level, favouring locations with most free

# space on creation. For example:

#

 path.data: /opt/cores/elasticsearch/data


# Path to temporary files:

#

 path.work: /opt/cores/elasticsearch/work


# Path to log files:

#

 path.logs: /opt/cores/elasticsearch/logs


# Path to where plugins are installed:

#

# path.plugins: /path/to/plugins

################################### Memory ####################################


# Elasticsearch performs poorly when JVM starts swapping: you should ensure that

# it _never_ swaps.

#

# Set this property to true to lock the memory:

#

 bootstrap.mlockall: true

################################## Discovery ##################################


# Discovery infrastructure ensures nodes can be found within a cluster

# and master node is elected. Multicast discovery is the default.


# Set to ensure a node sees N other master eligible nodes to be considered

# operational within the cluster. Its recommended to set it to a higher value

# than 1 when running more than 2 nodes in the cluster.

#

 discovery.zen.minimum_master_nodes: 2


# Set the time to wait for ping responses from other nodes when discovering.

# Set this option to a higher value on a slow or congested network

# to minimize discovery failures:

#

 discovery.zen.ping.timeout: 3s


# For more information, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-zen.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFlStmj9NvbUgRiDZa5KahFaXYNyA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-zen.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFlStmj9NvbUgRiDZa5KahFaXYNyA';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>


# Unicast discovery allows to explicitly control which nodes will be used

# to discover the cluster. It can be used when multicast is not present,

# or to restrict the cluster communication-wise.

#

# 1. Disable multicast discovery (enabled by default):

#

 discovery.zen.ping.multicast.enabled: false

#

# 2. Configure an initial list of master nodes in the cluster

#    to perform discovery when new nodes (master or data) are started:

#

 discovery.zen.ping.unicast.hosts: ["IP1","IP2"]


# EC2 discovery allows to use AWS EC2 API in order to perform discovery.

#

# You have to install the cloud-aws plugin for enabling the EC2 discovery.

#

# For more information, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-ec2.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHgW5G4UOJnux2jwHNgWliJJYFPKQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-ec2.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHgW5G4UOJnux2jwHNgWliJJYFPKQ';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>

#

# See <<a href="http://elasticsearch.org/tutorials/elasticsearch-on-ec2/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Ftutorials%2Felasticsearch-on-ec2%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGFqrWbd2zRkAaLdxk3JdmkKTsTBQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Ftutorials%2Felasticsearch-on-ec2%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGFqrWbd2zRkAaLdxk3JdmkKTsTBQ';return true;">http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>

# for a step-by-step tutorial.

cloud:


     aws:

         access_key: XXXXX

         secret_key: YYYYYYYYYYY

    discovery:

        type: ec2




On Friday, July 4, 2014 2:27:48 AM UTC-4, David Pilato wrote:
Agreed. Could you share your elasticsearch.yml file without touching anything but only replacing Key/secret?

Keep the formating.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 4 juil. 2014 à 06:51, Ross Simpson <[hidden email]> a écrit :

That specific exception (com.amazonaws.AmazonClientException) is thrown by the AWS client libraries, and it means the library couldn't find your AWS credentials.  I'm not sure why, as the details in your original post look correct.

FWIW, S3 snapshots are working well for me.  Here's my setup:
ES 1.1.1
AWS cloud plugin 2.1.0

elasticsearch.yml:
cloud.aws.access_key: ...
cloud
.aws.secret_key: ......

Repo registration:
$ curl -XPUT '<a href="http://localhost:9200/_snapshot/es-backups'" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;">http://localhost:9200/_snapshot/es-backups' -d '{"type":"s3","settings":{"compress":"true","base_path":"prod_backups","region":"us-east","bucket":"..."}}'
{"acknowledged":true}

In your latest post, it looks like you're running the command on a remote ES host (10.211.154.24).  Does that specific host have the AWS credentials in its ES config?  Snapshotting will require that all nodes in the cluster have the AWS credentials, because they will each be writing to S3.

Are there any relevant entries in the ES logs from startup?  



On Friday, 4 July 2014 04:25:44 UTC+10, sabdalla80 wrote:
I installed latest ES version too "1.2.1", still getting same error
{
   "error": "RemoteTransportException[[node 2][inet[/10.211.154.24:9300]][cluster/repository/put]]; nested: RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}

Any ideas? I would appreciate some feedback on how to figure out this problem because I would like to backup our index to S3.

On Wednesday, July 2, 2014 3:36:58 PM UTC-4, sabdalla80 wrote:
Unfortunately, I tried with and without the region setting, no difference.

On Tuesday, July 1, 2014 7:43:21 PM UTC-4, Glen Smith wrote:
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;">https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef64e5a5-9295-45f4-934a-6847bc46326e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

IronMike
In reply to this post by dadoonet
David, Great that helped. At first I created the space in "node 1" config but didn't do anything. So, I commented the credentials section from "node 2" config, That worked. So, I am a bit confused about this, Do I need to maintain both configs on the two instances, so they both have credentials in them, or do I just put it in "node 1" and since the cluster is configured, there will be no need to have credentials in "node 2" ?

Thanks

On Friday, July 4, 2014 11:03:45 AM UTC-4, David Pilato wrote:
Try to add some spaces before cloud, so like this…


  cloud:


     aws:

         access_key: XXXXX

         secret_key: YYYYYYYYYYY

    discovery:

        type: ec2


-- 
David Pilato | Technical Advocate | Elasticsearch.com
<a href="https://twitter.com/dadoonet" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;">@dadoonet | <a href="https://twitter.com/elasticsearchfr" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;">@elasticsearchfr


Le 4 juillet 2014 à 16:47:03, sabdalla80 (<a href="javascript:" target="_blank" gdf-obfuscated-mailto="FATe0_P_ycoJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">sabda...@...) a écrit:

My cluster has two instances, I have the same setup on both instances. To answer Ross's question, I use the curl command to register the repository from the instance(s) itself. Do I need to do anything else on the instance(s) as far as AWS credentials?

Here is what I have:
Enter ##################### Elasticsearch Configuration Example #####################


# This file contains an overview of various configuration settings,

# targeted at operations staff. Application developers should

# consult the guide at <<a href="http://elasticsearch.org/guide" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide\46sa\75D\46sntz\0751\46usg\75AFQjCNHGVzOZXgIUDzcAhujxiAQ65nxA2Q';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide\46sa\75D\46sntz\0751\46usg\75AFQjCNHGVzOZXgIUDzcAhujxiAQ65nxA2Q';return true;">http://elasticsearch.org/guide>.

#

# The installation procedure is covered at

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup.html\46sa\75D\46sntz\0751\46usg\75AFQjCNE8aLcBxYP2ZX6HT0KcrhSMb8lCuA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup.html\46sa\75D\46sntz\0751\46usg\75AFQjCNE8aLcBxYP2ZX6HT0KcrhSMb8lCuA';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

#

# Elasticsearch comes with reasonable defaults for most settings,

# so you can try it out without bothering with configuration.

#

# Most of the time, these defaults are just fine for running a production

# cluster. If you're fine-tuning your cluster, or wondering about the

# effect of certain configuration option, please _do ask_ on the

# mailing list or IRC channel [<a href="http://elasticsearch.org/community" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fcommunity\46sa\75D\46sntz\0751\46usg\75AFQjCNGkn9EDVM8wAXyGlGk_y3ruMtKGtQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fcommunity\46sa\75D\46sntz\0751\46usg\75AFQjCNGkn9EDVM8wAXyGlGk_y3ruMtKGtQ';return true;">http://elasticsearch.org/community].


# Any element in the configuration can be replaced with environment variables

# by placing them in ${...} notation. For example:

#

# node.rack: ${RACK_ENV_VAR}


# For information on supported formats and syntax for the config file, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-configuration.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHM-VEpNYJoEoG_96VPkPm2lXp_Cg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-configuration.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHM-VEpNYJoEoG_96VPkPm2lXp_Cg';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>



################################### Cluster ###################################


# Cluster name identifies your cluster for auto-discovery. If you're running

# multiple clusters on the same network, make sure you're using unique names.

#

 <a href="http://cluster.name" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fcluster.name\46sa\75D\46sntz\0751\46usg\75AFQjCNGbdss-_rGgAY2He8KM6ZUy2TdG8g';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fcluster.name\46sa\75D\46sntz\0751\46usg\75AFQjCNGbdss-_rGgAY2He8KM6ZUy2TdG8g';return true;">cluster.name: rexCluster

#################################### Node #####################################


# Node names are generated dynamically on startup, so you're relieved

# from configuring them manually. You can tie this node to a specific name:

#

 <a href="http://node.name" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fnode.name\46sa\75D\46sntz\0751\46usg\75AFQjCNE8Cff_rbggu0_UXL6VsgDVF5NFCg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fnode.name\46sa\75D\46sntz\0751\46usg\75AFQjCNE8Cff_rbggu0_UXL6VsgDVF5NFCg';return true;">node.name: "node 1"


# Every node can be configured to allow or deny being eligible as the master,

# and to allow or deny to store the data.

#

# Allow this node to be eligible as a master node (enabled by default):

#

 node.master: true

#

# Allow this node to store data (enabled by default):

#

# node.data: true


# You can exploit these settings to design advanced cluster topologies.

#

# 1. You want this node to never become a master node, only to hold data.

#    This will be the "workhorse" of your cluster.

#

# node.master: false

# node.data: true

#

# 2. You want this node to only serve as a master: to not store any data and

#    to have free resources. This will be the "coordinator" of your cluster.

#

# node.master: true

# node.data: false

#

# 3. You want this node to be neither master nor data node, but

#    to act as a "search load balancer" (fetching data from nodes,

#    aggregating results, etc.)

#

# node.master: false

# node.data: false


# Use the Cluster Health API [<a href="http://localhost:9200/_cluster/health" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_cluster%2Fhealth\46sa\75D\46sntz\0751\46usg\75AFQjCNGisf40DOD4uGeoIFwv68oiHJ0Nbg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_cluster%2Fhealth\46sa\75D\46sntz\0751\46usg\75AFQjCNGisf40DOD4uGeoIFwv68oiHJ0Nbg';return true;">http://localhost:9200/_cluster/health], the

# Node Info API [<a href="http://localhost:9200/_nodes" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_nodes\46sa\75D\46sntz\0751\46usg\75AFQjCNE99WYZY_Q1F7dQUvZcolOKUcd8Cw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_nodes\46sa\75D\46sntz\0751\46usg\75AFQjCNE99WYZY_Q1F7dQUvZcolOKUcd8Cw';return true;">http://localhost:9200/_nodes] or GUI tools

# such as <<a href="http://www.elasticsearch.org/overview/marvel/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Foverview%2Fmarvel%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFI3F2C99cAAKzrktL3jJ_ItyH5HQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Foverview%2Fmarvel%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFI3F2C99cAAKzrktL3jJ_ItyH5HQ';return true;">http://www.elasticsearch.org/overview/marvel/>,

# <<a href="http://github.com/karmi/elasticsearch-paramedic" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Fkarmi%2Felasticsearch-paramedic\46sa\75D\46sntz\0751\46usg\75AFQjCNFS5RH6734c8UlxtdcuA8CTomooXg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Fkarmi%2Felasticsearch-paramedic\46sa\75D\46sntz\0751\46usg\75AFQjCNFS5RH6734c8UlxtdcuA8CTomooXg';return true;">http://github.com/karmi/elasticsearch-paramedic>,

# <<a href="http://github.com/lukas-vlcek/bigdesk" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Flukas-vlcek%2Fbigdesk\46sa\75D\46sntz\0751\46usg\75AFQjCNES4Fnkp8Sklz8ywE8hEYD1YlcjZw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Flukas-vlcek%2Fbigdesk\46sa\75D\46sntz\0751\46usg\75AFQjCNES4Fnkp8Sklz8ywE8hEYD1YlcjZw';return true;">http://github.com/lukas-vlcek/bigdesk> and

# <<a href="http://mobz.github.com/elasticsearch-head" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmobz.github.com%2Felasticsearch-head\46sa\75D\46sntz\0751\46usg\75AFQjCNE1MKGqn-0PzflEARleUHZ-JtmtUw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmobz.github.com%2Felasticsearch-head\46sa\75D\46sntz\0751\46usg\75AFQjCNE1MKGqn-0PzflEARleUHZ-JtmtUw';return true;">http://mobz.github.com/elasticsearch-head> to inspect the cluster state.


# A node can have generic attributes associated with it, which can later be used

# for customized shard allocation filtering, or allocation awareness. An attribute

# Path to directory containing configuration (this file and logging.yml):

#

 path.conf: /etc/elasticsearch


# Path to directory where to store index data allocated for this node.

#

# path.data: /path/to/data

#

# Can optionally include more than one location, causing data to be striped across

# the locations (a la RAID 0) on a file level, favouring locations with most free

# space on creation. For example:

#

 path.data: /opt/cores/elasticsearch/data


# Path to temporary files:

#

 path.work: /opt/cores/elasticsearch/work


# Path to log files:

#

 path.logs: /opt/cores/elasticsearch/logs


# Path to where plugins are installed:

#

# path.plugins: /path/to/plugins

################################### Memory ####################################


# Elasticsearch performs poorly when JVM starts swapping: you should ensure that

# it _never_ swaps.

#

# Set this property to true to lock the memory:

#

 bootstrap.mlockall: true

################################## Discovery ##################################


# Discovery infrastructure ensures nodes can be found within a cluster

# and master node is elected. Multicast discovery is the default.


# Set to ensure a node sees N other master eligible nodes to be considered

# operational within the cluster. Its recommended to set it to a higher value

# than 1 when running more than 2 nodes in the cluster.

#

 discovery.zen.minimum_master_nodes: 2


# Set the time to wait for ping responses from other nodes when discovering.

# Set this option to a higher value on a slow or congested network

# to minimize discovery failures:

#

 discovery.zen.ping.timeout: 3s


# For more information, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-zen.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFlStmj9NvbUgRiDZa5KahFaXYNyA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-zen.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFlStmj9NvbUgRiDZa5KahFaXYNyA';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>


# Unicast discovery allows to explicitly control which nodes will be used

# to discover the cluster. It can be used when multicast is not present,

# or to restrict the cluster communication-wise.

#

# 1. Disable multicast discovery (enabled by default):

#

 discovery.zen.ping.multicast.enabled: false

#

# 2. Configure an initial list of master nodes in the cluster

#    to perform discovery when new nodes (master or data) are started:

#

 discovery.zen.ping.unicast.hosts: ["IP1","IP2"]


# EC2 discovery allows to use AWS EC2 API in order to perform discovery.

#

# You have to install the cloud-aws plugin for enabling the EC2 discovery.

#

# For more information, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-ec2.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHgW5G4UOJnux2jwHNgWliJJYFPKQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-ec2.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHgW5G4UOJnux2jwHNgWliJJYFPKQ';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>

#

# See <<a href="http://elasticsearch.org/tutorials/elasticsearch-on-ec2/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Ftutorials%2Felasticsearch-on-ec2%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGFqrWbd2zRkAaLdxk3JdmkKTsTBQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Ftutorials%2Felasticsearch-on-ec2%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGFqrWbd2zRkAaLdxk3JdmkKTsTBQ';return true;">http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>

# for a step-by-step tutorial.

cloud:


     aws:

         access_key: XXXXX

         secret_key: YYYYYYYYYYY

    discovery:

        type: ec2




On Friday, July 4, 2014 2:27:48 AM UTC-4, David Pilato wrote:
Agreed. Could you share your elasticsearch.yml file without touching anything but only replacing Key/secret?

Keep the formating.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 4 juil. 2014 à 06:51, Ross Simpson <[hidden email]> a écrit :

That specific exception (com.amazonaws.AmazonClientException) is thrown by the AWS client libraries, and it means the library couldn't find your AWS credentials.  I'm not sure why, as the details in your original post look correct.

FWIW, S3 snapshots are working well for me.  Here's my setup:
ES 1.1.1
AWS cloud plugin 2.1.0

elasticsearch.yml:
cloud.aws.access_key: ...
cloud
.aws.secret_key: ......

Repo registration:
$ curl -XPUT '<a href="http://localhost:9200/_snapshot/es-backups'" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;">http://localhost:9200/_snapshot/es-backups' -d '{"type":"s3","settings":{"compress":"true","base_path":"prod_backups","region":"us-east","bucket":"..."}}'
{"acknowledged":true}

In your latest post, it looks like you're running the command on a remote ES host (10.211.154.24).  Does that specific host have the AWS credentials in its ES config?  Snapshotting will require that all nodes in the cluster have the AWS credentials, because they will each be writing to S3.

Are there any relevant entries in the ES logs from startup?  



On Friday, 4 July 2014 04:25:44 UTC+10, sabdalla80 wrote:
I installed latest ES version too "1.2.1", still getting same error
{
   "error": "RemoteTransportException[[node 2][inet[/10.211.154.24:9300]][cluster/repository/put]]; nested: RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}

Any ideas? I would appreciate some feedback on how to figure out this problem because I would like to backup our index to S3.

On Wednesday, July 2, 2014 3:36:58 PM UTC-4, sabdalla80 wrote:
Unfortunately, I tried with and without the region setting, no difference.

On Tuesday, July 1, 2014 7:43:21 PM UTC-4, Glen Smith wrote:
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;"> https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;"> https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;"> https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="FATe0_P_ycoJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/ec5d719f-514b-4224-9138-0a61370f2870%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/ec5d719f-514b-4224-9138-0a61370f2870%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/ec5d719f-514b-4224-9138-0a61370f2870%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;"> https://groups.google.com/d/msgid/elasticsearch/ec5d719f-514b-4224-9138-0a61370f2870%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db55bb02-b5a1-44c3-8692-01d2bb7efbae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

dadoonet
I would recommend that both nodes have that config.
If you want to add a third node, because you want to replace the first one for any reason, it's better to have all in place.

One thing you can do is to prepare a clean machine, with all settings but no data. Then create an image.

If you want to scale out, just launch new instances using that ready to use image.

My 2 cents


-- 
David Pilato | Technical Advocate | Elasticsearch.com


Le 4 juillet 2014 à 17:39:02, sabdalla80 ([hidden email]) a écrit:

David, Great that helped. At first I created the space in "node 1" config but didn't do anything. So, I commented the credentials section from "node 2" config, That worked. So, I am a bit confused about this, Do I need to maintain both configs on the two instances, so they both have credentials in them, or do I just put it in "node 1" and since the cluster is configured, there will be no need to have credentials in "node 2" ?

Thanks

On Friday, July 4, 2014 11:03:45 AM UTC-4, David Pilato wrote:
Try to add some spaces before cloud, so like this…


  cloud:


     aws:

         access_key: XXXXX

         secret_key: YYYYYYYYYYY

    discovery:

        type: ec2


-- 
David Pilato | Technical Advocate | Elasticsearch.com
<a href="https://twitter.com/dadoonet" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;"> @dadoonet | <a href="https://twitter.com/elasticsearchfr" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;">@elasticsearchfr


Le 4 juillet 2014 à 16:47:03, sabdalla80 (<a href="javascript:" target="_blank" gdf-obfuscated-mailto="FATe0_P_ycoJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">sabda...@...) a écrit:

My cluster has two instances, I have the same setup on both instances. To answer Ross's question, I use the curl command to register the repository from the instance(s) itself. Do I need to do anything else on the instance(s) as far as AWS credentials?

Here is what I have:
Enter ##################### Elasticsearch Configuration Example #####################


# This file contains an overview of various configuration settings,

# targeted at operations staff. Application developers should

# consult the guide at <<a href="http://elasticsearch.org/guide" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide\46sa\75D\46sntz\0751\46usg\75AFQjCNHGVzOZXgIUDzcAhujxiAQ65nxA2Q';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide\46sa\75D\46sntz\0751\46usg\75AFQjCNHGVzOZXgIUDzcAhujxiAQ65nxA2Q';return true;">http://elasticsearch.org/guide>.

#

# The installation procedure is covered at

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup.html\46sa\75D\46sntz\0751\46usg\75AFQjCNE8aLcBxYP2ZX6HT0KcrhSMb8lCuA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup.html\46sa\75D\46sntz\0751\46usg\75AFQjCNE8aLcBxYP2ZX6HT0KcrhSMb8lCuA';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

#

# Elasticsearch comes with reasonable defaults for most settings,

# so you can try it out without bothering with configuration.

#

# Most of the time, these defaults are just fine for running a production

# cluster. If you're fine-tuning your cluster, or wondering about the

# effect of certain configuration option, please _do ask_ on the

# mailing list or IRC channel [<a href="http://elasticsearch.org/community" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fcommunity\46sa\75D\46sntz\0751\46usg\75AFQjCNGkn9EDVM8wAXyGlGk_y3ruMtKGtQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fcommunity\46sa\75D\46sntz\0751\46usg\75AFQjCNGkn9EDVM8wAXyGlGk_y3ruMtKGtQ';return true;">http://elasticsearch.org/community].


# Any element in the configuration can be replaced with environment variables

# by placing them in ${...} notation. For example:

#

# node.rack: ${RACK_ENV_VAR}


# For information on supported formats and syntax for the config file, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-configuration.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHM-VEpNYJoEoG_96VPkPm2lXp_Cg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-configuration.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHM-VEpNYJoEoG_96VPkPm2lXp_Cg';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>



################################### Cluster ###################################


# Cluster name identifies your cluster for auto-discovery. If you're running

# multiple clusters on the same network, make sure you're using unique names.

#

 <a href="http://cluster.name" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fcluster.name\46sa\75D\46sntz\0751\46usg\75AFQjCNGbdss-_rGgAY2He8KM6ZUy2TdG8g';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fcluster.name\46sa\75D\46sntz\0751\46usg\75AFQjCNGbdss-_rGgAY2He8KM6ZUy2TdG8g';return true;">cluster.name: rexCluster

#################################### Node #####################################


# Node names are generated dynamically on startup, so you're relieved

# from configuring them manually. You can tie this node to a specific name:

#

 <a href="http://node.name" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fnode.name\46sa\75D\46sntz\0751\46usg\75AFQjCNE8Cff_rbggu0_UXL6VsgDVF5NFCg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fnode.name\46sa\75D\46sntz\0751\46usg\75AFQjCNE8Cff_rbggu0_UXL6VsgDVF5NFCg';return true;">node.name: "node 1"


# Every node can be configured to allow or deny being eligible as the master,

# and to allow or deny to store the data.

#

# Allow this node to be eligible as a master node (enabled by default):

#

 node.master: true

#

# Allow this node to store data (enabled by default):

#

# node.data: true


# You can exploit these settings to design advanced cluster topologies.

#

# 1. You want this node to never become a master node, only to hold data.

#    This will be the "workhorse" of your cluster.

#

# node.master: false

# node.data: true

#

# 2. You want this node to only serve as a master: to not store any data and

#    to have free resources. This will be the "coordinator" of your cluster.

#

# node.master: true

# node.data: false

#

# 3. You want this node to be neither master nor data node, but

#    to act as a "search load balancer" (fetching data from nodes,

#    aggregating results, etc.)

#

# node.master: false

# node.data: false


# Use the Cluster Health API [<a href="http://localhost:9200/_cluster/health" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_cluster%2Fhealth\46sa\75D\46sntz\0751\46usg\75AFQjCNGisf40DOD4uGeoIFwv68oiHJ0Nbg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_cluster%2Fhealth\46sa\75D\46sntz\0751\46usg\75AFQjCNGisf40DOD4uGeoIFwv68oiHJ0Nbg';return true;">http://localhost:9200/_cluster/health], the

# Node Info API [<a href="http://localhost:9200/_nodes" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_nodes\46sa\75D\46sntz\0751\46usg\75AFQjCNE99WYZY_Q1F7dQUvZcolOKUcd8Cw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_nodes\46sa\75D\46sntz\0751\46usg\75AFQjCNE99WYZY_Q1F7dQUvZcolOKUcd8Cw';return true;">http://localhost:9200/_nodes] or GUI tools

# such as <<a href="http://www.elasticsearch.org/overview/marvel/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Foverview%2Fmarvel%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFI3F2C99cAAKzrktL3jJ_ItyH5HQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Foverview%2Fmarvel%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFI3F2C99cAAKzrktL3jJ_ItyH5HQ';return true;">http://www.elasticsearch.org/overview/marvel/>,

# <<a href="http://github.com/karmi/elasticsearch-paramedic" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Fkarmi%2Felasticsearch-paramedic\46sa\75D\46sntz\0751\46usg\75AFQjCNFS5RH6734c8UlxtdcuA8CTomooXg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Fkarmi%2Felasticsearch-paramedic\46sa\75D\46sntz\0751\46usg\75AFQjCNFS5RH6734c8UlxtdcuA8CTomooXg';return true;">http://github.com/karmi/elasticsearch-paramedic>,

# <<a href="http://github.com/lukas-vlcek/bigdesk" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Flukas-vlcek%2Fbigdesk\46sa\75D\46sntz\0751\46usg\75AFQjCNES4Fnkp8Sklz8ywE8hEYD1YlcjZw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Flukas-vlcek%2Fbigdesk\46sa\75D\46sntz\0751\46usg\75AFQjCNES4Fnkp8Sklz8ywE8hEYD1YlcjZw';return true;">http://github.com/lukas-vlcek/bigdesk> and

# <<a href="http://mobz.github.com/elasticsearch-head" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmobz.github.com%2Felasticsearch-head\46sa\75D\46sntz\0751\46usg\75AFQjCNE1MKGqn-0PzflEARleUHZ-JtmtUw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmobz.github.com%2Felasticsearch-head\46sa\75D\46sntz\0751\46usg\75AFQjCNE1MKGqn-0PzflEARleUHZ-JtmtUw';return true;">http://mobz.github.com/elasticsearch-head> to inspect the cluster state.


# A node can have generic attributes associated with it, which can later be used

# for customized shard allocation filtering, or allocation awareness. An attribute

# Path to directory containing configuration (this file and logging.yml):

#

 path.conf: /etc/elasticsearch


# Path to directory where to store index data allocated for this node.

#

# path.data: /path/to/data

#

# Can optionally include more than one location, causing data to be striped across

# the locations (a la RAID 0) on a file level, favouring locations with most free

# space on creation. For example:

#

 path.data: /opt/cores/elasticsearch/data


# Path to temporary files:

#

 path.work: /opt/cores/elasticsearch/work


# Path to log files:

#

 path.logs: /opt/cores/elasticsearch/logs


# Path to where plugins are installed:

#

# path.plugins: /path/to/plugins

################################### Memory ####################################


# Elasticsearch performs poorly when JVM starts swapping: you should ensure that

# it _never_ swaps.

#

# Set this property to true to lock the memory:

#

 bootstrap.mlockall: true

################################## Discovery ##################################


# Discovery infrastructure ensures nodes can be found within a cluster

# and master node is elected. Multicast discovery is the default.


# Set to ensure a node sees N other master eligible nodes to be considered

# operational within the cluster. Its recommended to set it to a higher value

# than 1 when running more than 2 nodes in the cluster.

#

 discovery.zen.minimum_master_nodes: 2


# Set the time to wait for ping responses from other nodes when discovering.

# Set this option to a higher value on a slow or congested network

# to minimize discovery failures:

#

 discovery.zen.ping.timeout: 3s


# For more information, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-zen.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFlStmj9NvbUgRiDZa5KahFaXYNyA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-zen.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFlStmj9NvbUgRiDZa5KahFaXYNyA';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>


# Unicast discovery allows to explicitly control which nodes will be used

# to discover the cluster. It can be used when multicast is not present,

# or to restrict the cluster communication-wise.

#

# 1. Disable multicast discovery (enabled by default):

#

 discovery.zen.ping.multicast.enabled: false

#

# 2. Configure an initial list of master nodes in the cluster

#    to perform discovery when new nodes (master or data) are started:

#

 discovery.zen.ping.unicast.hosts: ["IP1","IP2"]


# EC2 discovery allows to use AWS EC2 API in order to perform discovery.

#

# You have to install the cloud-aws plugin for enabling the EC2 discovery.

#

# For more information, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-ec2.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHgW5G4UOJnux2jwHNgWliJJYFPKQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-ec2.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHgW5G4UOJnux2jwHNgWliJJYFPKQ';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>

#

# See <<a href="http://elasticsearch.org/tutorials/elasticsearch-on-ec2/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Ftutorials%2Felasticsearch-on-ec2%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGFqrWbd2zRkAaLdxk3JdmkKTsTBQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Ftutorials%2Felasticsearch-on-ec2%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGFqrWbd2zRkAaLdxk3JdmkKTsTBQ';return true;">http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>

# for a step-by-step tutorial.

cloud:


     aws:

         access_key: XXXXX

         secret_key: YYYYYYYYYYY

    discovery:

        type: ec2




On Friday, July 4, 2014 2:27:48 AM UTC-4, David Pilato wrote:
Agreed. Could you share your elasticsearch.yml file without touching anything but only replacing Key/secret?

Keep the formating.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 4 juil. 2014 à 06:51, Ross Simpson <[hidden email]> a écrit :

That specific exception (com.amazonaws.AmazonClientException) is thrown by the AWS client libraries, and it means the library couldn't find your AWS credentials.  I'm not sure why, as the details in your original post look correct.

FWIW, S3 snapshots are working well for me.  Here's my setup:
ES 1.1.1
AWS cloud plugin 2.1.0

elasticsearch.yml:
cloud.aws.access_key: ...
cloud
.aws.secret_key: ......

Repo registration:
$ curl -XPUT '<a href="http://localhost:9200/_snapshot/es-backups'" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;">http://localhost:9200/_snapshot/es-backups' -d '{"type":"s3","settings":{"compress":"true","base_path":"prod_backups","region":"us-east","bucket":"..."}}'
{"acknowledged":true}

In your latest post, it looks like you're running the command on a remote ES host (10.211.154.24).  Does that specific host have the AWS credentials in its ES config?  Snapshotting will require that all nodes in the cluster have the AWS credentials, because they will each be writing to S3.

Are there any relevant entries in the ES logs from startup?  



On Friday, 4 July 2014 04:25:44 UTC+10, sabdalla80 wrote:
I installed latest ES version too "1.2.1", still getting same error
{
   "error": "RemoteTransportException[[node 2][inet[/10.211.154.24:9300]][cluster/repository/put]]; nested: RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}

Any ideas? I would appreciate some feedback on how to figure out this problem because I would like to backup our index to S3.

On Wednesday, July 2, 2014 3:36:58 PM UTC-4, sabdalla80 wrote:
Unfortunately, I tried with and without the region setting, no difference.

On Tuesday, July 1, 2014 7:43:21 PM UTC-4, Glen Smith wrote:
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;"> https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;"> https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;"> https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="FATe0_P_ycoJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/ec5d719f-514b-4224-9138-0a61370f2870%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/ec5d719f-514b-4224-9138-0a61370f2870%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/ec5d719f-514b-4224-9138-0a61370f2870%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;"> https://groups.google.com/d/msgid/elasticsearch/ec5d719f-514b-4224-9138-0a61370f2870%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db55bb02-b5a1-44c3-8692-01d2bb7efbae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53b6cb98.189a769b.2fae%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problem Configuring AWS S3 for Backups

IronMike
Good tip, I plan on doing just that for my third node.
Thanks again

On Friday, July 4, 2014 11:43:30 AM UTC-4, David Pilato wrote:
I would recommend that both nodes have that config.
If you want to add a third node, because you want to replace the first one for any reason, it's better to have all in place.

One thing you can do is to prepare a clean machine, with all settings but no data. Then create an image.

If you want to scale out, just launch new instances using that ready to use image.

My 2 cents


-- 
David Pilato | Technical Advocate | Elasticsearch.com
<a href="https://twitter.com/dadoonet" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;">@dadoonet | <a href="https://twitter.com/elasticsearchfr" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;">@elasticsearchfr


Le 4 juillet 2014 à 17:39:02, sabdalla80 (<a href="javascript:" target="_blank" gdf-obfuscated-mailto="BA3nGrZ8mtgJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">sabda...@...) a écrit:

David, Great that helped. At first I created the space in "node 1" config but didn't do anything. So, I commented the credentials section from "node 2" config, That worked. So, I am a bit confused about this, Do I need to maintain both configs on the two instances, so they both have credentials in them, or do I just put it in "node 1" and since the cluster is configured, there will be no need to have credentials in "node 2" ?

Thanks

On Friday, July 4, 2014 11:03:45 AM UTC-4, David Pilato wrote:
Try to add some spaces before cloud, so like this…


  cloud:


     aws:

         access_key: XXXXX

         secret_key: YYYYYYYYYYY

    discovery:

        type: ec2


-- 
David Pilato | Technical Advocate | Elasticsearch.com
<a href="https://twitter.com/dadoonet" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Fdadoonet\46sa\75D\46sntz\0751\46usg\75AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA';return true;"> @dadoonet | <a href="https://twitter.com/elasticsearchfr" style="color:rgb(17,85,204)" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Ftwitter.com%2Felasticsearchfr\46sa\75D\46sntz\0751\46usg\75AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA';return true;">@elasticsearchfr


Le 4 juillet 2014 à 16:47:03, sabdalla80 ([hidden email]) a écrit:

My cluster has two instances, I have the same setup on both instances. To answer Ross's question, I use the curl command to register the repository from the instance(s) itself. Do I need to do anything else on the instance(s) as far as AWS credentials?

Here is what I have:
Enter ##################### Elasticsearch Configuration Example #####################


# This file contains an overview of various configuration settings,

# targeted at operations staff. Application developers should

# consult the guide at <<a href="http://elasticsearch.org/guide" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide\46sa\75D\46sntz\0751\46usg\75AFQjCNHGVzOZXgIUDzcAhujxiAQ65nxA2Q';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide\46sa\75D\46sntz\0751\46usg\75AFQjCNHGVzOZXgIUDzcAhujxiAQ65nxA2Q';return true;">http://elasticsearch.org/guide>.

#

# The installation procedure is covered at

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup.html\46sa\75D\46sntz\0751\46usg\75AFQjCNE8aLcBxYP2ZX6HT0KcrhSMb8lCuA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup.html\46sa\75D\46sntz\0751\46usg\75AFQjCNE8aLcBxYP2ZX6HT0KcrhSMb8lCuA';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

#

# Elasticsearch comes with reasonable defaults for most settings,

# so you can try it out without bothering with configuration.

#

# Most of the time, these defaults are just fine for running a production

# cluster. If you're fine-tuning your cluster, or wondering about the

# effect of certain configuration option, please _do ask_ on the

# mailing list or IRC channel [<a href="http://elasticsearch.org/community" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fcommunity\46sa\75D\46sntz\0751\46usg\75AFQjCNGkn9EDVM8wAXyGlGk_y3ruMtKGtQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fcommunity\46sa\75D\46sntz\0751\46usg\75AFQjCNGkn9EDVM8wAXyGlGk_y3ruMtKGtQ';return true;">http://elasticsearch.org/community].


# Any element in the configuration can be replaced with environment variables

# by placing them in ${...} notation. For example:

#

# node.rack: ${RACK_ENV_VAR}


# For information on supported formats and syntax for the config file, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-configuration.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHM-VEpNYJoEoG_96VPkPm2lXp_Cg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsetup-configuration.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHM-VEpNYJoEoG_96VPkPm2lXp_Cg';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>



################################### Cluster ###################################


# Cluster name identifies your cluster for auto-discovery. If you're running

# multiple clusters on the same network, make sure you're using unique names.

#

 <a href="http://cluster.name" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fcluster.name\46sa\75D\46sntz\0751\46usg\75AFQjCNGbdss-_rGgAY2He8KM6ZUy2TdG8g';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fcluster.name\46sa\75D\46sntz\0751\46usg\75AFQjCNGbdss-_rGgAY2He8KM6ZUy2TdG8g';return true;">cluster.name: rexCluster

#################################### Node #####################################


# Node names are generated dynamically on startup, so you're relieved

# from configuring them manually. You can tie this node to a specific name:

#

 <a href="http://node.name" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fnode.name\46sa\75D\46sntz\0751\46usg\75AFQjCNE8Cff_rbggu0_UXL6VsgDVF5NFCg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fnode.name\46sa\75D\46sntz\0751\46usg\75AFQjCNE8Cff_rbggu0_UXL6VsgDVF5NFCg';return true;">node.name: "node 1"


# Every node can be configured to allow or deny being eligible as the master,

# and to allow or deny to store the data.

#

# Allow this node to be eligible as a master node (enabled by default):

#

 node.master: true

#

# Allow this node to store data (enabled by default):

#

# node.data: true


# You can exploit these settings to design advanced cluster topologies.

#

# 1. You want this node to never become a master node, only to hold data.

#    This will be the "workhorse" of your cluster.

#

# node.master: false

# node.data: true

#

# 2. You want this node to only serve as a master: to not store any data and

#    to have free resources. This will be the "coordinator" of your cluster.

#

# node.master: true

# node.data: false

#

# 3. You want this node to be neither master nor data node, but

#    to act as a "search load balancer" (fetching data from nodes,

#    aggregating results, etc.)

#

# node.master: false

# node.data: false


# Use the Cluster Health API [<a href="http://localhost:9200/_cluster/health" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_cluster%2Fhealth\46sa\75D\46sntz\0751\46usg\75AFQjCNGisf40DOD4uGeoIFwv68oiHJ0Nbg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_cluster%2Fhealth\46sa\75D\46sntz\0751\46usg\75AFQjCNGisf40DOD4uGeoIFwv68oiHJ0Nbg';return true;">http://localhost:9200/_cluster/health], the

# Node Info API [<a href="http://localhost:9200/_nodes" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_nodes\46sa\75D\46sntz\0751\46usg\75AFQjCNE99WYZY_Q1F7dQUvZcolOKUcd8Cw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_nodes\46sa\75D\46sntz\0751\46usg\75AFQjCNE99WYZY_Q1F7dQUvZcolOKUcd8Cw';return true;">http://localhost:9200/_nodes] or GUI tools

# such as <<a href="http://www.elasticsearch.org/overview/marvel/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Foverview%2Fmarvel%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFI3F2C99cAAKzrktL3jJ_ItyH5HQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Foverview%2Fmarvel%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFI3F2C99cAAKzrktL3jJ_ItyH5HQ';return true;">http://www.elasticsearch.org/overview/marvel/>,

# <<a href="http://github.com/karmi/elasticsearch-paramedic" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Fkarmi%2Felasticsearch-paramedic\46sa\75D\46sntz\0751\46usg\75AFQjCNFS5RH6734c8UlxtdcuA8CTomooXg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Fkarmi%2Felasticsearch-paramedic\46sa\75D\46sntz\0751\46usg\75AFQjCNFS5RH6734c8UlxtdcuA8CTomooXg';return true;">http://github.com/karmi/elasticsearch-paramedic>,

# <<a href="http://github.com/lukas-vlcek/bigdesk" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Flukas-vlcek%2Fbigdesk\46sa\75D\46sntz\0751\46usg\75AFQjCNES4Fnkp8Sklz8ywE8hEYD1YlcjZw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fgithub.com%2Flukas-vlcek%2Fbigdesk\46sa\75D\46sntz\0751\46usg\75AFQjCNES4Fnkp8Sklz8ywE8hEYD1YlcjZw';return true;">http://github.com/lukas-vlcek/bigdesk> and

# <<a href="http://mobz.github.com/elasticsearch-head" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmobz.github.com%2Felasticsearch-head\46sa\75D\46sntz\0751\46usg\75AFQjCNE1MKGqn-0PzflEARleUHZ-JtmtUw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmobz.github.com%2Felasticsearch-head\46sa\75D\46sntz\0751\46usg\75AFQjCNE1MKGqn-0PzflEARleUHZ-JtmtUw';return true;">http://mobz.github.com/elasticsearch-head> to inspect the cluster state.


# A node can have generic attributes associated with it, which can later be used

# for customized shard allocation filtering, or allocation awareness. An attribute

# Path to directory containing configuration (this file and logging.yml):

#

 path.conf: /etc/elasticsearch


# Path to directory where to store index data allocated for this node.

#

# path.data: /path/to/data

#

# Can optionally include more than one location, causing data to be striped across

# the locations (a la RAID 0) on a file level, favouring locations with most free

# space on creation. For example:

#

 path.data: /opt/cores/elasticsearch/data


# Path to temporary files:

#

 path.work: /opt/cores/elasticsearch/work


# Path to log files:

#

 path.logs: /opt/cores/elasticsearch/logs


# Path to where plugins are installed:

#

# path.plugins: /path/to/plugins

################################### Memory ####################################


# Elasticsearch performs poorly when JVM starts swapping: you should ensure that

# it _never_ swaps.

#

# Set this property to true to lock the memory:

#

 bootstrap.mlockall: true

################################## Discovery ##################################


# Discovery infrastructure ensures nodes can be found within a cluster

# and master node is elected. Multicast discovery is the default.


# Set to ensure a node sees N other master eligible nodes to be considered

# operational within the cluster. Its recommended to set it to a higher value

# than 1 when running more than 2 nodes in the cluster.

#

 discovery.zen.minimum_master_nodes: 2


# Set the time to wait for ping responses from other nodes when discovering.

# Set this option to a higher value on a slow or congested network

# to minimize discovery failures:

#

 discovery.zen.ping.timeout: 3s


# For more information, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-zen.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFlStmj9NvbUgRiDZa5KahFaXYNyA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-zen.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFlStmj9NvbUgRiDZa5KahFaXYNyA';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>


# Unicast discovery allows to explicitly control which nodes will be used

# to discover the cluster. It can be used when multicast is not present,

# or to restrict the cluster communication-wise.

#

# 1. Disable multicast discovery (enabled by default):

#

 discovery.zen.ping.multicast.enabled: false

#

# 2. Configure an initial list of master nodes in the cluster

#    to perform discovery when new nodes (master or data) are started:

#

 discovery.zen.ping.unicast.hosts: ["IP1","IP2"]


# EC2 discovery allows to use AWS EC2 API in order to perform discovery.

#

# You have to install the cloud-aws plugin for enabling the EC2 discovery.

#

# For more information, see

# <<a href="http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-ec2.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHgW5G4UOJnux2jwHNgWliJJYFPKQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmodules-discovery-ec2.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHgW5G4UOJnux2jwHNgWliJJYFPKQ';return true;">http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>

#

# See <<a href="http://elasticsearch.org/tutorials/elasticsearch-on-ec2/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Ftutorials%2Felasticsearch-on-ec2%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGFqrWbd2zRkAaLdxk3JdmkKTsTBQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Felasticsearch.org%2Ftutorials%2Felasticsearch-on-ec2%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGFqrWbd2zRkAaLdxk3JdmkKTsTBQ';return true;">http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>

# for a step-by-step tutorial.

cloud:


     aws:

         access_key: XXXXX

         secret_key: YYYYYYYYYYY

    discovery:

        type: ec2




On Friday, July 4, 2014 2:27:48 AM UTC-4, David Pilato wrote:
Agreed. Could you share your elasticsearch.yml file without touching anything but only replacing Key/secret?

Keep the formating.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 4 juil. 2014 à 06:51, Ross Simpson <[hidden email]> a écrit :

That specific exception (com.amazonaws.AmazonClientException) is thrown by the AWS client libraries, and it means the library couldn't find your AWS credentials.  I'm not sure why, as the details in your original post look correct.

FWIW, S3 snapshots are working well for me.  Here's my setup:
ES 1.1.1
AWS cloud plugin 2.1.0

elasticsearch.yml:
cloud.aws.access_key: ...
cloud
.aws.secret_key: ......

Repo registration:
$ curl -XPUT '<a href="http://localhost:9200/_snapshot/es-backups'" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fes-backups\47\46sa\75D\46sntz\0751\46usg\75AFQjCNF7h6gWHgxsrGgBISsf18G2J0UhCQ';return true;">http://localhost:9200/_snapshot/es-backups' -d '{"type":"s3","settings":{"compress":"true","base_path":"prod_backups","region":"us-east","bucket":"..."}}'
{"acknowledged":true}

In your latest post, it looks like you're running the command on a remote ES host (10.211.154.24).  Does that specific host have the AWS credentials in its ES config?  Snapshotting will require that all nodes in the cluster have the AWS credentials, because they will each be writing to S3.

Are there any relevant entries in the ES logs from startup?  



On Friday, 4 July 2014 04:25:44 UTC+10, sabdalla80 wrote:
I installed latest ES version too "1.2.1", still getting same error
{
   "error": "RemoteTransportException[[node 2][inet[/10.211.154.24:9300]][cluster/repository/put]]; nested: RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}

Any ideas? I would appreciate some feedback on how to figure out this problem because I would like to backup our index to S3.

On Wednesday, July 2, 2014 3:36:58 PM UTC-4, sabdalla80 wrote:
Unfortunately, I tried with and without the region setting, no difference.

On Tuesday, July 1, 2014 7:43:21 PM UTC-4, Glen Smith wrote:
I'm not sure it matters, but I noticed you aren't setting a region in either your config or when registering your repo.

On Tuesday, July 1, 2014 7:08:28 PM UTC-4, sabdalla80 wrote:
I am not sure the version is the problem, I guess I can upgrade from V1.1 to latest. 
"Not able to load credential from supply chain", Any idea this error is generated, Is there any other place that my credentials need to be besides .yml file?
Note, I am able to write/read to S3 remotely, so I don't have any priviliges problems that I can think of.

On Tuesday, July 1, 2014 4:44:17 PM UTC-4, David Pilato wrote:
I think 2.1.1 should work fine as well.

That said, you should upgrade to latest 1.1 (or 1.2)...

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 1 juil. 2014 à 22:13, Glen Smith <[hidden email]> a écrit :

According to
<a href="https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-cloud-aws%2Ftree%2Fes-1.1\46sa\75D\46sntz\0751\46usg\75AFQjCNF6af8-P5O6ZAMQKBHoMPx77FRaVw';return true;"> https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.1
you should use v2.1.0 of the plugin with ES 1.1.0.

On Tuesday, July 1, 2014 9:03:04 AM UTC-4, sabdalla80 wrote:
I am having a problem setting up backup and restore part of AWS on S3. 
I have 2.1.1 AWS plugin & ElasticSearch V1.1.0

My yml:

cloud:
    aws:
        access_key: #########
       
secret_key: #####################
   discovery
:
        type: ec2


When I try to register a repository:

PUT /_snapshot/es_repository{

"type": "s3",

"settings": {

  "bucket": "esbucket"

}}


I get this error, it complains about loading my credentials! Is this ElasticSearch problem or AWS?

Note I am running as root user "ubuntu" on Ec2 and also running AWS with root privileges as opposed to IAM role, not sure if it's a problem or not.

   "error": "RepositoryException[[es_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonClientException[Unable to load AWS credentials from any provider in the chain]; ",
   "status": 500
}ode here...





--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;"> https://groups.google.com/d/msgid/elasticsearch/e7db355a-7710-4408-80de-60960fd16d1d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;"> https://groups.google.com/d/msgid/elasticsearch/a3df1eb8-b6a2-4400-ba8c-58e1b2b33385%40googlegroups.com.
For more options, visit <a href="<a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='<a href="https://groups.google.com/d/optout';return" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout\47;return';return true;" onclick="this.href='https://groups.google.com/d/optout\47;return';return true;">https://groups.google.com/d/optout';return true;" onclick
...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4b7b69e9-6a59-480b-973e-b7069420ecd7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.