EC2 Discovery

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

EC2 Discovery

IronMan2014
I can't seem to make my EC2 cluster of 2 nodes/instances work.

- If I comment out the section below, I can connect to each instance individually and query it, I am using Sense plugin to query Elastic Search Instance.

- With the sections included as below, I cannot query neither node, I get "Request failed to get to the server (status code: 0):"


//////////////////////// In /instance 1/ elasticsearch.yml////////////////
cluster.name: mycluster
node.name: "node_1"

cloud:

   aws:

       access_key: <MY ACCESSKEY>

       secret_key: <My SecretKEY>

discovery:

   type: ec2

//////////////////////// In /instance 2/ elasticsearch.yml////////////////
cluster.name: mycluster
node.name: "node_2"

cloud:

  aws:

      access_key: <MY ACCESSKEY>

      secret_key: <My SecretKEY>

discovery:

  type: ec2



I also tried with:

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["Instance_1 IP:9300", "instance_2 IP:9300"]


Here is "mycluster.log" from both instances:

$ more /var/log/elasticsearch/mycluster.log 


[2014-03-20 19:00:37,635][INFO ][node                     ] [node_1] version[0.90.10], pid[3520], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:37,635][INFO ][node                     ] [node_1] initializing ...

[2014-03-20 19:00:37,698][INFO ][plugins                  ] [node_1] loaded [mapper-attachments, cloud-aws], sites []

[2014-03-20 19:01:17,898][INFO ][node                     ] [node_1] version[0.90.10], pid[3594], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:17,898][INFO ][node                     ] [node_1] initializing ...

[2014-03-20 19:01:17,961][INFO ][plugins                  ] [node_1] loaded [mapper-attachments, cloud-aws], sites []

[2014-03-20 19:03:50,048][INFO ][node                     ] [node_1] version[0.90.10], pid[3671], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:03:50,048][INFO ][node                     ] [node_1] initializing ...

[2014-03-20 19:03:50,111][INFO ][plugins                  ] [node_1] loaded [mapper-attachments, cloud-aws], sites []


$more /var/log/elasticsearch/mycluster.log 


[2014-03-20 19:00:29,465][INFO ][node                     ] [node_2] version[0.90.10], pid[2800], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:29,466][INFO ][node                     ] [node_2] initializing ...

[2014-03-20 19:00:29,528][INFO ][plugins                  ] [node_2] loaded [mapper-attachments, cloud-aws], sites []

[2014-03-20 19:01:26,645][INFO ][node                     ] [node_2] version[0.90.10], pid[2874], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:26,646][INFO ][node                     ] [node_2] initializing ...

[2014-03-20 19:01:26,709][INFO ][plugins                  ] [node_2] loaded [mapper-attachments, cloud-aws], sites []

[2014-03-20 19:03:58,357][INFO ][node                     ] [node_2] version[0.90.10], pid[2951], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:03:58,357][INFO ][node                     ] [node_2] initializing ...

[2014-03-20 19:03:58,420][INFO ][plugins                  ] [node_2] loaded [mapper-attachments, cloud-aws], sites []



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/69b43625-f560-4e42-a987-2ba2310d462b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

EC2 Discovery

IronMike
Any clues to what i am missing, i turned discovery trace on, but dont't see any useful info.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7f3dce1e-53d1-4c38-804f-6262896d43d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

EC2 Discovery

Raphael Miranda
are both machines in the same security group?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: EC2 Discovery

Norberto Meijome

Don't try ec2 discovery until you have tested that:
- you can connect from one machine to another on port 9300 ( nc as client and server, basic networking/ firewalling)
- run a simple aws ec2 describe instances call with the API key you plan to use, and you can see the machines you need there. Bonus points for filtering based on the rules you intense to use ( sec group, tags). This is to ensure your API keys have the correct access needed.

Once you have those basic steps working, use them on es config.

Make sure you enable ec2 discovery and disable the zen discovery ( it will run first and likely time out and ec2 disco won't get to exec).

The other thing to watch out for is contacting nodes which are too busy to ack your new nodes request for cluster info...but that would be a problem with zen disco too.

On 21/03/2014 12:31 PM, "Raphael Miranda" <[hidden email]> wrote:
are both machines in the same security group?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACj2-4LjvKAW9cnkrjQUR6%3Dk8FRf%3DKzmDUAUopHLVUMNc1ixOw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: EC2 Discovery

IronMike
I am not sure if I missed something, but what you mentioned I believe I already tried as showing in my original post.
I can connect to each machine individually and I am able ti index and query it fine with default configuration without any zen or ec2 settings. But, when I turned them on like I show on the post, I get this  "Request failed to get to the server (status code: 0):" when trying to query the instance.
Did you mean I should try to see if I can access one instance from the other? This I didn't try yet.

On Friday, March 21, 2014 4:46:40 AM UTC-4, Norberto Meijome wrote:

Don't try ec2 discovery until you have tested that:
- you can connect from one machine to another on port 9300 ( nc as client and server, basic networking/ firewalling)
- run a simple aws ec2 describe instances call with the API key you plan to use, and you can see the machines you need there. Bonus points for filtering based on the rules you intense to use ( sec group, tags). This is to ensure your API keys have the correct access needed.

Once you have those basic steps working, use them on es config.

Make sure you enable ec2 discovery and disable the zen discovery ( it will run first and likely time out and ec2 disco won't get to exec).

The other thing to watch out for is contacting nodes which are too busy to ack your new nodes request for cluster info...but that would be a problem with zen disco too.

On 21/03/2014 12:31 PM, "Raphael Miranda" <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="yVN8D-AE1JYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">raphael...@...> wrote:
are both machines in the same security group?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="yVN8D-AE1JYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7c9e9da8-6efe-4005-8a69-c00daa6ec711%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: EC2 Discovery

IronMike
In reply to this post by Norberto Meijome
I am not sure if I missed something, but what you mentioned I believe I already tried as showing in my original post.
I can connect from one instance to another.
I can connect to each machine individually and I am able to index and query it fine with default configuration without any zen or ec2 settings. But, when I turned them on like I show on the post, I get this  "Request failed to get to the server (status code: 0):" when trying to query the instance, and when I do this, it won't even log anything, it is not getting that far.


On Friday, March 21, 2014 4:46:40 AM UTC-4, Norberto Meijome wrote:

Don't try ec2 discovery until you have tested that:
- you can connect from one machine to another on port 9300 ( nc as client and server, basic networking/ firewalling)
- run a simple aws ec2 describe instances call with the API key you plan to use, and you can see the machines you need there. Bonus points for filtering based on the rules you intense to use ( sec group, tags). This is to ensure your API keys have the correct access needed.

Once you have those basic steps working, use them on es config.

Make sure you enable ec2 discovery and disable the zen discovery ( it will run first and likely time out and ec2 disco won't get to exec).

The other thing to watch out for is contacting nodes which are too busy to ack your new nodes request for cluster info...but that would be a problem with zen disco too.

On 21/03/2014 12:31 PM, "Raphael Miranda" <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="yVN8D-AE1JYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">raphael...@...> wrote:
are both machines in the same security group?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="yVN8D-AE1JYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48571118-fd84-45da-9aaf-0314c936336b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: EC2 Discovery

IronMike
In reply to this post by IronMan2014
Ok, we are seeing this error in the log, any clues?

Error injecting constructor, java.lang.IllegalStateException: This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.
  at org.elasticsearch.cache.NodeCache.<init>(Unknown Source)
  while locating org.elasticsearch.cache.NodeCache
Caused by: java.lang.IllegalStateException: This is a proxy used to support circular references involving constructors.


On Thursday, March 20, 2014 3:18:11 PM UTC-4, IronMan2014 wrote:
I can't seem to make my EC2 cluster of 2 nodes/instances work.

- If I comment out the section below, I can connect to each instance individually and query it, I am using Sense plugin to query Elastic Search Instance.

- With the sections included as below, I cannot query neither node, I get "Request failed to get to the server (status code: 0):"


//////////////////////// In /instance 1/ elasticsearch.yml////////////////
cluster.name: mycluster
node.name: "node_1"

cloud:

   aws:

       access_key: <MY ACCESSKEY>

       secret_key: <My SecretKEY>

discovery:

   type: ec2

//////////////////////// In /instance 2/ elasticsearch.yml////////////////
cluster.name: mycluster
node.name: "node_2"

cloud:

  aws:

      access_key: <MY ACCESSKEY>

      secret_key: <My SecretKEY>

discovery:

  type: ec2



I also tried with:

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["Instance_1 IP:9300", "instance_2 IP:9300"]


Here is "mycluster.log" from both instances:

$ more /var/log/elasticsearch/mycluster.log 


[2014-03-20 19:00:37,635][INFO ][node                     ] [node_1] version[0.90.10], pid[3520], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:37,635][INFO ][node                     ] [node_1] initializing ...

[2014-03-20 19:00:37,698][INFO ][plugins                  ] [node_1] loaded [mapper-attachments, cloud-aws], sites []

[2014-03-20 19:01:17,898][INFO ][node                     ] [node_1] version[0.90.10], pid[3594], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:17,898][INFO ][node                     ] [node_1] initializing ...

[2014-03-20 19:01:17,961][INFO ][plugins                  ] [node_1] loaded [mapper-attachments, cloud-aws], sites []

[2014-03-20 19:03:50,048][INFO ][node                     ] [node_1] version[0.90.10], pid[3671], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:03:50,048][INFO ][node                     ] [node_1] initializing ...

[2014-03-20 19:03:50,111][INFO ][plugins                  ] [node_1] loaded [mapper-attachments, cloud-aws], sites []


$more /var/log/elasticsearch/mycluster.log 


[2014-03-20 19:00:29,465][INFO ][node                     ] [node_2] version[0.90.10], pid[2800], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:29,466][INFO ][node                     ] [node_2] initializing ...

[2014-03-20 19:00:29,528][INFO ][plugins                  ] [node_2] loaded [mapper-attachments, cloud-aws], sites []

[2014-03-20 19:01:26,645][INFO ][node                     ] [node_2] version[0.90.10], pid[2874], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:26,646][INFO ][node                     ] [node_2] initializing ...

[2014-03-<s

...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0ed56a05-1538-4a4d-91d7-8a9880ef7508%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.