EC2 discovery issue

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: EC2 discovery issue

kimchy
Administrator
Hey,

  First, it seems like all the logging is DEBUG mode, thats noisy, all you need is discovery set to TRACE.

   If you see, on both nodes, the output: "using dynamic discovery nodes" is empty, meaning that it went ahead and queries aws for the list of nodes, and got none... . Maybe the groups / tags are not correct?

-shay.banon

On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote:

Shay,
here is the gist:
https://gist.github.com/1069376

On Jul 6, 10:35 pm, Shay Banon <shay.ba...@elasticsearch.com> wrote:
Can you set logging for discovery to TRACE in the logging file and gist it?

On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote:
Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
 name: perftests

discovery:
 type: ec2
zen:
 ping_timeout: 30s

cloud:
 account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
 key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 compute:
 type: amazon

network:
 bind_host: _eth0:ipv4_

On Jul 6, 7:16 pm, Shay Banon <[hidden email] (http://elasticsearch.com)> wrote:
You need to set the discovery type to ec2:http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html. The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You can set network.host to whatever value you want, for example, the internal IP address. If you want teh same config across nodes, you can bind to the network interface name (http://www.elasticsearch.org/guide/reference/modules/network.html), for example: _en0:ipv4_.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:
Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.


my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: _non_loopback:ipv4_

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...
- NoClassSettingsException[Failed to load class setting
[discovery.type] with value [cloud]]
ClassNotFoundException[cloud]

for the network.bind_host I tried eth0:ipv4 but this gives me host
notfound exception.

Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
discovery for him with the 16.2 version?

On Jul 6, 2:40 pm, Ridvan Gyundogan <[hidden email] (http://gmail.com)> wrote:
Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <[hidden email] (http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of ElasticSearch that is currently
master on github, not a released version.

You need to download and compile it yourself:


clint

Reply | Threaded
Open this post in threaded view
|

Re: EC2 discovery issue

Pavel Penchev
Hi all,

maybe related to this discussion - I just got ES working in amazon and there was one key point, the "cloud.aws.region" setting was not optional in my case. Without it Amazon was returning an empty list of instances.
The http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html page implies that one could skip this setting.

Regards,
Pavel



On  7.07.2011 17:49, Shay Banon wrote:
Hey,

  First, it seems like all the logging is DEBUG mode, thats noisy, all you need is discovery set to TRACE.

   If you see, on both nodes, the output: "using dynamic discovery nodes" is empty, meaning that it went ahead and queries aws for the list of nodes, and got none... . Maybe the groups / tags are not correct?

-shay.banon

On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote:

Shay,
here is the gist:
https://gist.github.com/1069376

On Jul 6, 10:35 pm, Shay Banon <shay.ba...@elasticsearch.com> wrote:
Can you set logging for discovery to TRACE in the logging file and gist it?

On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote:
Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
 name: perftests

discovery:
 type: ec2
zen:
 ping_timeout: 30s

cloud:
 account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
 key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 compute:
 type: amazon

network:
 bind_host: _eth0:ipv4_

On Jul 6, 7:16 pm, Shay Banon <[hidden email] (http://elasticsearch.com)> wrote:
You need to set the discovery type to ec2:http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html. The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You can set network.host to whatever value you want, for example, the internal IP address. If you want teh same config across nodes, you can bind to the network interface name (http://www.elasticsearch.org/guide/reference/modules/network.html), for example: _en0:ipv4_.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:
Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.


my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: _non_loopback:ipv4_

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...
- NoClassSettingsException[Failed to load class setting
[discovery.type] with value [cloud]]
ClassNotFoundException[cloud]

for the network.bind_host I tried eth0:ipv4 but this gives me host
notfound exception.

Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
discovery for him with the 16.2 version?

On Jul 6, 2:40 pm, Ridvan Gyundogan <[hidden email] (http://gmail.com)> wrote:
Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <[hidden email] (http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of ElasticSearch that is currently
master on github, not a released version.

You need to download and compile it yourself:


clint


Reply | Threaded
Open this post in threaded view
|

Re: EC2 discovery issue

Ridvan Gyundogan
Hi Pavel, 
thanks a lot for the hint.

Can you probably share your working elasticsearch.yml?

On Thu, Jul 7, 2011 at 6:41 PM, Pavel Penchev <[hidden email]> wrote:
Hi all,

maybe related to this discussion - I just got ES working in amazon and there was one key point, the "cloud.aws.region" setting was not optional in my case. Without it Amazon was returning an empty list of instances.
The http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html page implies that one could skip this setting.

Regards,
Pavel



On  7.07.2011 17:49, Shay Banon wrote:
Hey,

  First, it seems like all the logging is DEBUG mode, thats noisy, all you need is discovery set to TRACE.

   If you see, on both nodes, the output: "using dynamic discovery nodes" is empty, meaning that it went ahead and queries aws for the list of nodes, and got none... . Maybe the groups / tags are not correct?

-shay.banon

On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote:

Shay,
here is the gist:
https://gist.github.com/1069376

On Jul 6, 10:35 pm, Shay Banon <shay.ba...@elasticsearch.com> wrote:
Can you set logging for discovery to TRACE in the logging file and gist it?

On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote:
Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
 name: perftests

discovery:
 type: ec2
zen:
 ping_timeout: 30s

cloud:
 account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
 key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 compute:
 type: amazon

network:
 bind_host: _eth0:ipv4_

On Jul 6, 7:16 pm, Shay Banon <[hidden email] (http://elasticsearch.com)> wrote:
You need to set the discovery type to ec2:http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html. The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You can set network.host to whatever value you want, for example, the internal IP address. If you want teh same config across nodes, you can bind to the network interface name (http://www.elasticsearch.org/guide/reference/modules/network.html), for example: _en0:ipv4_.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:
Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.


my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: _non_loopback:ipv4_

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...
- NoClassSettingsException[Failed to load class setting
[discovery.type] with value [cloud]]
ClassNotFoundException[cloud]

for the network.bind_host I tried eth0:ipv4 but this gives me host
notfound exception.

Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
discovery for him with the 16.2 version?

On Jul 6, 2:40 pm, Ridvan Gyundogan <[hidden email] (http://gmail.com)> wrote:
Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <[hidden email] (http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of ElasticSearch that is currently
master on github, not a released version.

You need to download and compile it yourself:


clint



Reply | Threaded
Open this post in threaded view
|

Re: EC2 discovery issue

Ridvan Gyundogan
Pavel you are right the cloud.aws.region was the key point. 

Here my elasticsearch.yml in case someone else needs this too:

cluster:
  name: perftests

discovery:
    type: ec2
zen:
    ping_timeout: 30s

cloud:
    account: AAAAAAAAAAAAA
    key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    compute:
        type: amazon
    aws:
        region: eu-west-1

network:
    bind_host: _eth0:ipv4_




On Fri, Jul 8, 2011 at 9:14 AM, Ridvan Gyundogan <[hidden email]> wrote:
Hi Pavel, 
thanks a lot for the hint.

Can you probably share your working elasticsearch.yml?


On Thu, Jul 7, 2011 at 6:41 PM, Pavel Penchev <[hidden email]> wrote:
Hi all,

maybe related to this discussion - I just got ES working in amazon and there was one key point, the "cloud.aws.region" setting was not optional in my case. Without it Amazon was returning an empty list of instances.
The http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html page implies that one could skip this setting.

Regards,
Pavel



On  7.07.2011 17:49, Shay Banon wrote:
Hey,

  First, it seems like all the logging is DEBUG mode, thats noisy, all you need is discovery set to TRACE.

   If you see, on both nodes, the output: "using dynamic discovery nodes" is empty, meaning that it went ahead and queries aws for the list of nodes, and got none... . Maybe the groups / tags are not correct?

-shay.banon

On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote:

Shay,
here is the gist:
https://gist.github.com/1069376

On Jul 6, 10:35 pm, Shay Banon <shay.ba...@elasticsearch.com> wrote:
Can you set logging for discovery to TRACE in the logging file and gist it?

On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote:
Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
 name: perftests

discovery:
 type: ec2
zen:
 ping_timeout: 30s

cloud:
 account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
 key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 compute:
 type: amazon

network:
 bind_host: _eth0:ipv4_

On Jul 6, 7:16 pm, Shay Banon <[hidden email] (http://elasticsearch.com)> wrote:
You need to set the discovery type to ec2:http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html. The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You can set network.host to whatever value you want, for example, the internal IP address. If you want teh same config across nodes, you can bind to the network interface name (http://www.elasticsearch.org/guide/reference/modules/network.html), for example: _en0:ipv4_.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:
Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.


my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: _non_loopback:ipv4_

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...
- NoClassSettingsException[Failed to load class setting
[discovery.type] with value [cloud]]
ClassNotFoundException[cloud]

for the network.bind_host I tried eth0:ipv4 but this gives me host
notfound exception.

Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
discovery for him with the 16.2 version?

On Jul 6, 2:40 pm, Ridvan Gyundogan <[hidden email] (http://gmail.com)> wrote:
Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <[hidden email] (http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of ElasticSearch that is currently
master on github, not a released version.

You need to download and compile it yourself:


clint




Reply | Threaded
Open this post in threaded view
|

Re: EC2 discovery issue

Alex at Ikanow
Two quick and minor addenda to this discussion

1] I have a working ec2 cluster without having set the
"cloud.aws.region" parameter. (I'm in us-east-1, not sure if that
makes any difference.) Though I suppose I'll add it now!

(I also didn't need the "network.bind_host")

2] From a previous EC2 configuration discussion (http://
groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9), I
think the "ping_timeout" parameter you're looking for is
"discovery.ec2.ping_timeout".
Reply | Threaded
Open this post in threaded view
|

Re: EC2 discovery issue

Ridvan Gyundogan
Hi Alex, thanks for the notes.
1. It was required for the europe cloud.
2. Thanks I added it to my file and I see it is waiting 30s.

I se some other differences between the 2 elasticsearch.yml files. 
For example you use cloud.aws.access_key and In mine it is cloud.account. 

On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow <[hidden email]> wrote:
Two quick and minor addenda to this discussion

1] I have a working ec2 cluster without having set the
"cloud.aws.region" parameter. (I'm in us-east-1, not sure if that
makes any difference.) Though I suppose I'll add it now!

(I also didn't need the "network.bind_host")

2] From a previous EC2 configuration discussion (http://
<a href="http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/ 2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9" target="_blank">groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9), I
think the "ping_timeout" parameter you're looking for is
"discovery.ec2.ping_timeout".

Reply | Threaded
Open this post in threaded view
|

Re: EC2 discovery issue

kimchy
Administrator
Probably the region is hte problem then, if you set it to the euro one, does it work then? Set cloud.aws.region to eu-west.

The cloud.aws.account is a global setting that defines the account access key across the different aws services used in ES. Its the preferable way to set it, though one can set explicitly the ec2 access key if needed.

On Saturday, July 9, 2011 at 11:42 AM, Ridvan Gyundogan wrote:

Hi Alex, thanks for the notes.
1. It was required for the europe cloud.
2. Thanks I added it to my file and I see it is waiting 30s.

I se some other differences between the 2 elasticsearch.yml files. 
For example you use cloud.aws.access_key and In mine it is cloud.account. 

On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow <[hidden email]> wrote:
Two quick and minor addenda to this discussion

1] I have a working ec2 cluster without having set the
"cloud.aws.region" parameter. (I'm in us-east-1, not sure if that
makes any difference.) Though I suppose I'll add it now!

(I also didn't need the "network.bind_host")

2] From a previous EC2 configuration discussion (http://
<a href="http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/ 2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9" target="_blank">groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9), I
think the "ping_timeout" parameter you're looking for is
"discovery.ec2.ping_timeout".


Reply | Threaded
Open this post in threaded view
|

Re: EC2 discovery issue

Ridvan Gyundogan
Yes with cloud.aws.region = eu-west

On Sun, Jul 10, 2011 at 3:31 AM, Shay Banon <[hidden email]> wrote:
Probably the region is hte problem then, if you set it to the euro one, does it work then? Set cloud.aws.region to eu-west.

The cloud.aws.account is a global setting that defines the account access key across the different aws services used in ES. Its the preferable way to set it, though one can set explicitly the ec2 access key if needed.

On Saturday, July 9, 2011 at 11:42 AM, Ridvan Gyundogan wrote:

Hi Alex, thanks for the notes.
1. It was required for the europe cloud.
2. Thanks I added it to my file and I see it is waiting 30s.

I se some other differences between the 2 elasticsearch.yml files. 
For example you use cloud.aws.access_key and In mine it is cloud.account. 

On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow <[hidden email]> wrote:
Two quick and minor addenda to this discussion

1] I have a working ec2 cluster without having set the
"cloud.aws.region" parameter. (I'm in us-east-1, not sure if that
makes any difference.) Though I suppose I'll add it now!

(I also didn't need the "network.bind_host")

2] From a previous EC2 configuration discussion (http://
groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9
), I
think the "ping_timeout" parameter you're looking for is
"discovery.ec2.ping_timeout".



Reply | Threaded
Open this post in threaded view
|

Re: EC2 discovery issue

Ridvan Gyundogan
Sorry, I mean with cloud.aws.region = eu-west-1 it works. 

On Sun, Jul 10, 2011 at 11:05 AM, Ridvan Gyundogan <[hidden email]> wrote:
Yes with cloud.aws.region = eu-west


On Sun, Jul 10, 2011 at 3:31 AM, Shay Banon <[hidden email]> wrote:
Probably the region is hte problem then, if you set it to the euro one, does it work then? Set cloud.aws.region to eu-west.

The cloud.aws.account is a global setting that defines the account access key across the different aws services used in ES. Its the preferable way to set it, though one can set explicitly the ec2 access key if needed.

On Saturday, July 9, 2011 at 11:42 AM, Ridvan Gyundogan wrote:

Hi Alex, thanks for the notes.
1. It was required for the europe cloud.
2. Thanks I added it to my file and I see it is waiting 30s.

I se some other differences between the 2 elasticsearch.yml files. 
For example you use cloud.aws.access_key and In mine it is cloud.account. 

On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow <[hidden email]> wrote:
Two quick and minor addenda to this discussion

1] I have a working ec2 cluster without having set the
"cloud.aws.region" parameter. (I'm in us-east-1, not sure if that
makes any difference.) Though I suppose I'll add it now!

(I also didn't need the "network.bind_host")

2] From a previous EC2 configuration discussion (http://
groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9
), I
think the "ping_timeout" parameter you're looking for is
"discovery.ec2.ping_timeout".




12