discovery.ec2.host_type:public_dns not working as expected

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

discovery.ec2.host_type:public_dns not working as expected

John Chang
I have two nodes up in ec2  configured as follows:
discovery:
  type: ec2
  initial_state_timeout: 120s
  ec2:
    host_type: public_dns
    ports: 9300
    groups: dev_private

I cannot connect to them running tests from nodes on my dev machine, trying to connect as follows:
            NodeBuilder nb = NodeBuilder.nodeBuilder().client(true);
            nb.settings().put("discovery.zen.ping.multicast.enabled", "false");
            nb.settings().put("discovery.zen.ping.unicast.hosts", "elasticsearch1.dev5.cloud.test.com:9300,elasticsearch2.dev5.cloud.test.com:9300");
            nb.settings().put("cluster.name", "dev5");
            this.indexClient = nb.node().client();

It still seems to try to join the nodes on private ip (the ip mentioned below, 10.198.109.171, is a private ip for one of the nodes):

12:42:14,757 WARN  [elasticsearch[Fury][cached]-pool-3-thread-1] [discovery.zen] - [Fury] failed to connect to master [[Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Freakshow][inet[/10.198.109.171:9300]]
Caused by: org.elasticsearch.transport.ConnectTransportException: [Freakshow][inet[/10.198.109.171:9300]]
12:42:19,275 WARN  [elasticsearch[Fury][cached]-pool-3-thread-1] [discovery.zen] - [Fury] failed to connect to master [[Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300]]], retrying...

It does that over and over until it gives up:

12:42:39,845 WARN  [main] [discovery] - [Fury] waited for 30s and no initial state was set by the discovery

Reply | Threaded
Open this post in threaded view
|

Re: discovery.ec2.host_type:public_dns not working as expected

kimchy
Administrator
Hi,

  Can you check in the logs of the nodes what is the `publish_host`? This is the one you want if you want to be able to connect to them. Currently, the ec2 discovery is not tied to the definition of what is the host the nodes will be connected to, so you will need to configure it as well.

  Note, you don't have to configure it differently for each node. Usually, en1 will be the public one (if memory serves), so you can configure the `network.host` setting to _en1_. More info here: http://www.elasticsearch.com/docs/elasticsearch/modules/node/network/.

-shay.banon

On Mon, Oct 18, 2010 at 9:47 PM, John Chang <[hidden email]> wrote:

I have two nodes up in ec2  configured as follows:
discovery:
 type: ec2
 initial_state_timeout: 120s
 ec2:
   host_type: public_dns
   ports: 9300
   groups: dev_private

I cannot connect to them running tests from nodes on my dev machine, trying
to connect as follows:
           NodeBuilder nb = NodeBuilder.nodeBuilder().client(true);
           nb.settings().put("discovery.zen.ping.multicast.enabled",
"false");
           nb.settings().put("discovery.zen.ping.unicast.hosts",
"elasticsearch1.dev5.cloud.test.com:9300,elasticsearch2.dev5.cloud.test.com:9300");
           nb.settings().put("cluster.name", "dev5");
           this.indexClient = nb.node().client();

It still seems to try to join the nodes on private ip (the ip mentioned
below, 10.198.109.171, is a private ip for one of the nodes):

12:42:14,757 WARN  [elasticsearch[Fury][cached]-pool-3-thread-1]
[discovery.zen] - [Fury] failed to connect to master
[[Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300]]],
retrying...
org.elasticsearch.transport.ConnectTransportException:
[Freakshow][inet[/10.198.109.171:9300]]
Caused by: org.elasticsearch.transport.ConnectTransportException:
[Freakshow][inet[/10.198.109.171:9300]]
12:42:19,275 WARN  [elasticsearch[Fury][cached]-pool-3-thread-1]
[discovery.zen] - [Fury] failed to connect to master
[[Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300]]],
retrying...


It does that over and over until it gives up:

12:42:39,845 WARN  [main] [discovery] - [Fury] waited for 30s and no initial
state was set by the discovery


--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/discovery-ec2-host-type-public-dns-not-working-as-expected-tp1727789p1727789.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: discovery.ec2.host_type:public_dns not working as expected

John Chang
I'm sorry, I'm not sure I understand your response, but at any rate I think I failed for two reasons to give you a correct understanding of my situation.

We are trying to connect to two ec2-hosted elasticsearch nodes from outside the cloud.  We don't have access to ec2's private IP space from our dev network, so we need the elasticsearch discovery to publish the public dns name of the instances (eg ec2-204-236-195-66.compute-1.amazonaws.com), not an IP address. That public DNS name is the only way to access the elasticsearch nodes from both
inside and outside the cloud.  My dev machine outside the cloud is the one that produced the ConnectTransportException log errors copied above.

Also, the config of the nodes in the ec2 cloud I gave you was wrong.  This is what I should have given you:
discovery:
  host_type: public_dns
  type: zen
  zen:
    ping:
      unicast:
        hosts: elasticsearch1.dev5.cloud.test.com[9300],elasticsearch2.dev5.cloud.test.com[9300]
        enabled: true
      multicast:
        enabled: false
Reply | Threaded
Open this post in threaded view
|

Re: discovery.ec2.host_type:public_dns not working as expected

kimchy
Administrator
Can you try and connect using the transport client and not the node client?

On Tue, Oct 19, 2010 at 12:18 AM, John Chang <[hidden email]> wrote:

I'm sorry, I'm not sure I understand your response, but at any rate I think I
failed for two reasons to give you a correct understanding of my situation.

We are trying to connect to two ec2-hosted elasticsearch nodes from outside
the cloud.  We don't have access to ec2's private IP space from our dev
network, so we need the elasticsearch discovery to publish the public dns
name of the instances (eg ec2-204-236-195-66.compute-1.amazonaws.com), not
an IP address. That public DNS name is the only way to access the
elasticsearch nodes from both
inside and outside the cloud.  My dev machine outside the cloud is the one
that produced the ConnectTransportException log errors copied above.

Also, the config of the nodes in the ec2 cloud I gave you was wrong.  This
is what I should have given you:
discovery:
 host_type: public_dns
 type: zen
 zen:
   ping:
     unicast:
       hosts:
elasticsearch1.dev5.cloud.test.com[9300],elasticsearch2.dev5.cloud.test.com[9300]
       enabled: true
     multicast:
       enabled: false

--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/discovery-ec2-host-type-public-dns-not-working-as-expected-tp1727789p1728737.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: discovery.ec2.host_type:public_dns not working as expected

John Chang
Still can't connect, but the errors manifest differently.

I switched to connect the node in my dev machine (outside the ec2 cloud) this way:
            indexClient = new TransportClient()
                    .addTransportAddress(new InetSocketTransportAddress("<server1>", 9300))
                    .addTransportAddress(new InetSocketTransportAddress("<server2>", 9300));

I can get through the above lines without exception, but I see in the logs:
16:24:47,658 WARN  [main] [client.transport] - [Rodstvow] Node [Jimmy Woo][06aff47a-c269-4ac0-9170-a64d4a66ced1][inet[/10.206.98.255:9300]] not part of the cluster Cluster [elasticsearch], ignoring...
16:24:47,660 WARN  [main] [client.transport] - [Rodstvow] Node [Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300]] not part of the cluster Cluster [elasticsearch], ignoring...

10.206.98.255 and 10.198.109.171 are still the private ip's only visible inside the ec2 cloud.  Looking the logs of the nodes in the cloud, I don't see any evidence that I've joined.

Then, when I search I get a
org.elasticsearch.client.transport.NoNodeAvailableException: No node available

By the way, once I get it to work with the Transport client, how do I set the cluster name?
Reply | Threaded
Open this post in threaded view
|

Re: discovery.ec2.host_type:public_dns not working as expected

kimchy
Administrator
You need to set the cluster name on the transport client so it can join the cluster you started. When you construct a new TransportClient instance, you can pass it settings, in the settings, you can place "cluster.name" with the cluster name that you started the nodes with.

You won't see the client joining the nodes, as the transport client is not part of the cluster.

-shay.banon

On Tue, Oct 19, 2010 at 1:30 AM, John Chang <[hidden email]> wrote:

Still can't connect, but the errors manifest differently.

I switched to connect the node in my dev machine (outside the ec2 cloud)
this way:
           indexClient = new TransportClient()
                   .addTransportAddress(new
InetSocketTransportAddress("<server1>", 9300))
                   .addTransportAddress(new
InetSocketTransportAddress("<server2>", 9300));

I can get through the above lines without exception, but I see in the logs:
16:24:47,658 WARN  [main] [client.transport] - [Rodstvow] Node [Jimmy
Woo][06aff47a-c269-4ac0-9170-a64d4a66ced1][inet[/10.206.98.255:9300]] not
part of the cluster Cluster [elasticsearch], ignoring...
16:24:47,660 WARN  [main] [client.transport] - [Rodstvow] Node
[Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300]]
not part of the cluster Cluster [elasticsearch], ignoring...

10.206.98.255 and 10.198.109.171 are still the private ip's only visible
inside the ec2 cloud.  Looking the logs of the nodes in the cloud, I don't
see any evidence that I've joined.

Then, when I search I get a
org.elasticsearch.client.transport.NoNodeAvailableException: No node
available

By the way, once I get it to work with the Transport client, how do I set
the cluster name?
--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/discovery-ec2-host-type-public-dns-not-working-as-expected-tp1727789p1729092.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.