Question about Logstash Joining ES Cluster and Index

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Question about Logstash Joining ES Cluster and Index

@SQLBigG
My setup:

Logstash Node parsing NXLog eventlogs from windows servers and sending them to a 3 node ES Cluster.

Here's my logstash conf file:

input {
    tcp {
        type   => "eventlog"
        host   => "logstash01"
        port   => 3515
        codec => 'json'
    }
}

output {
      elasticsearch
        {
        index => "logstash-events-np-%{+YYYY-MM-dd}"
        }
}


Here is the elasticsearch.yml file on my logstash server:

cluster.name: ELCluster
node.name: logstash01
discovery.zen.ping.unicasthosts: ["elsearch01", "elsearch02"] (master nodes)

So when I look at HEAD, I can see that the logstash node has joined the cluster successfully, but my index is not being built.  In the logstash log, I see the following:

log4j, [2014-11-14T13:22:49.842]  WARN: org.elasticsearch.discovery: [logstash-logstash01-29863-2036] waited for 30s and no initial state was set by the discovery

Not sure why it's not building my index as I know there is data coming in from about 50 servers or so.  I'm a little lost at this point as I was able to get this far and don't know if I am missing a setting somewhere.  All servers are on the same network segment and there are no firewalls on the servers or between the servers.  I was able to build basic indexes without an issue from the ES cluster using the basic commands so I know it has the ability to create an index.

Originally, the current logstash server and elasticsearch server were our PoC servers and were successful and now we wanted to build a cluster for ES since it was going to be handling a large amount of data.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8166101a-4052-4ef0-be30-672aaefaa85d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Question about Logstash Joining ES Cluster and Index

InquiringMind
I highly recommend that you use the HTTP output. Works great, is immune to the ES version, and there are no performance issues that I've seen. It Just Works. 

For example, here's my sample logstash configuration's output settings:

output {
  # Uncomment for testing only:
  # stdout { codec => rubydebug }

  # Elasticsearch
  elasticsearch {
     # Specify http (with or without quotes around http) to direct the
     # output as JSON documents via the Elasticsearch HTTP REST API
     protocol => "http"
     codec => json
     manage_template => false

     # Or whatever target ES host is required
     host => "localhost"

     # Or whatever _type is desired:
     index_type => "sample"
  }
}

As you can probably surmise, I have my own default index creation template so there's no need to splatter it all over creation; logstash runs better on the host on which it's gathering the log files and I vastly prefer one central index template than keeping a bazillion logstash configurations in perfect sync. And if we happen replace logstash for something else, then I still have my index creation templates.

Hope this helps!

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/27854489-1f4d-4ebd-883c-64dc6235eed4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Question about Logstash Joining ES Cluster and Index

@SQLBigG
In reply to this post by @SQLBigG
Brian,

I've tried using the http protocol and it did not work.  The Cluster sees the logstash node, but it's not creating the index still.

Thanks,
Gerald

On Friday, November 14, 2014 4:26:22 PM UTC-5, @SQLBigG wrote:
My setup:

Logstash Node parsing NXLog eventlogs from windows servers and sending them to a 3 node ES Cluster.

Here's my logstash conf file:

input {
    tcp {
        type   => "eventlog"
        host   => "logstash01"
        port   => 3515
        codec => 'json'
    }
}

output {
      elasticsearch
        {
        index => "logstash-events-np-%{+YYYY-MM-dd}"
        }
}


Here is the elasticsearch.yml file on my logstash server:

<a href="http://cluster.name/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fcluster.name%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHuKo6ONeb7aSoV1uK3RDg4njnMvg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fcluster.name%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHuKo6ONeb7aSoV1uK3RDg4njnMvg';return true;">cluster.name: ELCluster
<a href="http://node.name/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fnode.name%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGFl3la7m1q_IbK2emMUtgBON_mXg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fnode.name%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGFl3la7m1q_IbK2emMUtgBON_mXg';return true;">node.name: logstash01
discovery.zen.ping.unicasthosts: ["elsearch01", "elsearch02"] (master nodes)

So when I look at HEAD, I can see that the logstash node has joined the cluster successfully, but my index is not being built.  In the logstash log, I see the following:

log4j, [2014-11-14T13:22:49.842]  WARN: org.elasticsearch.discovery: [logstash-logstash01-29863-2036] waited for 30s and no initial state was set by the discovery

Not sure why it's not building my index as I know there is data coming in from about 50 servers or so.  I'm a little lost at this point as I was able to get this far and don't know if I am missing a setting somewhere.  All servers are on the same network segment and there are no firewalls on the servers or between the servers.  I was able to build basic indexes without an issue from the ES cluster using the basic commands so I know it has the ability to create an index.

Originally, the current logstash server and elasticsearch server were our PoC servers and were successful and now we wanted to build a cluster for ES since it was going to be handling a large amount of data.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1d852a5d-8621-4b76-b127-c7a886c00561%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.