Elasticsearch crashed after start

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Elasticsearch crashed after start

Ann Yablunovskaya
Hi!

I don't understand, what happend.
OS CentOS 7.1
I have ES cluster with two servers.
It have the same configuration.

I tried to configure shield and marvel but my second ES instanse have suddenly crashed.


[root@server bin]# ./elasticsearch -v
Version: 1.5.1, Build: 5e38401/2015-04-09T13:41:35Z, JVM: 1.7.0_79

[root@server bin]# service elasticsearch-server start
Redirecting to /bin/systemctl start  elasticsearch-server.service

[root@server bin]# service elasticsearch-server status
Redirecting to /bin/systemctl status  elasticsearch-server.service
elasticsearch-server.service - Starts and stops a single elasticsearch instance on this system
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch-server.service; enabled)
   Active: active (running) since Thu 2015-04-23 21:50:09 EDT; 5s ago
     Docs: http://www.elasticsearch.org
  Process: 15958 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch-server.pid -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR (code=exited, status=0/SUCCESS)
 Main PID: 15964 (java)
   CGroup: /system.slice/elasticsearch-server.service
           └─15964 java -Xms30g -Xmx30g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiating...

Apr 23 21:50:08 server.local systemd[1]: PID file /var/run/elasticsearch/elasticsearch-server.pid not readable (yet?) after start.
Apr 23 21:50:09 server.local systemd[1]: Started Starts and stops a single elasticsearch instance on this system.

[root@server bin]# telnet localhost 9200
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

[root@server bin]# service elasticsearch-server status
Redirecting to /bin/systemctl status  elasticsearch-server.service
elasticsearch-server.service - Starts and stops a single elasticsearch instance on this system
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch-server.service; enabled)
   Active: failed (Result: exit-code) since Thu 2015-04-23 21:50:15 EDT; 16s ago
     Docs: http://www.elasticsearch.org
  Process: 15958 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch-server.pid -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR (code=exited, status=0/SUCCESS)
 Main PID: 15964 (code=exited, status=3)

Apr 23 21:50:08 server.local systemd[1]: PID file /var/run/elasticsearch/elasticsearch-server.pid not readable (yet?) after start.
Apr 23 21:50:09 server.local.local systemd[1]: Started Starts and stops a single elasticsearch instance on this system.
Apr 23 21:50:15 server.local.local systemd[1]: elasticsearch-server.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
Apr 23 21:50:15 server.local.local systemd[1]: Unit elasticsearch-server.service entered failed state.

I will be really appreciate any help!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b8029de2-a170-49d2-81fd-9f8ec963d1b9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

es.yml (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch crashed after start

Mark Walkom-2
What do the logs show?

On 24 April 2015 at 12:03, Ann Yablunovskaya <[hidden email]> wrote:
Hi!

I don't understand, what happend.
OS CentOS 7.1
I have ES cluster with two servers.
It have the same configuration.

I tried to configure shield and marvel but my second ES instanse have suddenly crashed.


[root@server bin]# ./elasticsearch -v
Version: 1.5.1, Build: 5e38401/2015-04-09T13:41:35Z, JVM: 1.7.0_79

[root@server bin]# service elasticsearch-server start
Redirecting to /bin/systemctl start  elasticsearch-server.service

[root@server bin]# service elasticsearch-server status
Redirecting to /bin/systemctl status  elasticsearch-server.service
elasticsearch-server.service - Starts and stops a single elasticsearch instance on this system
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch-server.service; enabled)
   Active: active (running) since Thu 2015-04-23 21:50:09 EDT; 5s ago
  Process: 15958 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch-server.pid -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR (code=exited, status=0/SUCCESS)
 Main PID: 15964 (java)
   CGroup: /system.slice/elasticsearch-server.service
           └─15964 java -Xms30g -Xmx30g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiating...

Apr 23 21:50:08 server.local systemd[1]: PID file /var/run/elasticsearch/elasticsearch-server.pid not readable (yet?) after start.
Apr 23 21:50:09 server.local systemd[1]: Started Starts and stops a single elasticsearch instance on this system.

[root@server bin]# telnet localhost 9200
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

[root@server bin]# service elasticsearch-server status
Redirecting to /bin/systemctl status  elasticsearch-server.service
elasticsearch-server.service - Starts and stops a single elasticsearch instance on this system
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch-server.service; enabled)
   Active: failed (Result: exit-code) since Thu 2015-04-23 21:50:15 EDT; 16s ago
  Process: 15958 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch-server.pid -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR (code=exited, status=0/SUCCESS)
 Main PID: 15964 (code=exited, status=3)

Apr 23 21:50:08 server.local systemd[1]: PID file /var/run/elasticsearch/elasticsearch-server.pid not readable (yet?) after start.
Apr 23 21:50:09 server.local.local systemd[1]: Started Starts and stops a single elasticsearch instance on this system.
Apr 23 21:50:15 server.local.local systemd[1]: elasticsearch-server.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
Apr 23 21:50:15 server.local.local systemd[1]: Unit elasticsearch-server.service entered failed state.

I will be really appreciate any help!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b8029de2-a170-49d2-81fd-9f8ec963d1b9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-kcSan1Nfr9%2B2KiRBc5G%3D1nYJTDXP95RS3Rt0SZNkO%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch crashed after start

Ann Yablunovskaya
In reply to this post by Ann Yablunovskaya
[2015-04-24 03:38:04,501][INFO ][node                     ] [server] version[1.5.1], pid[29957], build[5e38401/2015-04-09T13:41:35Z]
[2015-04-24 03:38:04,502][INFO ][node                     ] [server] initializing ...
[2015-04-24 03:38:04,536][INFO ][plugins                  ] [server] loaded [marvel, shield, license], sites [marvel]
[2015-04-24 03:38:04,562][ERROR][bootstrap                ] Exception
org.elasticsearch.ElasticsearchIllegalStateException: Failed to created node environment
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:162)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:213)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.nio.file.AccessDeniedException: /opt/elasticsearch/data/server/cluster/nodes/1
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:383)
        at java.nio.file.Files.createDirectory(Files.java:630)
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
        at java.nio.file.Files.createDirectories(Files.java:720)
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:105)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:160)
        ... 4 more

I fix this problem, sorry. The reason was in deleting elasticsearch user and missing file privileges. So, I changed file permissions and it works

пятница, 24 апреля 2015 г., 5:03:39 UTC+3 пользователь Ann Yablunovskaya написал:
Hi!

I don't understand, what happend.
OS CentOS 7.1
I have ES cluster with two servers.
It have the same configuration.

I tried to configure shield and marvel but my second ES instanse have suddenly crashed.


[root@server bin]# ./elasticsearch -v
Version: 1.5.1, Build: 5e38401/2015-04-09T13:41:35Z, JVM: 1.7.0_79

[root@server bin]# service elasticsearch-server start
Redirecting to /bin/systemctl start  elasticsearch-server.service

[root@server bin]# service elasticsearch-server status
Redirecting to /bin/systemctl status  elasticsearch-server.service
elasticsearch-server.service - Starts and stops a single elasticsearch instance on this system
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch-server.service; enabled)
   Active: active (running) since Thu 2015-04-23 21:50:09 EDT; 5s ago
     Docs: <a href="http://www.elasticsearch.org" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org\46sa\75D\46sntz\0751\46usg\75AFQjCNF_2NkEOVUWj108OD1X2pFTATnong';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org\46sa\75D\46sntz\0751\46usg\75AFQjCNF_2NkEOVUWj108OD1X2pFTATnong';return true;">http://www.elasticsearch.org
  Process: 15958 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch-server.pid -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR (code=exited, status=0/SUCCESS)
 Main PID: 15964 (java)
   CGroup: /system.slice/elasticsearch-server.service
           └─15964 java -Xms30g -Xmx30g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiating...

Apr 23 21:50:08 server.local systemd[1]: PID file /var/run/elasticsearch/elasticsearch-server.pid not readable (yet?) after start.
Apr 23 21:50:09 server.local systemd[1]: Started Starts and stops a single elasticsearch instance on this system.

[root@server bin]# telnet localhost 9200
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address <a href="http://127.0.0.1" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2F127.0.0.1\46sa\75D\46sntz\0751\46usg\75AFQjCNFPHMKRHWJMzqOSHIe8RUvnisW-4A';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2F127.0.0.1\46sa\75D\46sntz\0751\46usg\75AFQjCNFPHMKRHWJMzqOSHIe8RUvnisW-4A';return true;">127.0.0.1: Connection refused

[root@server bin]# service elasticsearch-server status
Redirecting to /bin/systemctl status  elasticsearch-server.service
elasticsearch-server.service - Starts and stops a single elasticsearch instance on this system
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch-server.service; enabled)
   Active: failed (Result: exit-code) since Thu 2015-04-23 21:50:15 EDT; 16s ago
     Docs: <a href="http://www.elasticsearch.org" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org\46sa\75D\46sntz\0751\46usg\75AFQjCNF_2NkEOVUWj108OD1X2pFTATnong';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org\46sa\75D\46sntz\0751\46usg\75AFQjCNF_2NkEOVUWj108OD1X2pFTATnong';return true;">http://www.elasticsearch.org
  Process: 15958 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch-server.pid -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR (code=exited, status=0/SUCCESS)
 Main PID: 15964 (code=exited, status=3)

Apr 23 21:50:08 server.local systemd[1]: PID file /var/run/elasticsearch/elasticsearch-server.pid not readable (yet?) after start.
Apr 23 21:50:09 server.local.local systemd[1]: Started Starts and stops a single elasticsearch instance on this system.
Apr 23 21:50:15 server.local.local systemd[1]: elasticsearch-server.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
Apr 23 21:50:15 server.local.local systemd[1]: Unit elasticsearch-server.service entered failed state.

I will be really appreciate any help!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9d87dbab-75fb-4d40-bb26-131138a88ac0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch crashed after start

kirandhareshwar
This post has NOT been accepted by the mailing list yet.
Hi,

I am facing the same issue.  please help!. Kindly let me know which file permissions to change to solve below issue.


[root@server1 init.d]# systemctl status elasticsearch
elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
   Active: failed (Result: exit-code) since Fri 2015-06-26 02:56:08 EDT; 1h 50min ago
     Docs: http://www.elastic.co
  Process: 21063 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=$PID_DIR/elasticsearch.pid -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.config=$CONF_FILE -Des.default.path.conf=$CONF_DIR (code=exited, status=3)
 Main PID: 21063 (code=exited, status=3)
   CGroup: /system.slice/elasticsearch.service

Jun 26 02:56:07 server1.tsp.nsn.com systemd[1]: Started Elasticsearch.
Jun 26 02:56:08 server1.tsp.nsn.com elasticsearch[21063]: {1.6.0}: Initialization Failed ...
Jun 26 02:56:08 server1.tsp.nsn.com elasticsearch[21063]: - ElasticsearchIllegalStateException[Failed to created node environment]
Jun 26 02:56:08 server1.tsp.nsn.com elasticsearch[21063]: AccessDeniedException[/var/lib/elasticsearch/elasticsearch/nodes/2]
Jun 26 02:56:08 server1.tsp.nsn.com systemd[1]: elasticsearch.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 26 02:56:08 server1.tsp.nsn.com systemd[1]: Unit elasticsearch.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.