Elasticsearch: 2-node cluster with failover

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Elasticsearch: 2-node cluster with failover

prakhar
I am using Elasticsearch 1.5.2 and trying to setup a 2-node cluster. These 2 nodes are primarily for failover strategy (if any one node goes down, the other one is still there to handle requests), I don't need to divide primary shards or something like that, (total data is no more than 500mb on hard-disk).

Everything goes well, until Split Brains thing kicks in. Now, since I don't have much data, I don't feel any requirement of 3 nodes. And I want to have failover mechanism too. Which means, discovery.zen.minimum_master_nodes cannot be more than 1.

Now, I have two questions:

 1. Is there any configuration possible, which could overcome 2 master nodes or Split Brains problem?
 2. If not, what all other options do I have to make it work? Like, keeping both in different clusters (one online, other one offline) and updating offline with online, time to time, for the time when online cluster goes down. Or, do I have to go for 3-node cluster?

I am going on production environment. Please help.

Env Info: Two machines (VMWare driven), having 4 cores, 8gb RAM and CentOS. We are firing 7-8 queries (1 _search and other _count) per search. Indexing op won't be very frequent (until we sync existing indices from DB through _bulk API), but searching is going to be quite frequent.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch: 2-node cluster with failover

dadoonet
Add another node (master only : data false) with minimal HEAP and set minimum master nodes to 2
or
set minimum master nodes to 2 but in that case your cluster will stop working in case of failure of any node or network failure


-- 
David Pilato - Developer | Evangelist 





Le 28 mai 2015 à 10:40, prakhar <[hidden email]> a écrit :

I am using *Elasticsearch 1.5.2* and trying to setup a 2-node cluster. These
2 nodes are primarily for failover strategy (if any one node goes down, the
other one is still there to handle requests), I don't need to divide primary
shards or something like that, (total data is no more than 500mb on
hard-disk).

Everything goes well, until *Split Brains* thing kicks in. Now, since I
don't have much data, I don't feel any requirement of 3 nodes. And I want to
have failover mechanism too. Which means,
/discovery.zen.minimum_master_nodes/ cannot be more than /1/.

Now, I have two questions:

1. Is there any configuration possible, which could overcome *2 master
nodes* or *Split Brains *problem?
2. If not, what all other options do I have to make it work? Like, keeping
both in different clusters (one online, other one offline) and updating
offline with online, time to time, for the time when online cluster goes
down. Or, do I have to go for 3-node cluster?

I am going on production environment. Please help.

Env Info: Two machines (VMWare driven), having 4 cores, 8gb RAM and CentOS.
We are firing 7-8 queries (1 /_search/ and other /_count/) per search.
Indexing op won't be very frequent (until we sync existing indices from DB
through /_bulk/ API), but searching is going to be quite frequent.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-2-node-cluster-with-failover-tp4074949.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1432802400465-4074949.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/42DC41A9-8F90-4BB9-BBBC-35D57105A809%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch: 2-node cluster with failover

prakhar
Thanks for your reply.

If I add one master only node on one of the two existing servers (2 nodes on the same physical server), with HEAP, lets say, 100mb; will that work? Or I have to add another machine for that?
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch: 2-node cluster with failover

Mark Walkom-2
That's not enough heap, you want 1GB minimum.


PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions!

On 28 May 2015 at 20:25, prakhar <[hidden email]> wrote:
Thanks for your reply.

If I add one *master only* node on one of the two existing servers (2 nodes
on the same physical server), with *HEAP*, lets say, /100mb/; will that
work? Or I have to add another machine for that?



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-2-node-cluster-with-failover-tp4074949p4074961.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1432808731045-4074961.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8ZLP5A1k80t-D61CKLZNZ%3D8PdP4w8tsA7_O4eBWdYq4A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch: 2-node cluster with failover

Magnus Bäck
In reply to this post by prakhar
On Thursday, May 28, 2015 at 12:25 CEST,
     prakhar <[hidden email]> wrote:

> If I add one *master only* node on one of the two existing servers (2
> nodes on the same physical server), with *HEAP*, lets say, /100mb/;
> will that work? Or I have to add another machine for that?

That should work, but it obviously means that if the machine with two
Elasticsearch nodes is taken out your cluster will be inoperable (but
it will survive if the one-node machine dies).

--
Magnus Bäck                | Software Engineer, Development Tools
[hidden email] | Sony Mobile Communications

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150528110806.GA25289%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch: 2-node cluster with failover

prakhar
In reply to this post by Mark Walkom-2
Sorry to bother you @Mark, but this term appropriate heap size always haunts me. Suppose, I have 2 virtual machines (4 core, 8 gb RAM each).
According to elasticsearch's documentation, half of RAM should be allocated to elasticsearch. So, this is what I came up with:

Node1: Elasticsearch 1.5.2 Master-Data node with 4 gb HEAP (server1)
Node2: Elasticsearch 1.5.2 Master-Only(no data) node with 1 gb HEAP (server1)
Node3: Elasticsearch 1.5.2 Master-Data node with 4 gb HEAP (server2)

Do you think this is a sane strategy? Moreover, if I make Master-Only node (Node2) invisible to load balancer i.e. I simply won't include it at all, will it still require 1 gb HEAP?
Please forgive me if I am asking something too naive.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch: 2-node cluster with failover

Mark Walkom-2
Master only nodes don't need to leverage FS caching as they hold no data. So you can easily increase their heap to 75% of system.
However you really don't want to have less than 1GB of heap irrespective of the role it plays.


PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions!

On 28 May 2015 at 23:11, prakhar <[hidden email]> wrote:
Sorry to bother you @Mark, but this term *appropriate heap size* always
haunts me. Suppose, I have 2 virtual machines (4 core, 8 gb RAM each).
According to elasticsearch's documentation, half of RAM should be allocated
to elasticsearch. So, this is what I came up with:

Node1: Elasticsearch 1.5.2 *Master-Data* node with 4 gb HEAP (*server1*)
Node2: Elasticsearch 1.5.2 *Master-Only*(no data) node with 1 gb HEAP
(*server1*)
Node3: Elasticsearch 1.5.2 *Master-Data* node with 4 gb HEAP (*server2*)

Do you think this is a sane strategy? Moreover, if I make *Master-Only* node
(Node2) invisible to load balancer i.e. I simply won't include it at all,
will it still require 1 gb HEAP?
Please forgive me if I am asking something too naive.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-2-node-cluster-with-failover-tp4074949p4074968.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1432818693390-4074968.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_Oj_d3qG%3DQoTW0A_ChjdXigoj2Rn2iOQurnGpCcU21dQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.