Problem with keeping in sync Elasticsearch across two data centers

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with keeping in sync Elasticsearch across two data centers

Dario Rossi
Hi, 
I've the following problem: our application publishes content to an Elasticsearch cluster. We use local data less node for querying elasticsearch then, so we don't use HTTP REST and the local nodes are the loadbalancer. Now they came with the requirement of having the cluster replicated to another data center too (and in the future maybe another too... ) for resilience. 

At the very beginning we thought of having one large cluster that goes across data centers (crazy). This solution has the following problems:

- The cluster has the split-brain problem (!)
- The client data less node will try to do requests across different data centers (is there a solution to this???). I can't find a way to avoid this. We don't want this to happen because of a) latency and b) firewalling issues.

So we started to think that this solution is not really viable. So we thought of having one cluster per data center, which seems more sensible. But then here we have the problem that we must publish data to all clusters and, if one fails, we have no means of rolling back (unless we try to set up a complicated version based rollback system). I find this very complicated and hard to maintain, although can be somewhat doable. 

My biggest problem is that we have to keep the data centers in the same state at any time, so that if one goes down, we can readily switch to the other.

Any ideas, or can you recommend some support to help use deal with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: Problem with keeping in sync Elasticsearch across two data centers

Michael Sick
Dario,


ES is not built to consistently cluster across DC's / larger network lags. 

On Fri, Feb 21, 2014 at 11:24 AM, Dario Rossi <[hidden email]> wrote:
Hi, 
I've the following problem: our application publishes content to an Elasticsearch cluster. We use local data less node for querying elasticsearch then, so we don't use HTTP REST and the local nodes are the loadbalancer. Now they came with the requirement of having the cluster replicated to another data center too (and in the future maybe another too... ) for resilience. 

At the very beginning we thought of having one large cluster that goes across data centers (crazy). This solution has the following problems:

- The cluster has the split-brain problem (!)
- The client data less node will try to do requests across different data centers (is there a solution to this???). I can't find a way to avoid this. We don't want this to happen because of a) latency and b) firewalling issues.

So we started to think that this solution is not really viable. So we thought of having one cluster per data center, which seems more sensible. But then here we have the problem that we must publish data to all clusters and, if one fails, we have no means of rolling back (unless we try to set up a complicated version based rollback system). I find this very complicated and hard to maintain, although can be somewhat doable. 

My biggest problem is that we have to keep the data centers in the same state at any time, so that if one goes down, we can readily switch to the other.

Any ideas, or can you recommend some support to help use deal with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: Problem with keeping in sync Elasticsearch across two data centers

Amit Soni
Hello Michael - Understand that ES is not built to maintain consistent cluster state across data centers. what I am wondering is whether there is a way for ElasticSearch to continue to replicate data onto a different data center (with some delay of course) so that when the primary center fails, the fail over data center still has most of the data (may be except for the last few seconds/minutes/hours).

Overall I am looking for a right way to implement cross data center deployment of elastic-search!

-Amit.


On Fri, Feb 21, 2014 at 9:37 AM, Michael Sick <[hidden email]> wrote:
Dario,


ES is not built to consistently cluster across DC's / larger network lags. 

On Fri, Feb 21, 2014 at 11:24 AM, Dario Rossi <[hidden email]> wrote:
Hi, 
I've the following problem: our application publishes content to an Elasticsearch cluster. We use local data less node for querying elasticsearch then, so we don't use HTTP REST and the local nodes are the loadbalancer. Now they came with the requirement of having the cluster replicated to another data center too (and in the future maybe another too... ) for resilience. 

At the very beginning we thought of having one large cluster that goes across data centers (crazy). This solution has the following problems:

- The cluster has the split-brain problem (!)
- The client data less node will try to do requests across different data centers (is there a solution to this???). I can't find a way to avoid this. We don't want this to happen because of a) latency and b) firewalling issues.

So we started to think that this solution is not really viable. So we thought of having one cluster per data center, which seems more sensible. But then here we have the problem that we must publish data to all clusters and, if one fails, we have no means of rolling back (unless we try to set up a complicated version based rollback system). I find this very complicated and hard to maintain, although can be somewhat doable. 

My biggest problem is that we have to keep the data centers in the same state at any time, so that if one goes down, we can readily switch to the other.

Any ideas, or can you recommend some support to help use deal with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: Problem with keeping in sync Elasticsearch across two data centers

Ivan Beveridge
Hi Amit,

It sounds like you need separate ES clusters (one per DC) and a way to
feed the data into them all consistently.

I happened to scan-read the tribenodes documentation - it looks like it
could work great for reads, but (IIRC) it will not do writes.

I suspect you want some message-passing system (eg RabbitMQ) or redis
(acting as a cache).

If you were writing (system) logs then something like logstash would
help interface. However, I suspect that is not the case so you would
need to find the integration solution (between message-passing / redis
and ES) that you need for your system.

This means that you could probably use something like tribenodes for the
reads and some message-passing/proxy system for writes.

Cheers

Ivan

On 22/02/2014 18:32, Amit Soni wrote:

> Hello Michael - Understand that ES is not built to maintain consistent
> cluster state across data centers. what I am wondering is whether there
> is a way for ElasticSearch to continue to replicate data onto a
> different data center (with some delay of course) so that when the
> primary center fails, the fail over data center still has most of the
> data (may be except for the last few seconds/minutes/hours).
>
> Overall I am looking for a right way to implement cross data center
> deployment of elastic-search!
>
> -Amit.
>
>
> On Fri, Feb 21, 2014 at 9:37 AM, Michael Sick
> <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Dario,
>
>     I believe that you're looking for
>     TribeNodes http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html
>
>     ES is not built to consistently cluster across DC's / larger network
>     lags.
>
>     On Fri, Feb 21, 2014 at 11:24 AM, Dario Rossi <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>         Hi,
>         I've the following problem: our application publishes content to
>         an Elasticsearch cluster. We use local data less node for
>         querying elasticsearch then, so we don't use HTTP REST and the
>         local nodes are the loadbalancer. Now they came with the
>         requirement of having the cluster replicated to another data
>         center too (and in the future maybe another too... ) for
>         resilience.
>
>         At the very beginning we thought of having one large cluster
>         that goes across data centers (crazy). This solution has the
>         following problems:
>
>         - The cluster has the split-brain problem (!)
>         - The client data less node will try to do requests across
>         different data centers (is there a solution to this???). I can't
>         find a way to avoid this. We don't want this to happen because
>         of a) latency and b) firewalling issues.
>
>         So we started to think that this solution is not really viable.
>         So we thought of having one cluster per data center, which seems
>         more sensible. But then here we have the problem that we must
>         publish data to all clusters and, if one fails, we have no means
>         of rolling back (unless we try to set up a complicated version
>         based rollback system). I find this very complicated and hard to
>         maintain, although can be somewhat doable.
>
>         My biggest problem is that we have to keep the data centers in
>         the same state at any time, so that if one goes down, we can
>         readily switch to the other.
>
>         Any ideas, or can you recommend some support to help use deal
>         with this?


--
Ivan Beveridge

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53091254.7020001%40livejournalinc.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: Problem with keeping in sync Elasticsearch across two data centers

Michael Sick
In reply to this post by Amit Soni
Hi Amit,

Ivan is correct. You might also check out I believe that you're looking for TribeNodes http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html and see if it fits your needs for cross-dc replication. 

--Mike

On Sat, Feb 22, 2014 at 1:32 PM, Amit Soni <[hidden email]> wrote:
Hello Michael - Understand that ES is not built to maintain consistent cluster state across data centers. what I am wondering is whether there is a way for ElasticSearch to continue to replicate data onto a different data center (with some delay of course) so that when the primary center fails, the fail over data center still has most of the data (may be except for the last few seconds/minutes/hours).

Overall I am looking for a right way to implement cross data center deployment of elastic-search!

-Amit.


On Fri, Feb 21, 2014 at 9:37 AM, Michael Sick <[hidden email]> wrote:
Dario,


ES is not built to consistently cluster across DC's / larger network lags. 

On Fri, Feb 21, 2014 at 11:24 AM, Dario Rossi <[hidden email]> wrote:
Hi, 
I've the following problem: our application publishes content to an Elasticsearch cluster. We use local data less node for querying elasticsearch then, so we don't use HTTP REST and the local nodes are the loadbalancer. Now they came with the requirement of having the cluster replicated to another data center too (and in the future maybe another too... ) for resilience. 

At the very beginning we thought of having one large cluster that goes across data centers (crazy). This solution has the following problems:

- The cluster has the split-brain problem (!)
- The client data less node will try to do requests across different data centers (is there a solution to this???). I can't find a way to avoid this. We don't want this to happen because of a) latency and b) firewalling issues.

So we started to think that this solution is not really viable. So we thought of having one cluster per data center, which seems more sensible. But then here we have the problem that we must publish data to all clusters and, if one fails, we have no means of rolling back (unless we try to set up a complicated version based rollback system). I find this very complicated and hard to maintain, although can be somewhat doable. 

My biggest problem is that we have to keep the data centers in the same state at any time, so that if one goes down, we can readily switch to the other.

Any ideas, or can you recommend some support to help use deal with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAP8axnBOi0R_c%3DcvMYrmpQK-z2%3D4-ik6tGk-ngGtnpjjn11%3DoQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: Problem with keeping in sync Elasticsearch across two data centers

hari

I think with current ES version you have 3 options.

- Use the great snapshot and restore feature to snapshot from a DC and restore in the other one
- Index in both DC (so two distinct clusters) from a client level
- Use Tribe node feature to search or index on multiple clusters

Reference post
https://groups.google.com/forum/#!searchin/elasticsearch/TribeNodes/elasticsearch/MG1RerVSWOk/qZFWvr0HPSwJ

On Saturday, February 22, 2014 6:03:13 PM UTC-6, Michael Sick wrote:
Hi Amit,

Ivan is correct. You might also check out I believe that you're looking for TribeNodes <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html" style="font-family:arial,sans-serif;font-size:12.800000190734863px" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html and see if it fits your needs for cross-dc replication. 

--Mike

On Sat, Feb 22, 2014 at 1:32 PM, Amit Soni <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="vztYvknlSk0J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">amits...@...> wrote:
Hello Michael - Understand that ES is not built to maintain consistent cluster state across data centers. what I am wondering is whether there is a way for ElasticSearch to continue to replicate data onto a different data center (with some delay of course) so that when the primary center fails, the fail over data center still has most of the data (may be except for the last few seconds/minutes/hours).

Overall I am looking for a right way to implement cross data center deployment of elastic-search!

-Amit.


On Fri, Feb 21, 2014 at 9:37 AM, Michael Sick <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="vztYvknlSk0J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">michae...@serenesoftware.com> wrote:
Dario,

I believe that you're looking for TribeNodes <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html

ES is not built to consistently cluster across DC's / larger network lags. 

On Fri, Feb 21, 2014 at 11:24 AM, Dario Rossi <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="vztYvknlSk0J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">dari...@...> wrote:
Hi, 
I've the following problem: our application publishes content to an Elasticsearch cluster. We use local data less node for querying elasticsearch then, so we don't use HTTP REST and the local nodes are the loadbalancer. Now they came with the requirement of having the cluster replicated to another data center too (and in the future maybe another too... ) for resilience. 

At the very beginning we thought of having one large cluster that goes across data centers (crazy). This solution has the following problems:

- The cluster has the split-brain problem (!)
- The client data less node will try to do requests across different data centers (is there a solution to this???). I can't find a way to avoid this. We don't want this to happen because of a) latency and b) firewalling issues.

So we started to think that this solution is not really viable. So we thought of having one cluster per data center, which seems more sensible. But then here we have the problem that we must publish data to all clusters and, if one fails, we have no means of rolling back (unless we try to set up a complicated version based rollback system). I find this very complicated and hard to maintain, although can be somewhat doable. 

My biggest problem is that we have to keep the data centers in the same state at any time, so that if one goes down, we can readily switch to the other.

Any ideas, or can you recommend some support to help use deal with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="vztYvknlSk0J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="vztYvknlSk0J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com';return true;">https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com.

For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="vztYvknlSk0J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com';return true;">https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com.

For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9928e3f4-f7a4-43c5-8c4b-c42ece1d3234%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: Problem with keeping in sync Elasticsearch across two data centers

Amit Soni
Thanks so much everyone for sharing your thoughts!

-Amit.


On Sun, Feb 23, 2014 at 10:24 AM, Hariharan Vadivelu <[hidden email]> wrote:

I think with current ES version you have 3 options.

- Use the great snapshot and restore feature to snapshot from a DC and restore in the other one
- Index in both DC (so two distinct clusters) from a client level
- Use Tribe node feature to search or index on multiple clusters

Reference post

On Saturday, February 22, 2014 6:03:13 PM UTC-6, Michael Sick wrote:
Hi Amit,

Ivan is correct. You might also check out I believe that you're looking for TribeNodes http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html and see if it fits your needs for cross-dc replication. 

--Mike

On Sat, Feb 22, 2014 at 1:32 PM, Amit Soni <[hidden email]> wrote:
Hello Michael - Understand that ES is not built to maintain consistent cluster state across data centers. what I am wondering is whether there is a way for ElasticSearch to continue to replicate data onto a different data center (with some delay of course) so that when the primary center fails, the fail over data center still has most of the data (may be except for the last few seconds/minutes/hours).

Overall I am looking for a right way to implement cross data center deployment of elastic-search!

-Amit.


On Fri, Feb 21, 2014 at 9:37 AM, Michael Sick <michae...@serenesoftware.com> wrote:
Dario,


ES is not built to consistently cluster across DC's / larger network lags. 

On Fri, Feb 21, 2014 at 11:24 AM, Dario Rossi <[hidden email]> wrote:
Hi, 
I've the following problem: our application publishes content to an Elasticsearch cluster. We use local data less node for querying elasticsearch then, so we don't use HTTP REST and the local nodes are the loadbalancer. Now they came with the requirement of having the cluster replicated to another data center too (and in the future maybe another too... ) for resilience. 

At the very beginning we thought of having one large cluster that goes across data centers (crazy). This solution has the following problems:

- The cluster has the split-brain problem (!)
- The client data less node will try to do requests across different data centers (is there a solution to this???). I can't find a way to avoid this. We don't want this to happen because of a) latency and b) firewalling issues.

So we started to think that this solution is not really viable. So we thought of having one cluster per data center, which seems more sensible. But then here we have the problem that we must publish data to all clusters and, if one fails, we have no means of rolling back (unless we try to set up a complicated version based rollback system). I find this very complicated and hard to maintain, although can be somewhat doable. 

My biggest problem is that we have to keep the data centers in the same state at any time, so that if one goes down, we can readily switch to the other.

Any ideas, or can you recommend some support to help use deal with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9928e3f4-f7a4-43c5-8c4b-c42ece1d3234%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAOGaQ%2BvfY33L1%3DieizyPLT9Q22FXXauV-%2BHjxexDYNVXMONTw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: Problem with keeping in sync Elasticsearch across two data centers

Dario Rossi
I will try the tribe node feature, even if I don't understand it completely... but I think it deserves some experimentation

Il giorno martedì 25 febbraio 2014 08:05:05 UTC, amit.soni ha scritto:
Thanks so much everyone for sharing your thoughts!

-Amit.


On Sun, Feb 23, 2014 at 10:24 AM, Hariharan Vadivelu <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="HIVwYjq8nOIJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hari...@...> wrote:

I think with current ES version you have 3 options.

- Use the great snapshot and restore feature to snapshot from a DC and restore in the other one
- Index in both DC (so two distinct clusters) from a client level
- Use Tribe node feature to search or index on multiple clusters

Reference post
<a href="https://groups.google.com/forum/#!searchin/elasticsearch/TribeNodes/elasticsearch/MG1RerVSWOk/qZFWvr0HPSwJ" target="_blank" onmousedown="this.href='https://groups.google.com/forum/#!searchin/elasticsearch/TribeNodes/elasticsearch/MG1RerVSWOk/qZFWvr0HPSwJ';return true;" onclick="this.href='https://groups.google.com/forum/#!searchin/elasticsearch/TribeNodes/elasticsearch/MG1RerVSWOk/qZFWvr0HPSwJ';return true;">https://groups.google.com/forum/#!searchin/elasticsearch/TribeNodes/elasticsearch/MG1RerVSWOk/qZFWvr0HPSwJ

On Saturday, February 22, 2014 6:03:13 PM UTC-6, Michael Sick wrote:
Hi Amit,

Ivan is correct. You might also check out I believe that you're looking for TribeNodes <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html" style="font-family:arial,sans-serif;font-size:12.800000190734863px" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html and see if it fits your needs for cross-dc replication. 

--Mike

On Sat, Feb 22, 2014 at 1:32 PM, Amit Soni <[hidden email]> wrote:
Hello Michael - Understand that ES is not built to maintain consistent cluster state across data centers. what I am wondering is whether there is a way for ElasticSearch to continue to replicate data onto a different data center (with some delay of course) so that when the primary center fails, the fail over data center still has most of the data (may be except for the last few seconds/minutes/hours).

Overall I am looking for a right way to implement cross data center deployment of elastic-search!

-Amit.


On Fri, Feb 21, 2014 at 9:37 AM, Michael Sick <michae...@serenesoftware.com> wrote:
Dario,

I believe that you're looking for TribeNodes <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html

ES is not built to consistently cluster across DC's / larger network lags. 

On Fri, Feb 21, 2014 at 11:24 AM, Dario Rossi <[hidden email]> wrote:
Hi, 
I've the following problem: our application publishes content to an Elasticsearch cluster. We use local data less node for querying elasticsearch then, so we don't use HTTP REST and the local nodes are the loadbalancer. Now they came with the requirement of having the cluster replicated to another data center too (and in the future maybe another too... ) for resilience. 

At the very beginning we thought of having one large cluster that goes across data centers (crazy). This solution has the following problems:

- The cluster has the split-brain problem (!)
- The client data less node will try to do requests across different data centers (is there a solution to this???). I can't find a way to avoid this. We don't want this to happen because of a) latency and b) firewalling issues.

So we started to think that this solution is not really viable. So we thought of having one cluster per data center, which seems more sensible. But then here we have the problem that we must publish data to all clusters and, if one fails, we have no means of rolling back (unless we try to set up a complicated version based rollback system). I find this very complicated and hard to maintain, although can be somewhat doable. 

My biggest problem is that we have to keep the data centers in the same state at any time, so that if one goes down, we can readily switch to the other.

Any ideas, or can you recommend some support to help use deal with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com';return true;">https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com.

For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com';return true;">https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com.

For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="HIVwYjq8nOIJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/9928e3f4-f7a4-43c5-8c4b-c42ece1d3234%40googlegroups.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/9928e3f4-f7a4-43c5-8c4b-c42ece1d3234%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/9928e3f4-f7a4-43c5-8c4b-c42ece1d3234%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/9928e3f4-f7a4-43c5-8c4b-c42ece1d3234%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ab2c578c-2ccd-4ce6-b095-450f84013fe6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: Problem with keeping in sync Elasticsearch across two data centers

Dario Rossi
From the docs it is not clear if having two clusters with the same indexes, a indexing operation will have effect on both...

There is a line that leaves me bit doubtful:

However, there are a few exceptions:

  • The merged view cannot handle indices with the same name in multiple clusters. It will pick one of them and discard the other.

Il giorno martedì 25 febbraio 2014 10:04:05 UTC, Dario Rossi ha scritto:
I will try the tribe node feature, even if I don't understand it completely... but I think it deserves some experimentation

Il giorno martedì 25 febbraio 2014 08:05:05 UTC, amit.soni ha scritto:
Thanks so much everyone for sharing your thoughts!

-Amit.


On Sun, Feb 23, 2014 at 10:24 AM, Hariharan Vadivelu <[hidden email]> wrote:

I think with current ES version you have 3 options.

- Use the great snapshot and restore feature to snapshot from a DC and restore in the other one
- Index in both DC (so two distinct clusters) from a client level
- Use Tribe node feature to search or index on multiple clusters

Reference post
<a href="https://groups.google.com/forum/#!searchin/elasticsearch/TribeNodes/elasticsearch/MG1RerVSWOk/qZFWvr0HPSwJ" target="_blank" onmousedown="this.href='https://groups.google.com/forum/#!searchin/elasticsearch/TribeNodes/elasticsearch/MG1RerVSWOk/qZFWvr0HPSwJ';return true;" onclick="this.href='https://groups.google.com/forum/#!searchin/elasticsearch/TribeNodes/elasticsearch/MG1RerVSWOk/qZFWvr0HPSwJ';return true;">https://groups.google.com/forum/#!searchin/elasticsearch/TribeNodes/elasticsearch/MG1RerVSWOk/qZFWvr0HPSwJ

On Saturday, February 22, 2014 6:03:13 PM UTC-6, Michael Sick wrote:
Hi Amit,

Ivan is correct. You might also check out I believe that you're looking for TribeNodes <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html" style="font-family:arial,sans-serif;font-size:12.800000190734863px" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html and see if it fits your needs for cross-dc replication. 

--Mike

On Sat, Feb 22, 2014 at 1:32 PM, Amit Soni <[hidden email]> wrote:
Hello Michael - Understand that ES is not built to maintain consistent cluster state across data centers. what I am wondering is whether there is a way for ElasticSearch to continue to replicate data onto a different data center (with some delay of course) so that when the primary center fails, the fail over data center still has most of the data (may be except for the last few seconds/minutes/hours).

Overall I am looking for a right way to implement cross data center deployment of elastic-search!

-Amit.


On Fri, Feb 21, 2014 at 9:37 AM, Michael Sick <michae...@serenesoftware.com> wrote:
Dario,

I believe that you're looking for TribeNodes <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fmaster%2Fmodules-tribe.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEfvIk4tSiBJ3mE850oFfj7LxVZIQ';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-tribe.html

ES is not built to consistently cluster across DC's / larger network lags. 

On Fri, Feb 21, 2014 at 11:24 AM, Dario Rossi <[hidden email]> wrote:
Hi, 
I've the following problem: our application publishes content to an Elasticsearch cluster. We use local data less node for querying elasticsearch then, so we don't use HTTP REST and the local nodes are the loadbalancer. Now they came with the requirement of having the cluster replicated to another data center too (and in the future maybe another too... ) for resilience. 

At the very beginning we thought of having one large cluster that goes across data centers (crazy). This solution has the following problems:

- The cluster has the split-brain problem (!)
- The client data less node will try to do requests across different data centers (is there a solution to this???). I can't find a way to avoid this. We don't want this to happen because of a) latency and b) firewalling issues.

So we started to think that this solution is not really viable. So we thought of having one cluster per data center, which seems more sensible. But then here we have the problem that we must publish data to all clusters and, if one fails, we have no means of rolling back (unless we try to set up a complicated version based rollback system). I find this very complicated and hard to maintain, although can be somewhat doable. 

My biggest problem is that we have to keep the data centers in the same state at any time, so that if one goes down, we can readily switch to the other.

Any ideas, or can you recommend some support to help use deal with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/5424a274-3f6b-4c12-9fe6-621e04f87a8d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com';return true;">https://groups.google.com/d/msgid/elasticsearch/CAP8axnDW4GCDnnzwA%2BcyR%2BN4g-26VV4CZ-ZW6SDGgxFL75qy%2Bw%40mail.gmail.com.

For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com';return true;">https://groups.google.com/d/msgid/elasticsearch/CAAOGaQKLUGepyKyR4oDNq1B7-uosp9SWCCeZmkRdQHsSJTSndA%40mail.gmail.com.

For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/9928e3f4-f7a4-43c5-8c4b-c42ece1d3234%40googlegroups.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/9928e3f4-f7a4-43c5-8c4b-c42ece1d3234%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/9928e3f4-f7a4-43c5-8c4b-c42ece1d3234%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/9928e3f4-f7a4-43c5-8c4b-c42ece1d3234%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/58485177-4a2a-4f09-b09f-4780add644f7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.