Elasticsearch.net client, endpoint strategy?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Elasticsearch.net client, endpoint strategy?

lasseschou
Hi,

Not sure if this is the right user group, but here goes:

I'm planning to use ElasticSearch.net as the client for connecting to my ES cluster. I have one question I haven't been able to find the answer to. I know that the ConnectionPool feature can check if nodes fail, but can the client also ensure that data is written to the right shard, or does it simply use round robin to connect?

Example:

- A document "1234" is created
- Based on the current number of shards, the document should be put on node 6 and a replica on node 11.
- Is the request sent directly to node 6, or is it sent to a random node which will then forward the request to the right servers?

Thanks,
Lasse

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch.net client, endpoint strategy?

vineeth mohan-2
Hello Lasse , 

Following is my idea on the whole thing - 

When a index request comes , based on the ID of the request , a hash function computes the shard to which the request has to be routed. 
Hence we are achieving a load balancing based on this procedure. It has to be also noted that the routing key can be controlled. 

Sniffing - While creating a client , you can use the sniffing feature for elasticsearch to determine all the nodes and use them all in a load balanced way. 

Thanks
           Vineeth

On Fri, Sep 12, 2014 at 2:39 PM, Lasse Schou <[hidden email]> wrote:
Hi,

Not sure if this is the right user group, but here goes:

I'm planning to use ElasticSearch.net as the client for connecting to my ES cluster. I have one question I haven't been able to find the answer to. I know that the ConnectionPool feature can check if nodes fail, but can the client also ensure that data is written to the right shard, or does it simply use round robin to connect?

Example:

- A document "1234" is created
- Based on the current number of shards, the document should be put on node 6 and a replica on node 11.
- Is the request sent directly to node 6, or is it sent to a random node which will then forward the request to the right servers?

Thanks,
Lasse

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch.net client, endpoint strategy?

lasseschou
Alright, so the clients don't have the knowledge to route the request to the right server. This would typically yields an extra request from the receiving node to the node that stores the data, but on the other hand it reduces the complexity of the client.

Can anybody confirm that this is the recommended strategy? In other databases like Couchbase, the client knows exactly which node to route the request to, based on a shard map that is automatically updated on regular time intervals.

2014-09-13 5:29 GMT+02:00 vineeth mohan <[hidden email]>:
Hello Lasse , 

Following is my idea on the whole thing - 

When a index request comes , based on the ID of the request , a hash function computes the shard to which the request has to be routed. 
Hence we are achieving a load balancing based on this procedure. It has to be also noted that the routing key can be controlled. 

Sniffing - While creating a client , you can use the sniffing feature for elasticsearch to determine all the nodes and use them all in a load balanced way. 

Thanks
           Vineeth

On Fri, Sep 12, 2014 at 2:39 PM, Lasse Schou <[hidden email]> wrote:
Hi,

Not sure if this is the right user group, but here goes:

I'm planning to use ElasticSearch.net as the client for connecting to my ES cluster. I have one question I haven't been able to find the answer to. I know that the ConnectionPool feature can check if nodes fail, but can the client also ensure that data is written to the right shard, or does it simply use round robin to connect?

Example:

- A document "1234" is created
- Based on the current number of shards, the document should be put on node 6 and a replica on node 11.
- Is the request sent directly to node 6, or is it sent to a random node which will then forward the request to the right servers?

Thanks,
Lasse

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/BpEov5aoxvU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADERWXrKrdB5GnsG-YQv3W0yygyUgNzqDqxYQ4Gyr7YZ12b4ag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch.net client, endpoint strategy?

Itamar Syn-Hershko
Yes, all REST clients (that is to exclude the native Java clients) do not participate in the cluster and therefor don't have and shouldn't have any knowledge of the cluster. The issue a request to a node in the cluster which then may or may not reroute it to 1 or more nodes in an efficient manner using the cluster meta-data it has stored locally. This is by design and has many optimizations in place as well, and also allows you as a client to use round-robin or make requests to a load balancer.

--

Itamar Syn-Hershko
http://code972.com | @synhershko
Freelance Developer & Consultant

On Sun, Sep 14, 2014 at 4:13 PM, Lasse Schou <[hidden email]> wrote:
Alright, so the clients don't have the knowledge to route the request to the right server. This would typically yields an extra request from the receiving node to the node that stores the data, but on the other hand it reduces the complexity of the client.

Can anybody confirm that this is the recommended strategy? In other databases like Couchbase, the client knows exactly which node to route the request to, based on a shard map that is automatically updated on regular time intervals.

2014-09-13 5:29 GMT+02:00 vineeth mohan <[hidden email]>:
Hello Lasse , 

Following is my idea on the whole thing - 

When a index request comes , based on the ID of the request , a hash function computes the shard to which the request has to be routed. 
Hence we are achieving a load balancing based on this procedure. It has to be also noted that the routing key can be controlled. 

Sniffing - While creating a client , you can use the sniffing feature for elasticsearch to determine all the nodes and use them all in a load balanced way. 

Thanks
           Vineeth

On Fri, Sep 12, 2014 at 2:39 PM, Lasse Schou <[hidden email]> wrote:
Hi,

Not sure if this is the right user group, but here goes:

I'm planning to use ElasticSearch.net as the client for connecting to my ES cluster. I have one question I haven't been able to find the answer to. I know that the ConnectionPool feature can check if nodes fail, but can the client also ensure that data is written to the right shard, or does it simply use round robin to connect?

Example:

- A document "1234" is created
- Based on the current number of shards, the document should be put on node 6 and a replica on node 11.
- Is the request sent directly to node 6, or is it sent to a random node which will then forward the request to the right servers?

Thanks,
Lasse

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/BpEov5aoxvU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADERWXrKrdB5GnsG-YQv3W0yygyUgNzqDqxYQ4Gyr7YZ12b4ag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtTSQdxd8q9-1-0YZLyXvDK-Cptkm%3D99T-d0_KdHqMkZw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch.net client, endpoint strategy?

lasseschou
Thanks for your clarifications.

2014-09-14 15:17 GMT+02:00 Itamar Syn-Hershko <[hidden email]>:
Yes, all REST clients (that is to exclude the native Java clients) do not participate in the cluster and therefor don't have and shouldn't have any knowledge of the cluster. The issue a request to a node in the cluster which then may or may not reroute it to 1 or more nodes in an efficient manner using the cluster meta-data it has stored locally. This is by design and has many optimizations in place as well, and also allows you as a client to use round-robin or make requests to a load balancer.

--

Itamar Syn-Hershko
http://code972.com | @synhershko
Freelance Developer & Consultant

On Sun, Sep 14, 2014 at 4:13 PM, Lasse Schou <[hidden email]> wrote:
Alright, so the clients don't have the knowledge to route the request to the right server. This would typically yields an extra request from the receiving node to the node that stores the data, but on the other hand it reduces the complexity of the client.

Can anybody confirm that this is the recommended strategy? In other databases like Couchbase, the client knows exactly which node to route the request to, based on a shard map that is automatically updated on regular time intervals.

<a href="tel:2014-09-13" value="+4520140913" target="_blank">2014-09-13 5:29 GMT+02:00 vineeth mohan <[hidden email]>:
Hello Lasse , 

Following is my idea on the whole thing - 

When a index request comes , based on the ID of the request , a hash function computes the shard to which the request has to be routed. 
Hence we are achieving a load balancing based on this procedure. It has to be also noted that the routing key can be controlled. 

Sniffing - While creating a client , you can use the sniffing feature for elasticsearch to determine all the nodes and use them all in a load balanced way. 

Thanks
           Vineeth

On Fri, Sep 12, 2014 at 2:39 PM, Lasse Schou <[hidden email]> wrote:
Hi,

Not sure if this is the right user group, but here goes:

I'm planning to use ElasticSearch.net as the client for connecting to my ES cluster. I have one question I haven't been able to find the answer to. I know that the ConnectionPool feature can check if nodes fail, but can the client also ensure that data is written to the right shard, or does it simply use round robin to connect?

Example:

- A document "1234" is created
- Based on the current number of shards, the document should be put on node 6 and a replica on node 11.
- Is the request sent directly to node 6, or is it sent to a random node which will then forward the request to the right servers?

Thanks,
Lasse

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/BpEov5aoxvU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADERWXrKrdB5GnsG-YQv3W0yygyUgNzqDqxYQ4Gyr7YZ12b4ag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/BpEov5aoxvU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtTSQdxd8q9-1-0YZLyXvDK-Cptkm%3D99T-d0_KdHqMkZw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADERWXre5Ou4639T0sJ85En3Ew2hOG1j0HMpgu6RaXkWPk-P8g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.