How do people typically handle shard failures in their results?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How do people typically handle shard failures in their results?

mooky
If I understand correctly, we can get an OK response from elastic (ie no error) but if there are shard failures in the response, it potentially means that results are incomplete/incorrect. From my observation, we can get failures on all shards - and elastic still returns OK (which was a bit surprising to me)

What kinds of approaches to people typically use to deal with shard failures?

For my application, if there are shard failures, essentially my results are inaccurate/incorrect - so I need to return an error to the client. Returning bad results is worse than returning an error. 

I am inclined to turn any shard failure into an exception.
Is this quite common? Does it make sense to add a feature to the elastic api ? (ie request.setTreatShardFailuresAsErrors(true)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/461fa217-d664-47e9-a60d-88ea9506327d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How do people typically handle shard failures in their results?

Itamar Syn-Hershko
Every man and his own business requirements and SLAs. Many scenarios I've seen would consider two-thirds of shards responding as an OK scenario and not even show a notice about results being partial. But of course it makes sense to throw an exception if your business requires / expects that.

FWIW, this is usually why you use replicas, so even if a shard goes down there's a back-up shard (ideally more than one) that you can fallback to.

--

Itamar Syn-Hershko
http://code972.com | @synhershko
Freelance Developer & Consultant


On Wed, Jun 18, 2014 at 12:45 PM, mooky <[hidden email]> wrote:
If I understand correctly, we can get an OK response from elastic (ie no error) but if there are shard failures in the response, it potentially means that results are incomplete/incorrect. From my observation, we can get failures on all shards - and elastic still returns OK (which was a bit surprising to me)

What kinds of approaches to people typically use to deal with shard failures?

For my application, if there are shard failures, essentially my results are inaccurate/incorrect - so I need to return an error to the client. Returning bad results is worse than returning an error. 

I am inclined to turn any shard failure into an exception.
Is this quite common? Does it make sense to add a feature to the elastic api ? (ie request.setTreatShardFailuresAsErrors(true)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/461fa217-d664-47e9-a60d-88ea9506327d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zu7ECzf7bA_YG0XoGaVrPr1CUyKzhq1VRn4XD47CH%2B0MQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How do people typically handle shard failures in their results?

kimchy
Administrator
In reply to this post by mooky
If it fails on the primary shard, then a failure is returned. If it worked, and a replica failed, then that replica is deemed a failed replica, and will get allocated somewhere else in the cluster. Maybe an example of where a failure on “all” shards would help here?

On Jun 18, 2014, at 11:45, mooky <[hidden email]> wrote:

If I understand correctly, we can get an OK response from elastic (ie no error) but if there are shard failures in the response, it potentially means that results are incomplete/incorrect. From my observation, we can get failures on all shards - and elastic still returns OK (which was a bit surprising to me)

What kinds of approaches to people typically use to deal with shard failures?

For my application, if there are shard failures, essentially my results are inaccurate/incorrect - so I need to return an error to the client. Returning bad results is worse than returning an error. 

I am inclined to turn any shard failure into an exception.
Is this quite common? Does it make sense to add a feature to the elastic api ? (ie request.setTreatShardFailuresAsErrors(true)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/461fa217-d664-47e9-a60d-88ea9506327d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/FFDC7083-24CB-484D-B337-65582596D555%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How do people typically handle shard failures in their results?

Nikolas Everett



On Fri, Jun 20, 2014 at 7:08 AM, Shay Banon <[hidden email]> wrote:
If it fails on the primary shard, then a failure is returned. If it worked, and a replica failed, then that replica is deemed a failed replica, and will get allocated somewhere else in the cluster. Maybe an example of where a failure on “all” shards would help here?

I think its more about searches and they can fail on one shard but not other for all sorts of reasons.  Queue full, unfortunate script, bug, only one shard had results and the query asked for something weird like to use the postings highlighter when postings aren't stored.  Lots of reasons.

I log the event and move on.  I toyed with outputting a warning to the user but didn't have time to implement it.  We're pretty diligent with our logs so we'd notice the log and run it down.

If the failure is caused by the queue being full only on one node, we'd likely notice that real quick as ganglia would lose it.  This happened to me recently when we put a node without an ssd into a cluster with ssds.  It couldn't keep up and dropped a ton of searches.  In our defense, we didn't know the rest of the cluster had ssds so we were double surprised.

Nik

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How do people typically handle shard failures in their results?

kimchy
Administrator
Ahh, I see. If its related to searches, then yea, the search response includes details about the total shards that the search was executed on, the successful shards, and failed shards. They are important to check to understand if one gets partial results.

In the REST API, if there is a total “failure”, then it will return the “worst” status code out of all the shards in the response. In the Java API, the search response will be returned (with no exception), so the content of the search has to be checked (which is a good practice anyhow). It might make sense to raise an exception in the Java API if all shards failed, I am on the fence on this one, since anyhow a check needs to be performed on the result.

On Jun 20, 2014, at 13:22, Nikolas Everett <[hidden email]> wrote:




On Fri, Jun 20, 2014 at 7:08 AM, Shay Banon <[hidden email]> wrote:
If it fails on the primary shard, then a failure is returned. If it worked, and a replica failed, then that replica is deemed a failed replica, and will get allocated somewhere else in the cluster. Maybe an example of where a failure on “all” shards would help here?

I think its more about searches and they can fail on one shard but not other for all sorts of reasons.  Queue full, unfortunate script, bug, only one shard had results and the query asked for something weird like to use the postings highlighter when postings aren't stored.  Lots of reasons.

I log the event and move on.  I toyed with outputting a warning to the user but didn't have time to implement it.  We're pretty diligent with our logs so we'd notice the log and run it down.

If the failure is caused by the queue being full only on one node, we'd likely notice that real quick as ganglia would lose it.  This happened to me recently when we put a node without an ssd into a cluster with ssds.  It couldn't keep up and dropped a ton of searches.  In our defense, we didn't know the rest of the cluster had ssds so we were double surprised.

Nik

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9A9246D5-338B-4B93-B2FD-4D3B93F621F2%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How do people typically handle shard failures in their results?

mooky
I see. Thanks.
I wasnt expecting to have to check for shard failures.
Would it make sense to have a config or request setting whereby any shard failures are reported as  exceptions (java api).
In the space where we have been using elastic, partial results are bad. We would end up having to always write code to turn shard failures into exceptions....

-Nick

On Friday, 20 June 2014 13:26:35 UTC+1, kimchy wrote:
Ahh, I see. If its related to searches, then yea, the search response includes details about the total shards that the search was executed on, the successful shards, and failed shards. They are important to check to understand if one gets partial results.

In the REST API, if there is a total “failure”, then it will return the “worst” status code out of all the shards in the response. In the Java API, the search response will be returned (with no exception), so the content of the search has to be checked (which is a good practice anyhow). It might make sense to raise an exception in the Java API if all shards failed, I am on the fence on this one, since anyhow a check needs to be performed on the result.

On Jun 20, 2014, at 13:22, Nikolas Everett <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Estb7tuuJYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">nik...@...> wrote:




On Fri, Jun 20, 2014 at 7:08 AM, Shay Banon <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Estb7tuuJYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">kim...@...> wrote:
If it fails on the primary shard, then a failure is returned. If it worked, and a replica failed, then that replica is deemed a failed replica, and will get allocated somewhere else in the cluster. Maybe an example of where a failure on “all” shards would help here?

I think its more about searches and they can fail on one shard but not other for all sorts of reasons.  Queue full, unfortunate script, bug, only one shard had results and the query asked for something weird like to use the postings highlighter when postings aren't stored.  Lots of reasons.

I log the event and move on.  I toyed with outputting a warning to the user but didn't have time to implement it.  We're pretty diligent with our logs so we'd notice the log and run it down.

If the failure is caused by the queue being full only on one node, we'd likely notice that real quick as ganglia would lose it.  This happened to me recently when we put a node without an ssd into a cluster with ssds.  It couldn't keep up and dropped a ton of searches.  In our defense, we didn't know the rest of the cluster had ssds so we were double surprised.

Nik

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Estb7tuuJYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/63f3b5a8-02b2-4a2f-bb47-b42b46ff4d54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...