Script terms facet in elasticsearch 0.90.0 returning incorrect results

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Script terms facet in elasticsearch 0.90.0 returning incorrect results

Eric Jain
https://gist.github.com/ejain/5507570 (0.20.6) vs https://gist.github.com/ejain/5507571 (0.90.0)

Looks like time zone offsets in dates are now ignored, and dates are assumed to be in the default time zone (i.e. Pacific Time for me) when being accessed from a script?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Script terms facet in elasticsearch 0.90.0 returning incorrect results

Eric Jain
For the record: Dates in scripts now appear to use the default time zone; previously they were always UTC. Not necessarily a bug, but a breaking change to be aware of...


On Thursday, May 2, 2013 11:54:22 PM UTC-7, Eric Jain wrote:
https://gist.github.com/ejain/5507570 (0.20.6) vs https://gist.github.com/ejain/5507571 (0.90.0)

Looks like time zone offsets in dates are now ignored, and dates are assumed to be in the default time zone (i.e. Pacific Time for me) when being accessed from a script?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Script terms facet in elasticsearch 0.90.0 returning incorrect results

Marcin Dojwa
Thanks for this one. What happens if you give '2012-11-30T15:00:00Z'? The given datetime is treated as UTC or default timezone too ?


2013/5/7 Eric Jain <[hidden email]>
For the record: Dates in scripts now appear to use the default time zone; previously they were always UTC. Not necessarily a bug, but a breaking change to be aware of...



On Thursday, May 2, 2013 11:54:22 PM UTC-7, Eric Jain wrote:
https://gist.github.com/ejain/5507570 (0.20.6) vs https://gist.github.com/ejain/5507571 (0.90.0)

Looks like time zone offsets in dates are now ignored, and dates are assumed to be in the default time zone (i.e. Pacific Time for me) when being accessed from a script?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Script terms facet in elasticsearch 0.90.0 returning incorrect results

Eric Jain
On Wed, May 8, 2013 at 5:37 AM, Marcin Dojwa <[hidden email]> wrote:
> Thanks for this one. What happens if you give '2012-11-30T15:00:00Z'? The
> given datetime is treated as UTC or default timezone too ?

If your local timezone offset is e.g. -07:00, doc['timestamp'].date is
now (0.90) '2012-11-30T08:00:00-07:00'. Previously (0.20), it would
have been '2012-11-30T15:00:00Z'; not because elasticsearch was
honoring the date's original timezone offset (I wish it did!), but
because the `date` property in a script would be always be UTC.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Script terms facet in elasticsearch 0.90.0 returning incorrect results

Marcin Dojwa
For me it sounds like a bug. Did you report the issue here https://github.com/elasticsearch/elasticsearch/issues


2013/5/8 Eric Jain <[hidden email]>
On Wed, May 8, 2013 at 5:37 AM, Marcin Dojwa <[hidden email]> wrote:
> Thanks for this one. What happens if you give '2012-11-30T15:00:00Z'? The
> given datetime is treated as UTC or default timezone too ?

If your local timezone offset is e.g. -07:00, doc['timestamp'].date is
now (0.90) '2012-11-30T08:00:00-07:00'. Previously (0.20), it would
have been '2012-11-30T15:00:00Z'; not because elasticsearch was
honoring the date's original timezone offset (I wish it did!), but
because the `date` property in a script would be always be UTC.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Script terms facet in elasticsearch 0.90.0 returning incorrect results

Eric Jain
On Thu, May 9, 2013 at 1:45 AM, Marcin Dojwa <[hidden email]> wrote:
> For me it sounds like a bug. Did you report the issue here
> https://github.com/elasticsearch/elasticsearch/issues ?

I haven't report the issue yet; I was hoping to hear from someone at
elasticsearch whether this change was intentional or not.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Script terms facet in elasticsearch 0.90.0 returning incorrect results

Marcin Dojwa
I suggest reporting an issue. If this is not an issue, you will get fast response there. Could you give a link to the issue here when you report it ? Solving this problem is very important for me too before I deploy 0.90 on production. Thanks.


2013/5/9 Eric Jain <[hidden email]>
On Thu, May 9, 2013 at 1:45 AM, Marcin Dojwa <[hidden email]> wrote:
> For me it sounds like a bug. Did you report the issue here
> https://github.com/elasticsearch/elasticsearch/issues ?

I haven't report the issue yet; I was hoping to hear from someone at
elasticsearch whether this change was intentional or not.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Script terms facet in elasticsearch 0.90.0 returning incorrect results

phill
In reply to this post by Eric Jain
On 5/8/2013 9:29 AM, Eric Jain wrote:
On Wed, May 8, 2013 at 5:37 AM, Marcin Dojwa [hidden email] wrote:
Thanks for this one. What happens if you give '2012-11-30T15:00:00Z'? The
given datetime is treated as UTC or default timezone too ?
If your local timezone offset is e.g. -07:00, doc['timestamp'].date is
now (0.90) '2012-11-30T08:00:00-07:00'. Previously (0.20), it would
have been '2012-11-30T15:00:00Z'; not because elasticsearch was
honoring the date's original timezone offset (I wish it did!), but
because the `date` property in a script would be always be UTC.


Actually both of those are the same millisecond value stored in the date field displayed differently (In the Summer 8 AM in Seattle is the same moment in time as 3 PM in London) Therefore, it depends on the definition of what the script aDateField.date is supposed to correspond to in the Joda date library and thus what toString method is used to convert the results back to the caller.  It is not a matter of honoring the original TZ, because it can't, the field stores a millisecond value; that's it.  On the way out nothing knows how it got that millisecond value into the field.  

As Eric as observed
it WAS converting the millisecond value to the result string using UTC.
but now
it IS converting to the millisecond value to the result string using the DEFAULT VM timezone.

"The default date parsing used if no format is specified is ISODateTimeFormat.dateOptionalTimeParser."
"... The first format will also act as the one that converts back from milliseconds to a string representation.  ..."

-- http://www.elasticsearch.org/guide/reference/mapping/date-format/

None of the formats specified make any claim as to what TZ will be used when formatting the millisecond value into a readable string, but certainly being able to pick one by setting default VM timezone is a valid choice, but so is the earlier idea of always using UTC, so I don't know which behavior is intended.  It is also important that all output formats do the same thing and not mix VM default with UTC.  One idea is that forcing UTC as the default on the way out reminds folks to think about what they want to see and to learn the difference between how it is stored and how it is transported back (hopefully it is short leap of logic to thinking about how it is transported into the index).

Just as a warning to all, having client and server us different timezones can make hard to find bugs when using the default input format which allows dates (without times) sent from clients generating local dates and then falling into the previous or next day, because they end up at "somebody else's" (the servers) midnight when it is converted for storage into a millisecond value.
Client says "2013-05-10" (thinking that means "2012-05-10 T 00:00:00 -07:00" or "2012-05-10 T 07:00:00Z"), server records a timestamp of 7 hours earlier "2012-05-10 T 00:00:00 Z".  If later the client code fully specifies things with times and timezones, things can be curiously mismatched with the original timestamp and get subtly confusing.

-Paul

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Script terms facet in elasticsearch 0.90.0 returning incorrect results

Eric Jain
In reply to this post by Marcin Dojwa
On Thursday, May 9, 2013 1:45:13 AM UTC-7, Marcin Dojwa wrote:
For me it sounds like a bug. Did you report the issue here https://github.com/elasticsearch/elasticsearch/issues

https://github.com/elasticsearch/elasticsearch/issues/3091
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.