_timestamp

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

_timestamp

vinh
When inserting docs, I store my own timestamp value in the default "_timestamp" field.  Then I search docs and fetch both "_source" and "_timestamp" fields.  I notice that the value of _source._timestamp correctly has my value.  But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am overriding it?

    }, {
      "_index" : "myindex",
      "_type" : "message",
      "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
      "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello there"},
      "fields" : {
        "_timestamp" : 1361672211995
      },
      "sort" : [ 1361672211995 ]
    } ]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: _timestamp

vinh
Sorry, copy-paste error.  The ES response is:

    }, {
      "_index" : "myindex",
      "_type" : "message",
      "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
      "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","_timestamp":1361672205131,"msg":"hello there"},
      "fields" : {
        "_timestamp" : 1361672211995
      },
      "sort" : [ 1361672211995 ]
    } ]

On Mar 5, 2013, at 2:29 PM, vinh <[hidden email]> wrote:

When inserting docs, I store my own timestamp value in the default "_timestamp" field.  Then I search docs and fetch both "_source" and "_timestamp" fields.  I notice that the value of _source._timestamp correctly has my value.  But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am overriding it?

    }, {
      "_index" : "myindex",
      "_type" : "message",
      "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
      "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello there"},
      "fields" : {
        "_timestamp" : 1361672211995
      },
      "sort" : [ 1361672211995 ]
    } ]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: _timestamp

Ivan Brusic
In reply to this post by vinh
Did you enable the timestamp field in your mapping? It is not enabled by default. Also, your example shows the field as "timestamp" without the underscore. Is this correct? Your source should contain whatever was inserted.

-- 
Ivan


On Tue, Mar 5, 2013 at 2:29 PM, vinh <[hidden email]> wrote:
When inserting docs, I store my own timestamp value in the default "_timestamp" field.  Then I search docs and fetch both "_source" and "_timestamp" fields.  I notice that the value of _source._timestamp correctly has my value.  But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am overriding it?

    }, {
      "_index" : "myindex",
      "_type" : "message",
      "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
      "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello there"},
      "fields" : {
        "_timestamp" : 1361672211995
      },
      "sort" : [ 1361672211995 ]
    } ]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: _timestamp

vinh
In reply to this post by vinh
My schema is:

{
  "template" : "*",
  "settings": {
    "number_of_shards" : 5,
    "number_of_replicas" : 1
  },
  "mappings" : {
    "message" : {
      "_source"    : {"enabled" : true},
      "_all"       : {"enabled": false},
      "_id"        : {"index": "not_analyzed", "store" : "yes"},
      "_timestamp" : {"enabled" : true, "index" : "not_analyzed", "store" : "yes"}
    }
  }
}

On Mar 5, 2013, at 2:41 PM, vinh <[hidden email]> wrote:

Sorry, copy-paste error.  The ES response is:

    }, {
      "_index" : "myindex",
      "_type" : "message",
      "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
      "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","_timestamp":1361672205131,"msg":"hello there"},
      "fields" : {
        "_timestamp" : 1361672211995
      },
      "sort" : [ 1361672211995 ]
    } ]

On Mar 5, 2013, at 2:29 PM, vinh <[hidden email]> wrote:

When inserting docs, I store my own timestamp value in the default "_timestamp" field.  Then I search docs and fetch both "_source" and "_timestamp" fields.  I notice that the value of _source._timestamp correctly has my value.  But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am overriding it?

    }, {
      "_index" : "myindex",
      "_type" : "message",
      "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
      "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello there"},
      "fields" : {
        "_timestamp" : 1361672211995
      },
      "sort" : [ 1361672211995 ]
    } ]


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: _timestamp

joergprante@gmail.com
Have you tried without "index" : "not_analyzed", "store" : "yes" ?

Jörg

Am 05.03.13 23:43, schrieb vinh:
>     "_timestamp" : {"enabled" : true, "index" : "not_analyzed",
> "store" : "yes"}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: _timestamp

Ivan Brusic
I have never used the timestamp field, but I have seen peculiarities with the boost field. 

From what I can tell from the documentation, the timestamp field is added automatically. Providing your own is conflicting with the internal version. And while I do not know all the details of the conflict resolution mechanism, my only advice would be to simply not get into the situation in the first place. Either you add the timestamp or let ElasticSearch do it. As you can see, maintaining your own is not difficult.

It appears that ElasticSearch is preserving the original in the source, but adding its own as the indexed field. The behavior seems ideal.

Cheers,
Ivan


On Tue, Mar 5, 2013 at 2:47 PM, Jörg Prante <[hidden email]> wrote:
Have you tried without "index" : "not_analyzed", "store" : "yes" ?

Jörg

Am 05.03.13 23:43, schrieb vinh:

    "_timestamp" : {"enabled" : true, "index" : "not_analyzed", "store" : "yes"}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: _timestamp

Benjamin Devèze
In reply to this post by vinh
Hey you can either set the timestamp value in your index request or extract it from the source. If you want to extract it from the source you have to tell ES where to find you timestamp field using path in your mapping as described here
http://www.elasticsearch.org/guide/reference/mapping/timestamp-field.html.

It won't automatically use a _timestamp field set in your source content.

On Tuesday, March 5, 2013 11:29:44 PM UTC+1, vinh wrote:
When inserting docs, I store my own timestamp value in the default "_timestamp" field.  Then I search docs and fetch both "_source" and "_timestamp" fields.  I notice that the value of _source._timestamp correctly has my value.  But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am overriding it?

    }, {
      "_index" : "myindex",
      "_type" : "message",
      "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
      "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello there"},
      "fields" : {
        "_timestamp" : 1361672211995
      },
      "sort" : [ 1361672211995 ]
    } ]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: _timestamp

vinh
In reply to this post by joergprante@gmail.com
Thanks Jorg,
No luck without that setting.  But I also need that setting, or else I can't sort on _timestamp.
-Vinh

On Mar 5, 2013, at 2:47 PM, Jörg Prante <[hidden email]> wrote:

> Have you tried without "index" : "not_analyzed", "store" : "yes" ?
>
> Jörg
>
> Am 05.03.13 23:43, schrieb vinh:
>>    "_timestamp" : {"enabled" : true, "index" : "not_analyzed", "store" : "yes"}
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: _timestamp

vinh
In reply to this post by Benjamin Devèze
Thanks Benjamin,

I am currently setting the _timestamp value in the request.  I'm trying to avoid mapping/extracting it to another field in the source, to avoid needing two fields for the same value.

-Vinh

On Mar 5, 2013, at 3:23 PM, Benjamin Devèze <[hidden email]> wrote:

Hey you can either set the timestamp value in your index request or extract it from the source. If you want to extract it from the source you have to tell ES where to find you timestamp field using path in your mapping as described here

It won't automatically use a _timestamp field set in your source content.

On Tuesday, March 5, 2013 11:29:44 PM UTC+1, vinh wrote:
When inserting docs, I store my own timestamp value in the default "_timestamp" field.  Then I search docs and fetch both "_source" and "_timestamp" fields.  I notice that the value of _source._timestamp correctly has my value.  But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am overriding it?

    }, {
      "_index" : "myindex",
      "_type" : "message",
      "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
      "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello there"},
      "fields" : {
        "_timestamp" : 1361672211995
      },
      "sort" : [ 1361672211995 ]
    } ]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.