Convert unix timestamp (seconds) to java (milliseconds)

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Convert unix timestamp (seconds) to java (milliseconds)

Nick Wood
Greetings,

My primary data store has a date_created field stored as a unix timestamp.  I've verified that if I multiply that value by 1000 and then index it, Elasticsearch correctly picks the field up as a date.  But updating the data in my primary data store is not very feasible, therefore, I'm looking for some way to have Elasticsearch do the conversion for me.  Lazy, eh?

I've tried adding this to my mapping

--------------------
        "transform" : {
            "script" : "ctx._source['date_created'] = ctx._source['date_created'] * 1000;",
            "lang": "groovy"
        },
        "properties" : { ...
--------------------

I was expecting that to multiply the date_created field by 1000 prior to parsing the field as a date, but it doesn't seem to be working.

Any ideas what I might be doing wrong, or suggestions for an alternate approach?

  Nick

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5544e84b-06e5-4199-adff-17bd4039ac19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Convert unix timestamp (seconds) to java (milliseconds)

Nick Wood
Replying to my own question for other searchers:

My problem had to do with types.  The mapping that ended up working is:

{
    "test" : {
"transform" : {
            "script" : "ctx._source['date_created'] = ctx._source['date_created'].toLong() * 1000",
            "lang": "groovy"
        },
        "properties" : {
            "date_created" : {"type" : "date"},
        }
    }
}

Note the "toLong()".

Also, once a script is modified, it appears as though the type has to be deleted and the mapping re-applied for the changes to take effect.

  Nick

On Wednesday, December 3, 2014 2:52:10 PM UTC-7, Nick Wood wrote:
Greetings,

My primary data store has a date_created field stored as a unix timestamp.  I've verified that if I multiply that value by 1000 and then index it, Elasticsearch correctly picks the field up as a date.  But updating the data in my primary data store is not very feasible, therefore, I'm looking for some way to have Elasticsearch do the conversion for me.  Lazy, eh?

I've tried adding this to my mapping

--------------------
        "transform" : {
            "script" : "ctx._source['date_created'] = ctx._source['date_created'] * 1000;",
            "lang": "groovy"
        },
        "properties" : { ...
--------------------

I was expecting that to multiply the date_created field by 1000 prior to parsing the field as a date, but it doesn't seem to be working.

Any ideas what I might be doing wrong, or suggestions for an alternate approach?

  Nick

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/991be6f5-31dc-444c-8748-383b8262d074%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Convert unix timestamp (seconds) to java (milliseconds)

Nikolas Everett

Transform doesn't change the source, just how it is indexed. I made it that way because I figured I'd you want to change the source you can do it on the application feeding elasticsearch. Transform is a way to index stuff but leave it out of the source. Its copy_to on steroids.

Another reason transform doesn't change the source is that it'd make updates and reindexing (like scan/scroll from one index to another) complex.

There is an option that will retransform results on the fly but I don't know it offhand. I believe its on the transform page in the docs.

Nik

On Dec 27, 2014 7:10 PM, "Nick Wood" <[hidden email]> wrote:
Replying to my own question for other searchers:

My problem had to do with types.  The mapping that ended up working is:

{
    "test" : {
"transform" : {
            "script" : "ctx._source['date_created'] = ctx._source['date_created'].toLong() * 1000",
            "lang": "groovy"
        },
        "properties" : {
            "date_created" : {"type" : "date"},
        }
    }
}

Note the "toLong()".

Also, once a script is modified, it appears as though the type has to be deleted and the mapping re-applied for the changes to take effect.

  Nick

On Wednesday, December 3, 2014 2:52:10 PM UTC-7, Nick Wood wrote:
Greetings,

My primary data store has a date_created field stored as a unix timestamp.  I've verified that if I multiply that value by 1000 and then index it, Elasticsearch correctly picks the field up as a date.  But updating the data in my primary data store is not very feasible, therefore, I'm looking for some way to have Elasticsearch do the conversion for me.  Lazy, eh?

I've tried adding this to my mapping

--------------------
        "transform" : {
            "script" : "ctx._source['date_created'] = ctx._source['date_created'] * 1000;",
            "lang": "groovy"
        },
        "properties" : { ...
--------------------

I was expecting that to multiply the date_created field by 1000 prior to parsing the field as a date, but it doesn't seem to be working.

Any ideas what I might be doing wrong, or suggestions for an alternate approach?

  Nick

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/991be6f5-31dc-444c-8748-383b8262d074%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2E8qCT7GRfLYJo_GzB_5x%2BC%3D_FuJ49M%2BNfD%3D5Lhi4zCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.