unix timestamp (epoch) in date type

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

unix timestamp (epoch) in date type

Edward Fjellskål

From:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date

"The date type will also accept a long number representing UTC
milliseconds since the epoch, regardless of the format it can handle."

Does that mean that I can save the epoch time + "000" and that
will be a valid date field?

Example:
Epoch: 1381041397
ES-Epoch: 1381041397000

Will that work also for the Kibana @timestamp field out of the box etc?

Regards,
Edward

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: unix timestamp (epoch) in date type

Boaz Leskes
Hi Edward,

Assuming your epoch time is in second, then yes, you need to multiply by 1000.

About kibana - if it is mapped as a date field it will work - regardless of the input form - be it milliseconds epochs or an ISO formatted string.

Cheers,
Boaz

On Sunday, October 6, 2013 8:01:18 PM UTC+2, Edward Fjellskål wrote:

From:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date

"The date type will also accept a long number representing UTC
milliseconds since the epoch, regardless of the format it can handle."

Does that mean that I can save the epoch time + "000" and that
will be a valid date field?

Example:
Epoch: 1381041397
ES-Epoch: 1381041397000

Will that work also for the Kibana @timestamp field out of the box etc?

Regards,
Edward

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: unix timestamp (epoch) in date type

Edward Fjellskål
Hi Boaz,

So my epoch is not a date field:

"epoch" : {
  "type" : "long"
},

I can probably test this fairly easy, but would that above work for
kibana if I store epoch as "epoch X 1000" ?

or would I need change the mapping to:

"epoch" : {
  "type" : "date"
},

and insert "epoch x 1000" ?

Thanks for your resonse :)

Edward



On 10/09/2013 12:30 PM, Boaz Leskes wrote:

> Hi Edward,
>
> Assuming your epoch time is in second, then yes, you need to multiply by
> 1000.
>
> About kibana - if it is mapped as a date field it will work - regardless
> of the input form - be it milliseconds epochs or an ISO formatted string.
>
> Cheers,
> Boaz
>
> On Sunday, October 6, 2013 8:01:18 PM UTC+2, Edward Fjellskål wrote:
>
>
>     From:
>     http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
>     <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date>
>
>
>     "The date type will also accept a long number representing UTC
>     milliseconds since the epoch, regardless of the format it can handle."
>
>     Does that mean that I can save the epoch time + "000" and that
>     will be a valid date field?
>
>     Example:
>     Epoch: 1381041397
>     ES-Epoch: 1381041397000
>
>     Will that work also for the Kibana @timestamp field out of the box etc?
>
>     Regards,
>     Edward
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: unix timestamp (epoch) in date type

shadyabhi
Hi Edward,

"type" should be "date" for Kibana to work.

On Thu, Oct 10, 2013 at 9:20 PM, Edward Fjellskål
<[hidden email]> wrote:

> Hi Boaz,
>
> So my epoch is not a date field:
>
> "epoch" : {
>   "type" : "long"
> },
>
> I can probably test this fairly easy, but would that above work for
> kibana if I store epoch as "epoch X 1000" ?
>
> or would I need change the mapping to:
>
> "epoch" : {
>   "type" : "date"
> },
>
> and insert "epoch x 1000" ?
>
> Thanks for your resonse :)
>
> Edward
>
>
>
> On 10/09/2013 12:30 PM, Boaz Leskes wrote:
>> Hi Edward,
>>
>> Assuming your epoch time is in second, then yes, you need to multiply by
>> 1000.
>>
>> About kibana - if it is mapped as a date field it will work - regardless
>> of the input form - be it milliseconds epochs or an ISO formatted string.
>>
>> Cheers,
>> Boaz
>>
>> On Sunday, October 6, 2013 8:01:18 PM UTC+2, Edward Fjellskål wrote:
>>
>>
>>     From:
>>     http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
>>     <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date>
>>
>>
>>     "The date type will also accept a long number representing UTC
>>     milliseconds since the epoch, regardless of the format it can handle."
>>
>>     Does that mean that I can save the epoch time + "000" and that
>>     will be a valid date field?
>>
>>     Example:
>>     Epoch: 1381041397
>>     ES-Epoch: 1381041397000
>>
>>     Will that work also for the Kibana @timestamp field out of the box etc?
>>
>>     Regards,
>>     Edward
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to [hidden email].
>> For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.



--
Regards,
Abhijeet Rastogi (shadyabhi)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: unix timestamp (epoch) in date type

Boaz Leskes
HI Edward,

As Abhijeet has already said - for kibana to work you must have your epoch field must be of the type date and if you send longs, it must be in milliseconds since the epoch.

Cheers,
Boaz


On Fri, Oct 11, 2013 at 6:30 AM, Abhijeet Rastogi <[hidden email]> wrote:
Hi Edward,

"type" should be "date" for Kibana to work.

On Thu, Oct 10, 2013 at 9:20 PM, Edward Fjellskål
<[hidden email]> wrote:
> Hi Boaz,
>
> So my epoch is not a date field:
>
> "epoch" : {
>   "type" : "long"
> },
>
> I can probably test this fairly easy, but would that above work for
> kibana if I store epoch as "epoch X 1000" ?
>
> or would I need change the mapping to:
>
> "epoch" : {
>   "type" : "date"
> },
>
> and insert "epoch x 1000" ?
>
> Thanks for your resonse :)
>
> Edward
>
>
>
> On 10/09/2013 12:30 PM, Boaz Leskes wrote:
>> Hi Edward,
>>
>> Assuming your epoch time is in second, then yes, you need to multiply by
>> 1000.
>>
>> About kibana - if it is mapped as a date field it will work - regardless
>> of the input form - be it milliseconds epochs or an ISO formatted string.
>>
>> Cheers,
>> Boaz
>>
>> On Sunday, October 6, 2013 8:01:18 PM UTC+2, Edward Fjellskål wrote:
>>
>>
>>     From:
>>     http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
>>     <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date>
>>
>>
>>     "The date type will also accept a long number representing UTC
>>     milliseconds since the epoch, regardless of the format it can handle."
>>
>>     Does that mean that I can save the epoch time + "000" and that
>>     will be a valid date field?
>>
>>     Example:
>>     Epoch: 1381041397
>>     ES-Epoch: 1381041397000
>>
>>     Will that work also for the Kibana @timestamp field out of the box etc?
>>
>>     Regards,
>>     Edward
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to [hidden email].
>> For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.



--
Regards,
Abhijeet Rastogi (shadyabhi)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: unix timestamp (epoch) in date type

Edward Fjellskål
Thanks for the input...
Im coding up something now :)

Regards,
Edward

On 10/11/13, Boaz Leskes <[hidden email]> wrote:

> HI Edward,
>
> As Abhijeet has already said - for kibana to work you must have your epoch
> field must be of the type date and if you send longs, it must be in
> milliseconds since the epoch.
>
> Cheers,
> Boaz
>
>
> On Fri, Oct 11, 2013 at 6:30 AM, Abhijeet Rastogi
> <[hidden email]>wrote:
>
>> Hi Edward,
>>
>> "type" should be "date" for Kibana to work.
>>
>> On Thu, Oct 10, 2013 at 9:20 PM, Edward Fjellskål
>> <[hidden email]> wrote:
>> > Hi Boaz,
>> >
>> > So my epoch is not a date field:
>> >
>> > "epoch" : {
>> >   "type" : "long"
>> > },
>> >
>> > I can probably test this fairly easy, but would that above work for
>> > kibana if I store epoch as "epoch X 1000" ?
>> >
>> > or would I need change the mapping to:
>> >
>> > "epoch" : {
>> >   "type" : "date"
>> > },
>> >
>> > and insert "epoch x 1000" ?
>> >
>> > Thanks for your resonse :)
>> >
>> > Edward
>> >
>> >
>> >
>> > On 10/09/2013 12:30 PM, Boaz Leskes wrote:
>> >> Hi Edward,
>> >>
>> >> Assuming your epoch time is in second, then yes, you need to multiply
>> >> by
>> >> 1000.
>> >>
>> >> About kibana - if it is mapped as a date field it will work -
>> >> regardless
>> >> of the input form - be it milliseconds epochs or an ISO formatted
>> string.
>> >>
>> >> Cheers,
>> >> Boaz
>> >>
>> >> On Sunday, October 6, 2013 8:01:18 PM UTC+2, Edward Fjellskål wrote:
>> >>
>> >>
>> >>     From:
>> >>
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
>> >>     <
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
>> >
>> >>
>> >>
>> >>     "The date type will also accept a long number representing UTC
>> >>     milliseconds since the epoch, regardless of the format it can
>> handle."
>> >>
>> >>     Does that mean that I can save the epoch time + "000" and that
>> >>     will be a valid date field?
>> >>
>> >>     Example:
>> >>     Epoch: 1381041397
>> >>     ES-Epoch: 1381041397000
>> >>
>> >>     Will that work also for the Kibana @timestamp field out of the box
>> etc?
>> >>
>> >>     Regards,
>> >>     Edward
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups "elasticsearch" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an email to [hidden email].
>> >> For more options, visit https://groups.google.com/groups/opt_out.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to [hidden email].
>> > For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>> --
>> Regards,
>> Abhijeet Rastogi (shadyabhi)
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>


--
Edward Bjarte Fjellskål
Senior Security Analyst
http://www.gamelinux.org/

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: unix timestamp (epoch) in date type

karnamonkster
Does it work? after changing the {"type" : "date"}
And setting a date format - 

{
      "messages" : {
        "_timestamp" : {
          "enabled" : true
        },
        "properties" : {
          "app_event_time" : {
            "type" : "date",
            "format" : "yyyy/MM/dd HH:mm:ss"
          },
          "event_time" : {
            "type" : "date",
            "format" : "yyyy/MM/dd HH:mm:ss"
          },

What could be the problem?

On Saturday, October 12, 2013 1:10:39 AM UTC+5:30, Edward Fjellskål wrote:
Thanks for the input...
Im coding up something now :)

Regards,
Edward

On 10/11/13, Boaz Leskes <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="kgvF1LGgYG8J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">b.le...@...> wrote:

> HI Edward,
>
> As Abhijeet has already said - for kibana to work you must have your epoch
> field must be of the type date and if you send longs, it must be in
> milliseconds since the epoch.
>
> Cheers,
> Boaz
>
>
> On Fri, Oct 11, 2013 at 6:30 AM, Abhijeet Rastogi
> <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="kgvF1LGgYG8J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">abhije...@...>wrote:
>
>> Hi Edward,
>>
>> "type" should be "date" for Kibana to work.
>>
>> On Thu, Oct 10, 2013 at 9:20 PM, Edward Fjellskål
>> <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="kgvF1LGgYG8J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">edwardfj...@...> wrote:
>> > Hi Boaz,
>> >
>> > So my epoch is not a date field:
>> >
>> > "epoch" : {
>> >   "type" : "long"
>> > },
>> >
>> > I can probably test this fairly easy, but would that above work for
>> > kibana if I store epoch as "epoch X 1000" ?
>> >
>> > or would I need change the mapping to:
>> >
>> > "epoch" : {
>> >   "type" : "date"
>> > },
>> >
>> > and insert "epoch x 1000" ?
>> >
>> > Thanks for your resonse :)
>> >
>> > Edward
>> >
>> >
>> >
>> > On 10/09/2013 12:30 PM, Boaz Leskes wrote:
>> >> Hi Edward,
>> >>
>> >> Assuming your epoch time is in second, then yes, you need to multiply
>> >> by
>> >> 1000.
>> >>
>> >> About kibana - if it is mapped as a date field it will work -
>> >> regardless
>> >> of the input form - be it milliseconds epochs or an ISO formatted
>> string.
>> >>
>> >> Cheers,
>> >> Boaz
>> >>
>> >> On Sunday, October 6, 2013 8:01:18 PM UTC+2, Edward Fjellskål wrote:
>> >>
>> >>
>> >>     From:
>> >>
>> <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-core-types.html%23date\46sa\75D\46sntz\0751\46usg\75AFQjCNHT6YO05hD5ZujCglHv4Vv14UJ0PQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-core-types.html%23date\46sa\75D\46sntz\0751\46usg\75AFQjCNHT6YO05hD5ZujCglHv4Vv14UJ0PQ';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
>> >>     <
>> <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-core-types.html%23date\46sa\75D\46sntz\0751\46usg\75AFQjCNHT6YO05hD5ZujCglHv4Vv14UJ0PQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-core-types.html%23date\46sa\75D\46sntz\0751\46usg\75AFQjCNHT6YO05hD5ZujCglHv4Vv14UJ0PQ';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
>> >
>> >>
>> >>
>> >>     "The date type will also accept a long number representing UTC
>> >>     milliseconds since the epoch, regardless of the format it can
>> handle."
>> >>
>> >>     Does that mean that I can save the epoch time + "000" and that
>> >>     will be a valid date field?
>> >>
>> >>     Example:
>> >>     Epoch: 1381041397
>> >>     ES-Epoch: 1381041397000
>> >>
>> >>     Will that work also for the Kibana @timestamp field out of the box
>> etc?
>> >>
>> >>     Regards,
>> >>     Edward
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups "elasticsearch" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="kgvF1LGgYG8J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
>> >> For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="kgvF1LGgYG8J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
>> > For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.
>>
>>
>>
>> --
>> Regards,
>> Abhijeet Rastogi (shadyabhi)
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="kgvF1LGgYG8J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
> For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.
>


--
Edward Bjarte Fjellskål
Senior Security Analyst
<a href="http://www.gamelinux.org/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.gamelinux.org%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFU5NEaIRrEbRLQ58NTdXwHLV0Mcw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.gamelinux.org%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFU5NEaIRrEbRLQ58NTdXwHLV0Mcw';return true;">http://www.gamelinux.org/

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/299b6673-b6e2-4662-8e10-b50cb78c9fad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: unix timestamp (epoch) in date type

karnamonkster


Does it work? after changing the {"type" : "date"}
And setting a date format - 

{
      "messages" : {
        "_timestamp" : {
          "enabled" : true
        },
        "properties" : {
          "app_event_time" : {
            "type" : "date",
            "format" : "yyyy/MM/dd HH:mm:ss"
          },
          "event_time" : {
            "type" : "date",
            "format" : "yyyy/MM/dd HH:mm:ss"
          },

What could be the problem?
Cause mine does not work

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0f3bfa6a-c8b1-4e68-8230-1a0eb8f61d33%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.