Need review for my REST query (template modification)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Need review for my REST query (template modification)

Aldian
Hi!

I am using the usual ELK stack with the default template (http://pastebin.com/DtYiazVr). In every log message, the date in stored in field named "log_date", which the date filter converts in a "@timestamp". I want to set the "log_date" field as "not_analyzed" so that I can sort it in Kibana without getting weird results.

I built the following query 

curl -XPUT localhost:9200/_template/template_1 -d '
{
    "template" : "logstash-*",
    "properties" : {
        "log_date" : {
                "type" : "string",
                "index" : "not_analyzed"
        }
    }
}

Can you confirm that the request is correct? I have doubts about the template name. I thought about calling url localhost:9200/_template/logstash in order to modify the existing template rather than creating a new one, but I am afraid of what could happen the day I restart logstash, so my thinking is that if all works as intended, both logstash default template and that one will apply.

Also I believe that templates are only about future data. Is there any way to retro apply it back on existing indexes?

Thanks for your help

Aldian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1c7fe34e-7792-409d-83ab-3c39fe883e95%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Need review for my REST query (template modification)

Aldian
Don't tell me nobody here ever made such a simple request?

On Thursday, January 22, 2015 at 11:57:26 AM UTC+1, Aldian wrote:
Hi!

I am using the usual ELK stack with the default template (<a href="http://www.google.com/url?q=http%3A%2F%2Fpastebin.com%2FDtYiazVr&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNEIrorp9e9dSh98HTjQoPQoEor3Vg" rel="nofollow" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpastebin.com%2FDtYiazVr\46sa\75D\46sntz\0751\46usg\75AFQjCNEIrorp9e9dSh98HTjQoPQoEor3Vg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fpastebin.com%2FDtYiazVr\46sa\75D\46sntz\0751\46usg\75AFQjCNEIrorp9e9dSh98HTjQoPQoEor3Vg';return true;">http://pastebin.com/DtYiazVr). In every log message, the date in stored in field named "log_date", which the date filter converts in a "@timestamp". I want to set the "log_date" field as "not_analyzed" so that I can sort it in Kibana without getting weird results.

I built the following query 

curl -XPUT localhost:9200/_template/template_1 -d '
{
    "template" : "logstash-*",
    "properties" : {
        "log_date" : {
                "type" : "string",
                "index" : "not_analyzed"
        }
    }
}

Can you confirm that the request is correct? I have doubts about the template name. I thought about calling url localhost:9200/_template/logstash in order to modify the existing template rather than creating a new one, but I am afraid of what could happen the day I restart logstash, so my thinking is that if all works as intended, both logstash default template and that one will apply.

Also I believe that templates are only about future data. Is there any way to retro apply it back on existing indexes?

Thanks for your help

Aldian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2de42363-b3a8-4561-bda8-b844d4c287a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Need review for my REST query (template modification)

Magnus Bäck
In reply to this post by Aldian
On Thursday, January 22, 2015 at 11:57 CET,
     Aldian <[hidden email]> wrote:

> I am using the usual ELK stack with the default template
> ([1]http://pastebin.com/DtYiazVr). In every log message, the date in
> stored in field named "log_date", which the date filter converts in a
> "@timestamp". I want to set the "log_date" field as "not_analyzed" so
> that I can sort it in Kibana without getting weird results.

You're storing the same timestamp in two fields? Why?

> I built the following query
> curl -XPUT localhost:9200/_template/template_1 -d '
> {
>     "template" : "logstash-*",
>     "properties" : {
>         "log_date" : {
>                 "type" : "string",
>                 "index" : "not_analyzed"
>         }
>     }
> }
> Can you confirm that the request is correct?

It looks okay. You may want to use the 'order' key to make sure these
two matching index templates are applied in a well-defined and obvious
order.

> I have doubts about the template name. I thought about calling url
> localhost:9200/_template/logstash in order to modify the existing
> template rather than creating a new one, but I am afraid of what could
> happen the day I restart logstash, so my thinking is that if all works
> as intended, both logstash default template and that one will apply.

I disable Logstash's index template handling and maintain my own
template (that started out as a copy of Logstash's).

> Also I believe that templates are only about future data. Is there any
> way to retro apply it back on existing indexes?

You'll have to reindex the data, e.g. using es-reindex.

https://github.com/geronime/es-reindex

--
Magnus Bäck                | Software Engineer, Development Tools
[hidden email] | Sony Mobile Communications

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150129122021.GE30366%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Need review for my REST query (template modification)

Aldian
In reply to this post by Aldian
Thank you for replying, I am going to give it a try. 

As for your question, the way Kibana displays timestamps is a bit cryptic. The content of the log_date field is way more readable by a human being. 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9553d7cf-584b-4e35-a148-332e30934d7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Need review for my REST query (template modification)

Aldian
In reply to this post by Aldian
Hi 

I tried the request today and got back an "{"acknowledged":true}". Then I tried

curl -XGET localhost:9200/_template/logstashdate
{"logstashdate":{"order":1,"template":"logstash-*","settings":{},"mappings":{},"aliases":{}}}

As you can see the properties have been completely ignored. I suspect I should have included those into the mapping field, something like what is done with the default template. Next I am planning to try this one:

curl -XPUT localhost:9200/_template/logstashdate -d '
{
    "order": 1,
"template" : "logstash-*",
"mappings": {
"_default_": {
"properties" : {
"log_date" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}
}

I am mostly guessing there, as the documentation is mostly example based and I did not find the relevant one.


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/49f241d0-131e-4c4f-b78e-ca17f55878d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.