Elasticsearch mapping for adding custom metadata payload to log entries in ELK stack

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Elasticsearch mapping for adding custom metadata payload to log entries in ELK stack

Tomi SF
We're using ELK stack for log management of a distributed application. In our log entries we have a field called "data" that can be of arbitrary type. We use this field to allow the developer to log any custom metadata releated to the incident that occurred. For example the following three entries could be samples of items we'd like to be able to log and store to Elasticsearch.

{"message" : "Fire Alarm", "level" : "emergency", "data" : { "Location" : 1, "Temperature" : 76.3 } }
{"message" : "Remote query exceeded warning timeout", "level" : "warning", "data" : 500}
{"message" : "UPS not available", "level" : "warning", "data" : { "Location" : "San Francisco", "Rack" : 1 } }

All other fields map nicely to Elasticsearch built-in field types but we don't know how to treat the "data" field as the value can be of any Elasticsearch type and different for each entry. When using the built-in Elasticsearch mapping shipping with Logstash, only the mapping of the first entry will be used. In the above example only the first message would be stored to the Elasticsearch index. 

It's okay if the data field is not indexed. But we'd still like to be able to store and retrieve it from elasticsearch. What kind of mapping document would work? I was trying different options but didn't seem to find one that would do what we needed. The logstash 1.5 default Elasticsearch mapping is below.

{
  "template" : "logstash-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true},
       "dynamic_templates" : [ {
         "message_field" : {
           "match" : "message",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true
           }
         }
       }, {
         "string_fields" : {
           "match" : "*",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true,
               "fields" : {
                 "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
               }
           }
         }
       } ],
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" },
                    "geoip"  : {
           "type" : "object",
             "dynamic": true,
             "path": "full",
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         }
       }
    }
  }
}



Thanks,

Tomi

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch mapping for adding custom metadata payload to log entries in ELK stack

dadoonet
Hi Tomi,


You could do it like this:

DELETE test
PUT test
{
  "mappings": {
    "doc": {
      "properties": {
        "data": {
          "type": "object",
          "enabled": false
        },
        "level": {
          "type": "string"
        },
        "message": {
          "type": "string"
        }
      }
    }
  }
}
PUT test/doc/1
{
  "message" : "Fire Alarm", 
  "level" : "emergency", 
  "data" : { 
    "Location" : 1, 
    "Temperature" : 76.3 
  }
}
PUT test/doc/2
{
  "message" : "Remote query exceeded warning timeout", 
  "level" : "warning", 
  "data" : 500
}
PUT test/doc/3
{
  "message" : "UPS not available", 
  "level" : "warning", 
  "data" : { 
    "Location" : "San Francisco", 
    "Rack" : 1 
  }
}

HTH

-- 
David Pilato | Technical Advocate | Elasticsearch.com



Le 8 févr. 2015 à 02:52, Tomi SF <[hidden email]> a écrit :

We're using ELK stack for log management of a distributed application. In our log entries we have a field called "data" that can be of arbitrary type. We use this field to allow the developer to log any custom metadata releated to the incident that occurred. For example the following three entries could be samples of items we'd like to be able to log and store to Elasticsearch.

{"message" : "Fire Alarm", "level" : "emergency", "data" : { "Location" : 1, "Temperature" : 76.3 } }
{"message" : "Remote query exceeded warning timeout", "level" : "warning", "data" : 500}
{"message" : "UPS not available", "level" : "warning", "data" : { "Location" : "San Francisco", "Rack" : 1 } }

All other fields map nicely to Elasticsearch built-in field types but we don't know how to treat the "data" field as the value can be of any Elasticsearch type and different for each entry. When using the built-in Elasticsearch mapping shipping with Logstash, only the mapping of the first entry will be used. In the above example only the first message would be stored to the Elasticsearch index. 

It's okay if the data field is not indexed. But we'd still like to be able to store and retrieve it from elasticsearch. What kind of mapping document would work? I was trying different options but didn't seem to find one that would do what we needed. The logstash 1.5 default Elasticsearch mapping is below.

{
  "template" : "logstash-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true},
       "dynamic_templates" : [ {
         "message_field" : {
           "match" : "message",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true
           }
         }
       }, {
         "string_fields" : {
           "match" : "*",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true,
               "fields" : {
                 "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
               }
           }
         }
       } ],
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" },
                    "geoip"  : {
           "type" : "object",
             "dynamic": true,
             "path": "full",
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         }
       }
    }
  }
}



Thanks,

Tomi


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/057E8068-E1A5-47CE-83C0-9B91B61AFA98%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.