Cannot search in sub-attribute

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Cannot search in sub-attribute

Chris Kinsalb
Hello,

I created  a doc type which has several attributes including an attribute @fields configured like this:

"@fields": {
              "type": "object",
              "index": "not_analyzed",
              "dynamic": true,
              "path": "full"
            }


Here is a sample input:

{
  "@fields": {
    "hostname": "walter",
    "ip": "192.168.10.237",
    "user": {
      "firstname": "Donny",
      "lastname": "Kerabatsos"
    },
    "message": "Shut the f* up, Donny"
  }
}


Now I can search for a doc by referencing the  hostname like this:

{
  "query": {
    "filtered": {
      "filter": {
        "and": [
          {
            "term": {
              "@fields.hostname": "walter"
            }
          }
        ]
      }
    }
  }
}


But this search yields no result:


{
  "query": {
    "filtered": {
      "filter": {
        "and": [
          {
            "term": {
              "@fields.user.firstname": "Donny"
            }
          }
        ]
      }
    }
  }
}


I don't understand why, but perhaps I misunderstood something about the object type. My impression was, that it is possible to insert an arbitrary json and perform searches on any of it's attributes (also "nested" attributes), due to the dynamic mapping.

1.) What do I have to do in order to make this work?
2.) I DO not want to create a default schema for this, since the json data which is inserted into @fields changes (different hierarchies, attribute names, etc.).

Thanks a lot & cheers,
- Chris




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Cannot search in sub-attribute

kimchy
Administrator
you should use the match query instead of the term query, since the internal fields ends up being analyzed.

I see you try to set the index field on the object mapping to mark it as not_analyzed, this will not work and is not supported (I think you assume that al internal mapping will inherit it, which they do not). if you want to achieve this, check into dynamic templates in the root object mappings. 


On Sat, Jun 1, 2013 at 4:18 PM, Chris Kinsalb <[hidden email]> wrote:

Hello,

I created  a doc type which has several attributes including an attribute @fields configured like this:

"@fields": {
              "type": "object",
              "index": "not_analyzed",
              "dynamic": true,
              "path": "full"
            }


Here is a sample input:

{
  "@fields": {
    "hostname": "walter",
    "ip": "192.168.10.237",
    "user": {
      "firstname": "Donny",
      "lastname": "Kerabatsos"
    },
    "message": "Shut the f* up, Donny"
  }
}


Now I can search for a doc by referencing the  hostname like this:

{
  "query": {
    "filtered": {
      "filter": {
        "and": [
          {
            "term": {
              "@fields.hostname": "walter"
            }
          }
        ]
      }
    }
  }
}


But this search yields no result:


{
  "query": {
    "filtered": {
      "filter": {
        "and": [
          {
            "term": {
              "@fields.user.firstname": "Donny"
            }
          }
        ]
      }
    }
  }
}


I don't understand why, but perhaps I misunderstood something about the object type. My impression was, that it is possible to insert an arbitrary json and perform searches on any of it's attributes (also "nested" attributes), due to the dynamic mapping.

1.) What do I have to do in order to make this work?
2.) I DO not want to create a default schema for this, since the json data which is inserted into @fields changes (different hierarchies, attribute names, etc.).

Thanks a lot & cheers,
- Chris




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Cannot search in sub-attribute

Chris Kinsalb
Hello Shay,

thanks for the explanation.
Yes, that is indeed what I thought. However, the problem is that I wanted my input to be totally schema-less, which I cannot seem to achieve using dynamic templates since I will not know what the input objects will look like.

However, I do have an idea on how to achieve this:

Using dynamic templates could I do something along these lines?

        "dynamic_templates": [
            {
              "fields_template": {
                "mapping": {
                  "index": "not_analyzed"
                },
                "match": "fields.*"
              }
            }
          ]


Now, whenever I get a json input I would transform it to a flat hierarchy, e.g.:

input:

{
  foo: 'bar',
  user: {
    firstname: 'bla',
    lastname: 'narf'
  }
}


Transform output:

{
  fields.foo: 'bar',
  fields.user.firstname: 'bla',
  fields.user.lastname: 'narf'
}

This way:

- each field will automatically have the correct type (e.g. string or integer)
- I can run facet queries (statistics) on the fields


Does this sound like a viable solution to you?
(will try it out now)

Cheers & regards,
- Chris


On Saturday, June 1, 2013 5:19:40 PM UTC+2, kimchy wrote:
you should use the match query instead of the term query, since the internal fields ends up being analyzed.

I see you try to set the index field on the object mapping to mark it as not_analyzed, this will not work and is not supported (I think you assume that al internal mapping will inherit it, which they do not). if you want to achieve this, check into dynamic templates in the root object mappings. 


On Sat, Jun 1, 2013 at 4:18 PM, Chris Kinsalb <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="4_HrLcBcEBQJ">hc.ki...@...> wrote:

Hello,

I created  a doc type which has several attributes including an attribute @fields configured like this:

"@fields": {
              "type": "object",
              "index": "not_analyzed",
              "dynamic": true,
              "path": "full"
            }


Here is a sample input:

{
  "@fields": {
    "hostname": "walter",
    "ip": "192.168.10.237",
    "user": {
      "firstname": "Donny",
      "lastname": "Kerabatsos"
    },
    "message": "Shut the f* up, Donny"
  }
}


Now I can search for a doc by referencing the  hostname like this:

{
  "query": {
    "filtered": {
      "filter": {
        "and": [
          {
            "term": {
              "@fields.hostname": "walter"
            }
          }
        ]
      }
    }
  }
}


But this search yields no result:


{
  "query": {
    "filtered": {
      "filter": {
        "and": [
          {
            "term": {
              "@fields.user.firstname": "Donny"
            }
          }
        ]
      }
    }
  }
}


I don't understand why, but perhaps I misunderstood something about the object type. My impression was, that it is possible to insert an arbitrary json and perform searches on any of it's attributes (also "nested" attributes), due to the dynamic mapping.

1.) What do I have to do in order to make this work?
2.) I DO not want to create a default schema for this, since the json data which is inserted into @fields changes (different hierarchies, attribute names, etc.).

Thanks a lot & cheers,
- Chris




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="4_HrLcBcEBQJ">elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Cannot search in sub-attribute

Chris Kinsalb
fyi:

My own suggested solution works as intended. I can now add arbitrarily nested objects, with each attribute flattened down to its full path.

The query to server:9200/index/{doctype}/ _mapping shows that the field types are created correctly was well.


- chris


On Monday, June 3, 2013 8:18:54 AM UTC+2, Chris Kinsalb wrote:
Hello Shay,

thanks for the explanation.
Yes, that is indeed what I thought. However, the problem is that I wanted my input to be totally schema-less, which I cannot seem to achieve using dynamic templates since I will not know what the input objects will look like.

However, I do have an idea on how to achieve this:

Using dynamic templates could I do something along these lines?

        "dynamic_templates": [
            {
              "fields_template": {
                "mapping": {
                  "index": "not_analyzed"
                },
                "match": "fields.*"
              }
            }
          ]


Now, whenever I get a json input I would transform it to a flat hierarchy, e.g.:

input:

{
  foo: 'bar',
  user: {
    firstname: 'bla',
    lastname: 'narf'
  }
}


Transform output:

{
  fields.foo: 'bar',
  fields.user.firstname: 'bla',
  fields.user.lastname: 'narf'
}

This way:

- each field will automatically have the correct type (e.g. string or integer)
- I can run facet queries (statistics) on the fields


Does this sound like a viable solution to you?
(will try it out now)

Cheers & regards,
- Chris


On Saturday, June 1, 2013 5:19:40 PM UTC+2, kimchy wrote:
you should use the match query instead of the term query, since the internal fields ends up being analyzed.

I see you try to set the index field on the object mapping to mark it as not_analyzed, this will not work and is not supported (I think you assume that al internal mapping will inherit it, which they do not). if you want to achieve this, check into dynamic templates in the root object mappings. 


On Sat, Jun 1, 2013 at 4:18 PM, Chris Kinsalb <[hidden email]> wrote:

Hello,

I created  a doc type which has several attributes including an attribute @fields configured like this:

"@fields": {
              "type": "object",
              "index": "not_analyzed",
              "dynamic": true,
              "path": "full"
            }


Here is a sample input:

{
  "@fields": {
    "hostname": "walter",
    "ip": "192.168.10.237",
    "user": {
      "firstname": "Donny",
      "lastname": "Kerabatsos"
    },
    "message": "Shut the f* up, Donny"
  }
}


Now I can search for a doc by referencing the  hostname like this:

{
  "query": {
    "filtered": {
      "filter": {
        "and": [
          {
            "term": {
              "@fields.hostname": "walter"
            }
          }
        ]
      }
    }
  }
}


But this search yields no result:


{
  "query": {
    "filtered": {
      "filter": {
        "and": [
          {
            "term": {
              "@fields.user.firstname": "Donny"
            }
          }
        ]
      }
    }
  }
}


I don't understand why, but perhaps I misunderstood something about the object type. My impression was, that it is possible to insert an arbitrary json and perform searches on any of it's attributes (also "nested" attributes), due to the dynamic mapping.

1.) What do I have to do in order to make this work?
2.) I DO not want to create a default schema for this, since the json data which is inserted into @fields changes (different hierarchies, attribute names, etc.).

Thanks a lot & cheers,
- Chris




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.