inexplicable behavior using 'terms' filter

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

inexplicable behavior using 'terms' filter

Luca Puccini
Hello everyone,

I'm pretty new to ES, so I hope I won't be writing any blasfemity. In this case please forgive me.

I have an ES index
{
   
"settings": {
       
"analysis": {
           
"analyzer": {
               
"de_analyzer": {
                   
"type": "snowball",
                   
"language": "German"
               
}
           
}
       
}
   
}
}

with an ES map
uselessF

The data, retrieved by a river from a mongodb instance, are in the form:
{
   
"someInfo1" : {
       
...
   
}
   
"someInfo1" : {
       
...
   
}
   
"someInfo1" : {
       
...
   
}
   
"importantInfo" : {
       
"field1": true,
       
"field2": 2123848121,
       
...
       
"magnetId": "stringId1",
       
"type": "stringType"
   
}
}

all data are correctly indexed by the river, but when I execute a query with the terms filter in it I got no results, even though there are some in the index, of this I'm sure (please notice in the query below that importantInfo.importantFlag is not present in the map).
{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "peer"
               
}
           
},
           
"filter": {
               
"and": [
                   
{
                       
"term": {
                           
"importantInfo.importantField": "stringField"
                       
}
                   
},
                   
{
                       
"term": {
                           
"importantInfo.importantFlag": true
                       
}
                   
},
                   
{
                       
"terms" : {
                           
"importantInfo.importantId": [ "stringId1", "stringId2" ]
                       
}
                   
}

               
]
           
}
       
}
   
},
   
"size": 10,
   
"from": 0,
   
"fields": [
       
"importantInfo.toBeSearched1",
       
"importantInfo.toBeSearched2",
       
"importantInfo.importantId"
   
]
}

If I use the following query instead, I got some results that respect all the filters

{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "peer"
               
}
           
},
           
"filter": {
               
"and": [
                   
{
                       
"term": {
                           
"importantInfo.importantField": "stringField"
                       
}
                   
},
                   
{
                       
"term": {
                           
"importantInfo.importantFlag": true
                       
}
                   
}
               
]
           
}
       
}
   
},
   
"size": 10,
   
"from": 0,
   
"fields": [
       
"importantInfo.toBeSearched1",
       
"importantInfo.toBeSearched2",
       
"importantInfo.importantId"
   
]
}

Also, when I use terms, for example, on importantField instead of importantId I have correct results as well.
{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "peer"
               
}
           
},
           
"filter": {
               
"and": [
                    {
                      "terms" : {
                        
"importantInfo.importantField": [ "
stringField1", "stringField2" ]
                     
}
                 
}
,
                    {
                       
"term": {
                           
"importantInfo.importantFlag": true
                       
}
                   
}
               
]
           
}
       
}
   
},
   
"size": 10,
   
"from": 0,
   
"fields": [
       
"importantInfo.toBeSearched1",
       
"importantInfo.toBeSearched2",
       
"importantInfo.importantId"
   
]
}

I'm trying hard to understand what could possibly go wrong with that filter. The field is mapped exactly as the others, but it cannot be used to filter results.

I'm setting up a script to simulate this behavior and I will post it here soon as it'll be ready, but in the meantime maybe someone can spot some error I've made or option I've missed.

thanks in advance
     L

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eb92d62d-db0c-4ab4-9aa3-3ae03de2b4b0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: inexplicable behavior using 'terms' filter

Luca Puccini
Of course the map is wrong. here's the right one

ES map:
{
   
"mapName": {
     
"_all" : {"enabled" : false},
     
"properties": {
       
"uselessField1": {
           
"type": "boolean",
           
"index": "no"
       
},
       
"uselessField2": {
           
"type": "long",
           
"index": "no"
       
},
       
...
       
"toBeSearched1": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"toBeSearched2": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"importantId": {
           
"type": "string",
           
"index": "not_analyzed"
       
},
       
"importantField": {
           
"type": "string",
           
"index": "not_analyzed"
       
}
     
}
   
}
}


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d34e9092-e6f4-4a9f-b545-662154ea80d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: inexplicable behavior using 'terms' filter

simonw-2
luca, what do the values in 'importantId` look like? Are they identical to the values you pass in? The field is not analyzed though. It would also be helpful if you could show the results that you would expect so we can see what values they have.

simon

On Thursday, April 24, 2014 5:53:42 PM UTC+2, Luca wrote:
Of course the map is wrong. here's the right one

ES map:
{
   
"mapName": {
     
"_all" : {"enabled" : false},
     
"properties": {
       
"uselessField1": {
           
"type": "boolean",
           
"index": "no"
       
},
       
"uselessField2": {
           
"type": "long",
           
"index": "no"
       
},
       
...
       
"toBeSearched1": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"toBeSearched2": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"importantId": {
           
"type": "string",
           
"index": "not_analyzed"
       
},
       
"importantField": {
           
"type": "string",
           
"index": "not_analyzed"
       
}
     
}
   
}
}


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: inexplicable behavior using 'terms' filter

Luca Puccini
Hi Simon, thank you for your answer:

the values of importantId field are a finite set of strings. Those values are 100% identical to the ones required in the query. They are not analyzed because, correct me if I'm wrong, here says that the filter works on not analyzed fields.

Just to be clear, suppose that importantId can assume these values: ['apple', 'pear', 'orange', 'apricot', 'banana']
I want the filter to be able to eliminate all the elements that are not ['apple', 'banana'].

About the results I'm building an example, and soon as it will be running I'll send it here, hopefully by the end of the day.

thanks again!
   Luca

--

Luca Puccini
dev @ mixd.tv

Skype: lucap283



On Thu, Apr 24, 2014 at 6:42 PM, simonw <[hidden email]> wrote:
luca, what do the values in 'importantId` look like? Are they identical to the values you pass in? The field is not analyzed though. It would also be helpful if you could show the results that you would expect so we can see what values they have.

simon


On Thursday, April 24, 2014 5:53:42 PM UTC+2, Luca wrote:
Of course the map is wrong. here's the right one

ES map:
{
   
"mapName": {
     
"_all" : {"enabled" : false},
     
"properties": {
       
"uselessField1": {
           
"type": "boolean",
           
"index": "no"
       
},
       
"uselessField2": {
           
"type": "long",
           
"index": "no"
       
},
       
...
       
"toBeSearched1": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"toBeSearched2": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"importantId": {
           
"type": "string",
           
"index": "not_analyzed"
       
},
       
"importantField": {
           
"type": "string",
           
"index": "not_analyzed"
       
}
     
}
   
}
}


--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2B0cMOLieuCoqmLousA_ZK%2B6DbqToB%3DxnM8LL8u95ZVCKKAi1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: inexplicable behavior using 'terms' filter

Luca Puccini

Attached you can find a script that basically create an index, a map and add 3 elements to that.

the following queries returns respectively 1 (correctly) and 0 results

{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "termA"
               
}
           
},
           
"filter": {
               
"and": [
                   
{
                       
"term": {
                           
"importantInfo.fieldNotInMap": true
                       
}
                   
}
               
]
           
}
       
}
   
}
}
{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "termA"
               
}
           
},
           
"filter": {
               
"and": [
                   
{
                       
"term": {
                           
"importantInfo.fieldNotInMap": true
                       
}
                   
},
                   
{
                       
"term" : {
                           
"importantInfo.importantField": "typeA"
                       
}
                   
},
                   
{
                       
"terms" : {
                           
"importantInfo.importantId": [ "stringId2", "stringId1" ]
                       
}
                   
}
               
]
           
}
       
}
   
}
}

As I told you before this is not 100% my problem. On my system I can actually filter using terms on another field (importantField in the above example) that is exactly of the same type of importantId. I haven't been able to reproduce that. BTW if I could understand how to manage terms in this case in order to have the expected result it would be great.

thanks
   Luca



On Friday, April 25, 2014 4:37:00 PM UTC+2, Luca wrote:
Hi Simon, thank you for your answer:

the values of importantId field are a finite set of strings. Those values are 100% identical to the ones required in the query. They are not analyzed because, correct me if I'm wrong, <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-terms-filter.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fquery-dsl-terms-filter.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFV-34jiB7ukB0slKhYyr4te0aL1g';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fquery-dsl-terms-filter.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFV-34jiB7ukB0slKhYyr4te0aL1g';return true;">here says that the filter works on not analyzed fields.

Just to be clear, suppose that importantId can assume these values: ['apple', 'pear', 'orange', 'apricot', 'banana']
I want the filter to be able to eliminate all the elements that are not ['apple', 'banana'].

About the results I'm building an example, and soon as it will be running I'll send it here, hopefully by the end of the day.

thanks again!
   Luca



On Thu, Apr 24, 2014 at 6:42 PM, simonw <[hidden email]> wrote:
luca, what do the values in 'importantId` look like? Are they identical to the values you pass in? The field is not analyzed though. It would also be helpful if you could show the results that you would expect so we can see what values they have.

simon


On Thursday, April 24, 2014 5:53:42 PM UTC+2, Luca wrote:
Of course the map is wrong. here's the right one

ES map:
{
   
"mapName": {
     
"_all" : {"enabled" : false},
     
"properties": {
       
"uselessField1": {
           
"type": "boolean",
           
"index": "no"
       
},
       
"uselessField2": {
           
"type": "long",
           
"index": "no"
       
},
       
...
       
"toBeSearched1": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"toBeSearched2": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"importantId": {
           
"type": "string",
           
"index": "not_analyzed"
       
},
       
"importantField": {
           
"type": "string",
           
"index": "not_analyzed"
       
}
     
}
   
}
}


--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit <a href="https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe" target="_blank" onmousedown="this.href='https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe';return true;" onclick="this.href='https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe';return true;">https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cb904048-7ce7-4018-9792-510e97b851c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

elasticSearchExample.sh (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: inexplicable behavior using 'terms' filter

Luca Cavanna
Hi Luca,
I think you should have a look at the analyze and check how your different fields end up in the index. If the second query doesn't return any result it means that the filters you added there don't match any document.

What I also noticed is that in your mapping you refer to importantId and so on, but they are under the importantInfo object in your documents. Have a look at your generated mapping too using the get mapping api and see if what you get back is what you submitted or maybe it contains additional fields with default text analysis.

Cheers
Luca

On Friday, April 25, 2014 5:39:22 PM UTC+2, Luca wrote:

Attached you can find a script that basically create an index, a map and add 3 elements to that.

the following queries returns respectively 1 (correctly) and 0 results

{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "termA"
               
}
           
},
           
"filter": {
               
"and": [
                   
{
                       
"term": {
                           
"importantInfo.fieldNotInMap": true
                       
}
                   
}
               
]
           
}
       
}
   
}
}
{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "termA"
               
}
           
},
           
"filter": {
               
"and": [
                   
{
                       
"term": {
                           
"importantInfo.fieldNotInMap": true
                       
}
                   
},
                   
{
                       
"term" : {
                           
"importantInfo.importantField": "typeA"
                       
}
                   
},
                   
{
                       
"terms" : {
                           
"importantInfo.importantId": [ "stringId2", "stringId1" ]
                       
}
                   
}
               
]
           
}
       
}
   
}
}

As I told you before this is not 100% my problem. On my system I can actually filter using terms on another field (importantField in the above example) that is exactly of the same type of importantId. I haven't been able to reproduce that. BTW if I could understand how to manage terms in this case in order to have the expected result it would be great.

thanks
   Luca



On Friday, April 25, 2014 4:37:00 PM UTC+2, Luca wrote:
Hi Simon, thank you for your answer:

the values of importantId field are a finite set of strings. Those values are 100% identical to the ones required in the query. They are not analyzed because, correct me if I'm wrong, <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-terms-filter.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fquery-dsl-terms-filter.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFV-34jiB7ukB0slKhYyr4te0aL1g';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fquery-dsl-terms-filter.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFV-34jiB7ukB0slKhYyr4te0aL1g';return true;">here says that the filter works on not analyzed fields.

Just to be clear, suppose that importantId can assume these values: ['apple', 'pear', 'orange', 'apricot', 'banana']
I want the filter to be able to eliminate all the elements that are not ['apple', 'banana'].

About the results I'm building an example, and soon as it will be running I'll send it here, hopefully by the end of the day.

thanks again!
   Luca



On Thu, Apr 24, 2014 at 6:42 PM, simonw <[hidden email]> wrote:
luca, what do the values in 'importantId` look like? Are they identical to the values you pass in? The field is not analyzed though. It would also be helpful if you could show the results that you would expect so we can see what values they have.

simon


On Thursday, April 24, 2014 5:53:42 PM UTC+2, Luca wrote:
Of course the map is wrong. here's the right one

ES map:
{
   
"mapName": {
     
"_all" : {"enabled" : false},
     
"properties": {
       
"uselessField1": {
           
"type": "boolean",
           
"index": "no"
       
},
       
"uselessField2": {
           
"type": "long",
           
"index": "no"
       
},
       
...
       
"toBeSearched1": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"toBeSearched2": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"importantId": {
           
"type": "string",
           
"index": "not_analyzed"
       
},
       
"importantField": {
           
"type": "string",
           
"index": "not_analyzed"
       
}
     
}
   
}
}


--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit <a href="https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe" target="_blank" onmousedown="this.href='https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe';return true;" onclick="this.href='https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe';return true;">https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f719d9a1-3592-4ae2-bed7-54b907a3af16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: inexplicable behavior using 'terms' filter

simonw-3
as my colleague Luca Cavanna said there is something wrong with your mapping... try this: https://gist.github.com/s1monw/11299421

this should work with your queries...

On Friday, April 25, 2014 6:33:15 PM UTC+2, Luca Cavanna wrote:
Hi Luca,
I think you should have a look at the analyze and check how your different fields end up in the index. If the second query doesn't return any result it means that the filters you added there don't match any document.

What I also noticed is that in your mapping you refer to importantId and so on, but they are under the importantInfo object in your documents. Have a look at your generated mapping too using the get mapping api and see if what you get back is what you submitted or maybe it contains additional fields with default text analysis.

Cheers
Luca

On Friday, April 25, 2014 5:39:22 PM UTC+2, Luca wrote:

Attached you can find a script that basically create an index, a map and add 3 elements to that.

the following queries returns respectively 1 (correctly) and 0 results

{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "termA"
               
}
           
},
           
"filter": {
               
"and": [
                   
{
                       
"term": {
                           
"importantInfo.fieldNotInMap": true
                       
}
                   
}
               
]
           
}
       
}
   
}
}
{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "termA"
               
}
           
},
           
"filter": {
               
"and": [
                   
{
                       
"term": {
                           
"importantInfo.fieldNotInMap": true
                       
}
                   
},
                   
{
                       
"term" : {
                           
"importantInfo.importantField": "typeA"
                       
}
                   
},
                   
{
                       
"terms" : {
                           
"importantInfo.importantId": [ "stringId2", "stringId1" ]
                       
}
                   
}
               
]
           
}
       
}
   
}
}

As I told you before this is not 100% my problem. On my system I can actually filter using terms on another field (importantField in the above example) that is exactly of the same type of importantId. I haven't been able to reproduce that. BTW if I could understand how to manage terms in this case in order to have the expected result it would be great.

thanks
   Luca



On Friday, April 25, 2014 4:37:00 PM UTC+2, Luca wrote:
Hi Simon, thank you for your answer:

the values of importantId field are a finite set of strings. Those values are 100% identical to the ones required in the query. They are not analyzed because, correct me if I'm wrong, <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-terms-filter.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fquery-dsl-terms-filter.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFV-34jiB7ukB0slKhYyr4te0aL1g';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fquery-dsl-terms-filter.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFV-34jiB7ukB0slKhYyr4te0aL1g';return true;">here says that the filter works on not analyzed fields.

Just to be clear, suppose that importantId can assume these values: ['apple', 'pear', 'orange', 'apricot', 'banana']
I want the filter to be able to eliminate all the elements that are not ['apple', 'banana'].

About the results I'm building an example, and soon as it will be running I'll send it here, hopefully by the end of the day.

thanks again!
   Luca



On Thu, Apr 24, 2014 at 6:42 PM, simonw <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="hJifLC26kkUJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">simon.w...@elasticsearch.com> wrote:
luca, what do the values in 'importantId` look like? Are they identical to the values you pass in? The field is not analyzed though. It would also be helpful if you could show the results that you would expect so we can see what values they have.

simon


On Thursday, April 24, 2014 5:53:42 PM UTC+2, Luca wrote:
Of course the map is wrong. here's the right one

ES map:
{
   
"mapName": {
     
"_all" : {"enabled" : false},
     
"properties": {
       
"uselessField1": {
           
"type": "boolean",
           
"index": "no"
       
},
       
"uselessField2": {
           
"type": "long",
           
"index": "no"
       
},
       
...
       
"toBeSearched1": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"toBeSearched2": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"importantId": {
           
"type": "string",
           
"index": "not_analyzed"
       
},
       
"importantField": {
           
"type": "string",
           
"index": "not_analyzed"
       
}
     
}
   
}
}


--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit <a href="https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe" target="_blank" onmousedown="this.href='https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe';return true;" onclick="this.href='https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe';return true;">https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="hJifLC26kkUJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/e77ccaee-de1e-4e47-8729-38a26eed1a4c%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4d710143-89d4-4fc8-9b2e-0d6bf28bf2a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: inexplicable behavior using 'terms' filter

Luca Puccini
Hello Luca and Simon,

thank you very much for your answer, after a few tests everything seems to work fine now. I tried that solution by myself but I must have made some mistakes back then. By the way, is there a reason why the query works if you just filter using "term" and does not work if you use "terms"? Shouldn't simply don't work in both cases, due to the fact that the mapping is wrong?

Luca


--

Luca Puccini
dev @ mixd.tv

Skype: lucap283



On Fri, Apr 25, 2014 at 8:52 PM, simonw <[hidden email]> wrote:
as my colleague Luca Cavanna said there is something wrong with your mapping... try this: https://gist.github.com/s1monw/11299421

this should work with your queries...


On Friday, April 25, 2014 6:33:15 PM UTC+2, Luca Cavanna wrote:
Hi Luca,
I think you should have a look at the analyze and check how your different fields end up in the index. If the second query doesn't return any result it means that the filters you added there don't match any document.

What I also noticed is that in your mapping you refer to importantId and so on, but they are under the importantInfo object in your documents. Have a look at your generated mapping too using the get mapping api and see if what you get back is what you submitted or maybe it contains additional fields with default text analysis.

Cheers
Luca

On Friday, April 25, 2014 5:39:22 PM UTC+2, Luca wrote:

Attached you can find a script that basically create an index, a map and add 3 elements to that.

the following queries returns respectively 1 (correctly) and 0 results

{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "termA"
               
}
           
},
           
"filter": {
               
"and": [
                   
{
                       
"term": {
                           
"importantInfo.fieldNotInMap": true
                       
}
                   
}
               
]
           
}
       
}
   
}
}
{
   
"query": {
       
"filtered": {
           
"query": {
               
"multi_match": {
                   
"fields": [
                       
"importantInfo.toBeSearched1^2",
                       
"importantInfo.toBeSearched2"
                   
],
                   
"query": "termA"
               
}
           
},
           
"filter": {
               
"and": [
                   
{
                       
"term": {
                           
"importantInfo.fieldNotInMap": true
                       
}
                   
},
                   
{
                       
"term" : {
                           
"importantInfo.importantField": "typeA"
                       
}
                   
},
                   
{
                       
"terms" : {
                           
"importantInfo.importantId": [ "stringId2", "stringId1" ]
                       
}
                   
}
               
]
           
}
       
}
   
}
}

As I told you before this is not 100% my problem. On my system I can actually filter using terms on another field (importantField in the above example) that is exactly of the same type of importantId. I haven't been able to reproduce that. BTW if I could understand how to manage terms in this case in order to have the expected result it would be great.

thanks
   Luca



On Friday, April 25, 2014 4:37:00 PM UTC+2, Luca wrote:
Hi Simon, thank you for your answer:

the values of importantId field are a finite set of strings. Those values are 100% identical to the ones required in the query. They are not analyzed because, correct me if I'm wrong, here says that the filter works on not analyzed fields.

Just to be clear, suppose that importantId can assume these values: ['apple', 'pear', 'orange', 'apricot', 'banana']
I want the filter to be able to eliminate all the elements that are not ['apple', 'banana'].

About the results I'm building an example, and soon as it will be running I'll send it here, hopefully by the end of the day.

thanks again!
   Luca



On Thu, Apr 24, 2014 at 6:42 PM, simonw <simon.w...@elasticsearch.com> wrote:
luca, what do the values in 'importantId` look like? Are they identical to the values you pass in? The field is not analyzed though. It would also be helpful if you could show the results that you would expect so we can see what values they have.

simon


On Thursday, April 24, 2014 5:53:42 PM UTC+2, Luca wrote:
Of course the map is wrong. here's the right one

ES map:
{
   
"mapName": {
     
"_all" : {"enabled" : false},
     
"properties": {
       
"uselessField1": {
           
"type": "boolean",
           
"index": "no"
       
},
       
"uselessField2": {
           
"type": "long",
           
"index": "no"
       
},
       
...
       
"toBeSearched1": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"toBeSearched2": {
           
"type": "string",
           
"index": "de_analyzer"
       
},
       
"importantId": {
           
"type": "string",
           
"index": "not_analyzed"
       
},
       
"importantField": {
           
"type": "string",
           
"index": "not_analyzed"
       
}
     
}
   
}
}


--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearc...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/_7lP_AJrk-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4d710143-89d4-4fc8-9b2e-0d6bf28bf2a7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2B0cMO%2BKjdtW9V6yds4pDurRn6mscZZfXM7Tq-4iqQPVKOqV7g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.