Quantcast

Sorting on a custom script field in Java

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Sorting on a custom script field in Java

echin1999-2

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

SampleCodeAndOutput.rtf (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

Igor Motov-3
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

echin1999-2
Hi Igor,

thanks for the tip..  I added the following code to my test program (essentially wrapping the filter i already had with a custom fiilter score):

qb = QueryBuilders.customFiltersScoreQuery(qb).add(FilterBuilders.termFilter("ln", someValue_2), 100);


and added a sort on the _score, and it seems to work. (boost the score to 100 if field "ln" contained someValue_2, otherwise not).   Is that the correct usage, or is there a better way that doesn't require wrapping the existing filter?   

Another thing i noticed is that when I iterated through the hits, and printed the score, I would get a NaN for the score if I added a secondary sort against another field.  If just a single sort by score, then the score printed fine.   Is that expected behavior?

Thanks
Ed


On Wednesday, February 27, 2013 10:11:52 AM UTC-5, Igor Motov wrote:
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

Igor Motov-3
Yes, that's correct usage. Could you provide a complete example of the issue with NaN?

On Wednesday, February 27, 2013 6:13:00 PM UTC-5, echin1999 wrote:
Hi Igor,

thanks for the tip..  I added the following code to my test program (essentially wrapping the filter i already had with a custom fiilter score):

qb = QueryBuilders.customFiltersScoreQuery(qb).add(FilterBuilders.termFilter("ln", someValue_2), 100);


and added a sort on the _score, and it seems to work. (boost the score to 100 if field "ln" contained someValue_2, otherwise not).   Is that the correct usage, or is there a better way that doesn't require wrapping the existing filter?   

Another thing i noticed is that when I iterated through the hits, and printed the score, I would get a NaN for the score if I added a secondary sort against another field.  If just a single sort by score, then the score printed fine.   Is that expected behavior?

Thanks
Ed


On Wednesday, February 27, 2013 10:11:52 AM UTC-5, Igor Motov wrote:
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

echin1999-2

Here's a downsized version of the code (see attachment on original post for complete version except that attachment only has one sort):


qb = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),FilterBuilders.termFilter("cmp", "1"));

        

SearchRequestBuilder srb = client.prepareSearch().setQuery(qb);


I create two sortBuilders...


FieldSortBuilder sortBuilder = new FieldSortBuilder("_score");

sortBuilder.ignoreUnmapped(true);

sortBuilder.order(SortOrder.DESC);

FieldSortBuilder sortBuilder2 = new FieldSortBuilder("ln");

sortBuilder2.ignoreUnmapped(true);

sortBuilder2.order(SortOrder.ASC);


If I only use the first sortBuilder, i get no issues… (score = 1)

but if i use both (uncomment the second line here:)


srb.addSort(sortBuilder);

// srb.addSort(sortBuilder2);


// execute 

SearchResponse response = srb.execute().actionGet();

Then I get NaN for the hit.getScore() - please note, however, the sorting appears to be fine - hard to tell for sure though.

// print response

SearchHits hits = response.getHits();


for(SearchHit hit: hits)

{

    System.out.println(hit.getScore()+":"+hit.getSourceAsString());

}



The document looks like:


{"tgl":["pension","fUNd","funkychickeEN"],"ln":"Chin","gui":[5,6],"d":"my description at Sat Feb 16 18:11:20 EST 2013","dur":180,"nui":[5],"n":"MyPublishTest","pdt":1361056280000,"cmp":1,"fls":{"mvd":1361056280761,"pvd":1361056280762,"ort":3,"ori":1361056280763},"own":5,"by":"Chin Ed","cid":28,"fn":"Ed"}


Thanks

Ed



On Wednesday, February 27, 2013 8:15:04 PM UTC-5, Igor Motov wrote:
Yes, that's correct usage. Could you provide a complete example of the issue with NaN?

On Wednesday, February 27, 2013 6:13:00 PM UTC-5, echin1999 wrote:
Hi Igor,

thanks for the tip..  I added the following code to my test program (essentially wrapping the filter i already had with a custom fiilter score):

qb = QueryBuilders.customFiltersScoreQuery(qb).add(FilterBuilders.termFilter("ln", someValue_2), 100);


and added a sort on the _score, and it seems to work. (boost the score to 100 if field "ln" contained someValue_2, otherwise not).   Is that the correct usage, or is there a better way that doesn't require wrapping the existing filter?   

Another thing i noticed is that when I iterated through the hits, and printed the score, I would get a NaN for the score if I added a secondary sort against another field.  If just a single sort by score, then the score printed fine.   Is that expected behavior?

Thanks
Ed


On Wednesday, February 27, 2013 10:11:52 AM UTC-5, Igor Motov wrote:
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

Igor Motov-3
I cannot reproduce it for some reason. Here's what I tried:

    @Test
    public void searchSort() {
        Node node = startNode("node1");

        try {
            node.client().admin().indices().prepareDelete("test").execute().actionGet();
        } catch (Exception e) {
            // ignore
        }
        node.client().admin().indices().prepareCreate("test")
                .setSettings(settingsBuilder().put("index.number_of_shards", 1)).execute().actionGet();


        for (int i = 0; i < 20; i++) {
            node.client().prepareIndex("test", "type1", Integer.toString(i))
                    .setSource("field", "test " + i, "num", i).execute().actionGet();

        }
        node.client().admin().indices().prepareRefresh().execute().actionGet();

        SearchResponse searchResponse = node.client().prepareSearch("test")
                .setQuery(QueryBuilders.matchQuery("field", "test 10"))
                .addSort(SortBuilders.fieldSort("_score").ignoreUnmapped(true).order(SortOrder.DESC))
                .addSort(SortBuilders.fieldSort("num").ignoreUnmapped(true).order(SortOrder.ASC))
                .execute().actionGet();
        assertThat(searchResponse.getHits().totalHits(), equalTo(20l));

        assertThat(searchResponse.getHits().hits()[0].getScore(), greaterThan(0.0f));

    }

Could you modify this test in such a way that it reproduces the issue? Which version of elasticsearch are you using?


On Wednesday, February 27, 2013 9:59:48 PM UTC-5, echin1999 wrote:

Here's a downsized version of the code (see attachment on original post for complete version except that attachment only has one sort):


qb = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),FilterBuilders.termFilter("cmp", "1"));

        

SearchRequestBuilder srb = client.prepareSearch().setQuery(qb);


I create two sortBuilders...


FieldSortBuilder sortBuilder = new FieldSortBuilder("_score");

sortBuilder.ignoreUnmapped(true);

sortBuilder.order(SortOrder.DESC);

FieldSortBuilder sortBuilder2 = new FieldSortBuilder("ln");

sortBuilder2.ignoreUnmapped(true);

sortBuilder2.order(SortOrder.ASC);


If I only use the first sortBuilder, i get no issues… (score = 1)

but if i use both (uncomment the second line here:)


srb.addSort(sortBuilder);

// srb.addSort(sortBuilder2);


// execute 

SearchResponse response = srb.execute().actionGet();

Then I get NaN for the hit.getScore() - please note, however, the sorting appears to be fine - hard to tell for sure though.

// print response

SearchHits hits = response.getHits();


for(SearchHit hit: hits)

{

    System.out.println(hit.getScore()+":"+hit.getSourceAsString());

}



The document looks like:


{"tgl":["pension","fUNd","funkychickeEN"],"ln":"Chin","gui":[5,6],"d":"my description at Sat Feb 16 18:11:20 EST 2013","dur":180,"nui":[5],"n":"MyPublishTest","pdt":1361056280000,"cmp":1,"fls":{"mvd":1361056280761,"pvd":1361056280762,"ort":3,"ori":1361056280763},"own":5,"by":"Chin Ed","cid":28,"fn":"Ed"}


Thanks

Ed



On Wednesday, February 27, 2013 8:15:04 PM UTC-5, Igor Motov wrote:
Yes, that's correct usage. Could you provide a complete example of the issue with NaN?

On Wednesday, February 27, 2013 6:13:00 PM UTC-5, echin1999 wrote:
Hi Igor,

thanks for the tip..  I added the following code to my test program (essentially wrapping the filter i already had with a custom fiilter score):

qb = QueryBuilders.customFiltersScoreQuery(qb).add(FilterBuilders.termFilter("ln", someValue_2), 100);


and added a sort on the _score, and it seems to work. (boost the score to 100 if field "ln" contained someValue_2, otherwise not).   Is that the correct usage, or is there a better way that doesn't require wrapping the existing filter?   

Another thing i noticed is that when I iterated through the hits, and printed the score, I would get a NaN for the score if I added a secondary sort against another field.  If just a single sort by score, then the score printed fine.   Is that expected behavior?

Thanks
Ed


On Wednesday, February 27, 2013 10:11:52 AM UTC-5, Igor Motov wrote:
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

echin1999-2
I'm currently using Version: 0.20.2.    Wow. I see there is 0.20.5 available  - i will upgrade to that to see if there's a difference.

as for trying to reproduce it using your test code, could you provide me a pointer to the rest of the code that has some of the supporting convenience methods (ie. startNode, etc).

thanks

On Thursday, February 28, 2013 9:44:25 AM UTC-5, Igor Motov wrote:
I cannot reproduce it for some reason. Here's what I tried:

    @Test
    public void searchSort() {
        Node node = startNode("node1");

        try {
            node.client().admin().indices().prepareDelete("test").execute().actionGet();
        } catch (Exception e) {
            // ignore
        }
        node.client().admin().indices().prepareCreate("test")
                .setSettings(settingsBuilder().put("index.number_of_shards", 1)).execute().actionGet();


        for (int i = 0; i < 20; i++) {
            node.client().prepareIndex("test", "type1", Integer.toString(i))
                    .setSource("field", "test " + i, "num", i).execute().actionGet();

        }
        node.client().admin().indices().prepareRefresh().execute().actionGet();

        SearchResponse searchResponse = node.client().prepareSearch("test")
                .setQuery(QueryBuilders.matchQuery("field", "test 10"))
                .addSort(SortBuilders.fieldSort("_score").ignoreUnmapped(true).order(SortOrder.DESC))
                .addSort(SortBuilders.fieldSort("num").ignoreUnmapped(true).order(SortOrder.ASC))
                .execute().actionGet();
        assertThat(searchResponse.getHits().totalHits(), equalTo(20l));

        assertThat(searchResponse.getHits().hits()[0].getScore(), greaterThan(0.0f));

    }

Could you modify this test in such a way that it reproduces the issue? Which version of elasticsearch are you using?


On Wednesday, February 27, 2013 9:59:48 PM UTC-5, echin1999 wrote:

Here's a downsized version of the code (see attachment on original post for complete version except that attachment only has one sort):


qb = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),FilterBuilders.termFilter("cmp", "1"));

        

SearchRequestBuilder srb = client.prepareSearch().setQuery(qb);


I create two sortBuilders...


FieldSortBuilder sortBuilder = new FieldSortBuilder("_score");

sortBuilder.ignoreUnmapped(true);

sortBuilder.order(SortOrder.DESC);

FieldSortBuilder sortBuilder2 = new FieldSortBuilder("ln");

sortBuilder2.ignoreUnmapped(true);

sortBuilder2.order(SortOrder.ASC);


If I only use the first sortBuilder, i get no issues… (score = 1)

but if i use both (uncomment the second line here:)


srb.addSort(sortBuilder);

// srb.addSort(sortBuilder2);


// execute 

SearchResponse response = srb.execute().actionGet();

Then I get NaN for the hit.getScore() - please note, however, the sorting appears to be fine - hard to tell for sure though.

// print response

SearchHits hits = response.getHits();


for(SearchHit hit: hits)

{

    System.out.println(hit.getScore()+":"+hit.getSourceAsString());

}



The document looks like:


{"tgl":["pension","fUNd","funkychickeEN"],"ln":"Chin","gui":[5,6],"d":"my description at Sat Feb 16 18:11:20 EST 2013","dur":180,"nui":[5],"n":"MyPublishTest","pdt":1361056280000,"cmp":1,"fls":{"mvd":1361056280761,"pvd":1361056280762,"ort":3,"ori":1361056280763},"own":5,"by":"Chin Ed","cid":28,"fn":"Ed"}


Thanks

Ed



On Wednesday, February 27, 2013 8:15:04 PM UTC-5, Igor Motov wrote:
Yes, that's correct usage. Could you provide a complete example of the issue with NaN?

On Wednesday, February 27, 2013 6:13:00 PM UTC-5, echin1999 wrote:
Hi Igor,

thanks for the tip..  I added the following code to my test program (essentially wrapping the filter i already had with a custom fiilter score):

qb = QueryBuilders.customFiltersScoreQuery(qb).add(FilterBuilders.termFilter("ln", someValue_2), 100);


and added a sort on the _score, and it seems to work. (boost the score to 100 if field "ln" contained someValue_2, otherwise not).   Is that the correct usage, or is there a better way that doesn't require wrapping the existing filter?   

Another thing i noticed is that when I iterated through the hits, and printed the score, I would get a NaN for the score if I added a secondary sort against another field.  If just a single sort by score, then the score printed fine.   Is that expected behavior?

Thanks
Ed


On Wednesday, February 27, 2013 10:11:52 AM UTC-5, Igor Motov wrote:
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

echin1999-2
In reply to this post by Igor Motov-3
so.. upgrading to 0.20.5 didn't seem to help. However I've observed an unusual pattern.

I am running a single elasticsearch process locally.   If I restart the elastic search process (kill and start), the problem goes away.  If i restart again, the problem comes back.  This   alternating behavior repeats for each restart.   It makes no difference if i run in the foreground (-f) or not.  Also, if i run in the foreground, i don't see any errors. 

Are there specific debug settings to tweak to perhaps get more information as to what is going on?  
   

On Thursday, February 28, 2013 9:44:25 AM UTC-5, Igor Motov wrote:
I cannot reproduce it for some reason. Here's what I tried:

    @Test
    public void searchSort() {
        Node node = startNode("node1");

        try {
            node.client().admin().indices().prepareDelete("test").execute().actionGet();
        } catch (Exception e) {
            // ignore
        }
        node.client().admin().indices().prepareCreate("test")
                .setSettings(settingsBuilder().put("index.number_of_shards", 1)).execute().actionGet();


        for (int i = 0; i < 20; i++) {
            node.client().prepareIndex("test", "type1", Integer.toString(i))
                    .setSource("field", "test " + i, "num", i).execute().actionGet();

        }
        node.client().admin().indices().prepareRefresh().execute().actionGet();

        SearchResponse searchResponse = node.client().prepareSearch("test")
                .setQuery(QueryBuilders.matchQuery("field", "test 10"))
                .addSort(SortBuilders.fieldSort("_score").ignoreUnmapped(true).order(SortOrder.DESC))
                .addSort(SortBuilders.fieldSort("num").ignoreUnmapped(true).order(SortOrder.ASC))
                .execute().actionGet();
        assertThat(searchResponse.getHits().totalHits(), equalTo(20l));

        assertThat(searchResponse.getHits().hits()[0].getScore(), greaterThan(0.0f));

    }

Could you modify this test in such a way that it reproduces the issue? Which version of elasticsearch are you using?


On Wednesday, February 27, 2013 9:59:48 PM UTC-5, echin1999 wrote:

Here's a downsized version of the code (see attachment on original post for complete version except that attachment only has one sort):


qb = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),FilterBuilders.termFilter("cmp", "1"));

        

SearchRequestBuilder srb = client.prepareSearch().setQuery(qb);


I create two sortBuilders...


FieldSortBuilder sortBuilder = new FieldSortBuilder("_score");

sortBuilder.ignoreUnmapped(true);

sortBuilder.order(SortOrder.DESC);

FieldSortBuilder sortBuilder2 = new FieldSortBuilder("ln");

sortBuilder2.ignoreUnmapped(true);

sortBuilder2.order(SortOrder.ASC);


If I only use the first sortBuilder, i get no issues… (score = 1)

but if i use both (uncomment the second line here:)


srb.addSort(sortBuilder);

// srb.addSort(sortBuilder2);


// execute 

SearchResponse response = srb.execute().actionGet();

Then I get NaN for the hit.getScore() - please note, however, the sorting appears to be fine - hard to tell for sure though.

// print response

SearchHits hits = response.getHits();


for(SearchHit hit: hits)

{

    System.out.println(hit.getScore()+":"+hit.getSourceAsString());

}



The document looks like:


{"tgl":["pension","fUNd","funkychickeEN"],"ln":"Chin","gui":[5,6],"d":"my description at Sat Feb 16 18:11:20 EST 2013","dur":180,"nui":[5],"n":"MyPublishTest","pdt":1361056280000,"cmp":1,"fls":{"mvd":1361056280761,"pvd":1361056280762,"ort":3,"ori":1361056280763},"own":5,"by":"Chin Ed","cid":28,"fn":"Ed"}


Thanks

Ed



On Wednesday, February 27, 2013 8:15:04 PM UTC-5, Igor Motov wrote:
Yes, that's correct usage. Could you provide a complete example of the issue with NaN?

On Wednesday, February 27, 2013 6:13:00 PM UTC-5, echin1999 wrote:
Hi Igor,

thanks for the tip..  I added the following code to my test program (essentially wrapping the filter i already had with a custom fiilter score):

qb = QueryBuilders.customFiltersScoreQuery(qb).add(FilterBuilders.termFilter("ln", someValue_2), 100);


and added a sort on the _score, and it seems to work. (boost the score to 100 if field "ln" contained someValue_2, otherwise not).   Is that the correct usage, or is there a better way that doesn't require wrapping the existing filter?   

Another thing i noticed is that when I iterated through the hits, and printed the score, I would get a NaN for the score if I added a secondary sort against another field.  If just a single sort by score, then the score printed fine.   Is that expected behavior?

Thanks
Ed


On Wednesday, February 27, 2013 10:11:52 AM UTC-5, Igor Motov wrote:
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

Igor Motov-3
You can include the the test that I posted class that I sent you into a class inherited from AbstractNodesTests. Can you enable TRACE logging on your elasticsearch server, run your test and post the log somewhere? 

On Thursday, February 28, 2013 11:05:57 AM UTC-5, echin1999 wrote:
so.. upgrading to 0.20.5 didn't seem to help. However I've observed an unusual pattern.

I am running a single elasticsearch process locally.   If I restart the elastic search process (kill and start), the problem goes away.  If i restart again, the problem comes back.  This   alternating behavior repeats for each restart.   It makes no difference if i run in the foreground (-f) or not.  Also, if i run in the foreground, i don't see any errors. 

Are there specific debug settings to tweak to perhaps get more information as to what is going on?  
   

On Thursday, February 28, 2013 9:44:25 AM UTC-5, Igor Motov wrote:
I cannot reproduce it for some reason. Here's what I tried:

    @Test
    public void searchSort() {
        Node node = startNode("node1");

        try {
            node.client().admin().indices().prepareDelete("test").execute().actionGet();
        } catch (Exception e) {
            // ignore
        }
        node.client().admin().indices().prepareCreate("test")
                .setSettings(settingsBuilder().put("index.number_of_shards", 1)).execute().actionGet();


        for (int i = 0; i < 20; i++) {
            node.client().prepareIndex("test", "type1", Integer.toString(i))
                    .setSource("field", "test " + i, "num", i).execute().actionGet();

        }
        node.client().admin().indices().prepareRefresh().execute().actionGet();

        SearchResponse searchResponse = node.client().prepareSearch("test")
                .setQuery(QueryBuilders.matchQuery("field", "test 10"))
                .addSort(SortBuilders.fieldSort("_score").ignoreUnmapped(true).order(SortOrder.DESC))
                .addSort(SortBuilders.fieldSort("num").ignoreUnmapped(true).order(SortOrder.ASC))
                .execute().actionGet();
        assertThat(searchResponse.getHits().totalHits(), equalTo(20l));

        assertThat(searchResponse.getHits().hits()[0].getScore(), greaterThan(0.0f));

    }

Could you modify this test in such a way that it reproduces the issue? Which version of elasticsearch are you using?


On Wednesday, February 27, 2013 9:59:48 PM UTC-5, echin1999 wrote:

Here's a downsized version of the code (see attachment on original post for complete version except that attachment only has one sort):


qb = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),FilterBuilders.termFilter("cmp", "1"));

        

SearchRequestBuilder srb = client.prepareSearch().setQuery(qb);


I create two sortBuilders...


FieldSortBuilder sortBuilder = new FieldSortBuilder("_score");

sortBuilder.ignoreUnmapped(true);

sortBuilder.order(SortOrder.DESC);

FieldSortBuilder sortBuilder2 = new FieldSortBuilder("ln");

sortBuilder2.ignoreUnmapped(true);

sortBuilder2.order(SortOrder.ASC);


If I only use the first sortBuilder, i get no issues… (score = 1)

but if i use both (uncomment the second line here:)


srb.addSort(sortBuilder);

// srb.addSort(sortBuilder2);


// execute 

SearchResponse response = srb.execute().actionGet();

Then I get NaN for the hit.getScore() - please note, however, the sorting appears to be fine - hard to tell for sure though.

// print response

SearchHits hits = response.getHits();


for(SearchHit hit: hits)

{

    System.out.println(hit.getScore()+":"+hit.getSourceAsString());

}



The document looks like:


{"tgl":["pension","fUNd","funkychickeEN"],"ln":"Chin","gui":[5,6],"d":"my description at Sat Feb 16 18:11:20 EST 2013","dur":180,"nui":[5],"n":"MyPublishTest","pdt":1361056280000,"cmp":1,"fls":{"mvd":1361056280761,"pvd":1361056280762,"ort":3,"ori":1361056280763},"own":5,"by":"Chin Ed","cid":28,"fn":"Ed"}


Thanks

Ed



On Wednesday, February 27, 2013 8:15:04 PM UTC-5, Igor Motov wrote:
Yes, that's correct usage. Could you provide a complete example of the issue with NaN?

On Wednesday, February 27, 2013 6:13:00 PM UTC-5, echin1999 wrote:
Hi Igor,

thanks for the tip..  I added the following code to my test program (essentially wrapping the filter i already had with a custom fiilter score):

qb = QueryBuilders.customFiltersScoreQuery(qb).add(FilterBuilders.termFilter("ln", someValue_2), 100);


and added a sort on the _score, and it seems to work. (boost the score to 100 if field "ln" contained someValue_2, otherwise not).   Is that the correct usage, or is there a better way that doesn't require wrapping the existing filter?   

Another thing i noticed is that when I iterated through the hits, and printed the score, I would get a NaN for the score if I added a secondary sort against another field.  If just a single sort by score, then the score printed fine.   Is that expected behavior?

Thanks
Ed


On Wednesday, February 27, 2013 10:11:52 AM UTC-5, Igor Motov wrote:
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

echin1999-2
For the server, how do i set the log level? I see in the elasticsearch.yml file settings for the slowlog and gc, but nothing for generic logging.

Also, is AbstractNodesTests a class that comes with the release distribution?  I don't see it in any of the jars.  Or do i need to go to github and download the src?

thanks.


On Thursday, February 28, 2013 4:06:49 PM UTC-5, Igor Motov wrote:
You can include the the test that I posted class that I sent you into a class inherited from AbstractNodesTests. Can you enable TRACE logging on your elasticsearch server, run your test and post the log somewhere? 

On Thursday, February 28, 2013 11:05:57 AM UTC-5, echin1999 wrote:
so.. upgrading to 0.20.5 didn't seem to help. However I've observed an unusual pattern.

I am running a single elasticsearch process locally.   If I restart the elastic search process (kill and start), the problem goes away.  If i restart again, the problem comes back.  This   alternating behavior repeats for each restart.   It makes no difference if i run in the foreground (-f) or not.  Also, if i run in the foreground, i don't see any errors. 

Are there specific debug settings to tweak to perhaps get more information as to what is going on?  
   

On Thursday, February 28, 2013 9:44:25 AM UTC-5, Igor Motov wrote:
I cannot reproduce it for some reason. Here's what I tried:

    @Test
    public void searchSort() {
        Node node = startNode("node1");

        try {
            node.client().admin().indices().prepareDelete("test").execute().actionGet();
        } catch (Exception e) {
            // ignore
        }
        node.client().admin().indices().prepareCreate("test")
                .setSettings(settingsBuilder().put("index.number_of_shards", 1)).execute().actionGet();


        for (int i = 0; i < 20; i++) {
            node.client().prepareIndex("test", "type1", Integer.toString(i))
                    .setSource("field", "test " + i, "num", i).execute().actionGet();

        }
        node.client().admin().indices().prepareRefresh().execute().actionGet();

        SearchResponse searchResponse = node.client().prepareSearch("test")
                .setQuery(QueryBuilders.matchQuery("field", "test 10"))
                .addSort(SortBuilders.fieldSort("_score").ignoreUnmapped(true).order(SortOrder.DESC))
                .addSort(SortBuilders.fieldSort("num").ignoreUnmapped(true).order(SortOrder.ASC))
                .execute().actionGet();
        assertThat(searchResponse.getHits().totalHits(), equalTo(20l));

        assertThat(searchResponse.getHits().hits()[0].getScore(), greaterThan(0.0f));

    }

Could you modify this test in such a way that it reproduces the issue? Which version of elasticsearch are you using?


On Wednesday, February 27, 2013 9:59:48 PM UTC-5, echin1999 wrote:

Here's a downsized version of the code (see attachment on original post for complete version except that attachment only has one sort):


qb = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),FilterBuilders.termFilter("cmp", "1"));

        

SearchRequestBuilder srb = client.prepareSearch().setQuery(qb);


I create two sortBuilders...


FieldSortBuilder sortBuilder = new FieldSortBuilder("_score");

sortBuilder.ignoreUnmapped(true);

sortBuilder.order(SortOrder.DESC);

FieldSortBuilder sortBuilder2 = new FieldSortBuilder("ln");

sortBuilder2.ignoreUnmapped(true);

sortBuilder2.order(SortOrder.ASC);


If I only use the first sortBuilder, i get no issues… (score = 1)

but if i use both (uncomment the second line here:)


srb.addSort(sortBuilder);

// srb.addSort(sortBuilder2);


// execute 

SearchResponse response = srb.execute().actionGet();

Then I get NaN for the hit.getScore() - please note, however, the sorting appears to be fine - hard to tell for sure though.

// print response

SearchHits hits = response.getHits();


for(SearchHit hit: hits)

{

    System.out.println(hit.getScore()+":"+hit.getSourceAsString());

}



The document looks like:


{"tgl":["pension","fUNd","funkychickeEN"],"ln":"Chin","gui":[5,6],"d":"my description at Sat Feb 16 18:11:20 EST 2013","dur":180,"nui":[5],"n":"MyPublishTest","pdt":1361056280000,"cmp":1,"fls":{"mvd":1361056280761,"pvd":1361056280762,"ort":3,"ori":1361056280763},"own":5,"by":"Chin Ed","cid":28,"fn":"Ed"}


Thanks

Ed



On Wednesday, February 27, 2013 8:15:04 PM UTC-5, Igor Motov wrote:
Yes, that's correct usage. Could you provide a complete example of the issue with NaN?

On Wednesday, February 27, 2013 6:13:00 PM UTC-5, echin1999 wrote:
Hi Igor,

thanks for the tip..  I added the following code to my test program (essentially wrapping the filter i already had with a custom fiilter score):

qb = QueryBuilders.customFiltersScoreQuery(qb).add(FilterBuilders.termFilter("ln", someValue_2), 100);


and added a sort on the _score, and it seems to work. (boost the score to 100 if field "ln" contained someValue_2, otherwise not).   Is that the correct usage, or is there a better way that doesn't require wrapping the existing filter?   

Another thing i noticed is that when I iterated through the hits, and printed the score, I would get a NaN for the score if I added a secondary sort against another field.  If just a single sort by score, then the score printed fine.   Is that expected behavior?

Thanks
Ed


On Wednesday, February 27, 2013 10:11:52 AM UTC-5, Igor Motov wrote:
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

Igor Motov-3
The log level can be set in logging.yml file which can be found in the same directory as elasticsearch.yml. Just replace INFO in the first line with TRACE. 

The AbstractNodesTests is part of elasticsearch tests, so it's not included in the release distribution. 

On Thursday, February 28, 2013 7:57:09 PM UTC-5, echin1999 wrote:
For the server, how do i set the log level? I see in the elasticsearch.yml file settings for the slowlog and gc, but nothing for generic logging.

Also, is AbstractNodesTests a class that comes with the release distribution?  I don't see it in any of the jars.  Or do i need to go to github and download the src?

thanks.


On Thursday, February 28, 2013 4:06:49 PM UTC-5, Igor Motov wrote:
You can include the the test that I posted class that I sent you into a class inherited from AbstractNodesTests. Can you enable TRACE logging on your elasticsearch server, run your test and post the log somewhere? 

On Thursday, February 28, 2013 11:05:57 AM UTC-5, echin1999 wrote:
so.. upgrading to 0.20.5 didn't seem to help. However I've observed an unusual pattern.

I am running a single elasticsearch process locally.   If I restart the elastic search process (kill and start), the problem goes away.  If i restart again, the problem comes back.  This   alternating behavior repeats for each restart.   It makes no difference if i run in the foreground (-f) or not.  Also, if i run in the foreground, i don't see any errors. 

Are there specific debug settings to tweak to perhaps get more information as to what is going on?  
   

On Thursday, February 28, 2013 9:44:25 AM UTC-5, Igor Motov wrote:
I cannot reproduce it for some reason. Here's what I tried:

    @Test
    public void searchSort() {
        Node node = startNode("node1");

        try {
            node.client().admin().indices().prepareDelete("test").execute().actionGet();
        } catch (Exception e) {
            // ignore
        }
        node.client().admin().indices().prepareCreate("test")
                .setSettings(settingsBuilder().put("index.number_of_shards", 1)).execute().actionGet();


        for (int i = 0; i < 20; i++) {
            node.client().prepareIndex("test", "type1", Integer.toString(i))
                    .setSource("field", "test " + i, "num", i).execute().actionGet();

        }
        node.client().admin().indices().prepareRefresh().execute().actionGet();

        SearchResponse searchResponse = node.client().prepareSearch("test")
                .setQuery(QueryBuilders.matchQuery("field", "test 10"))
                .addSort(SortBuilders.fieldSort("_score").ignoreUnmapped(true).order(SortOrder.DESC))
                .addSort(SortBuilders.fieldSort("num").ignoreUnmapped(true).order(SortOrder.ASC))
                .execute().actionGet();
        assertThat(searchResponse.getHits().totalHits(), equalTo(20l));

        assertThat(searchResponse.getHits().hits()[0].getScore(), greaterThan(0.0f));

    }

Could you modify this test in such a way that it reproduces the issue? Which version of elasticsearch are you using?


On Wednesday, February 27, 2013 9:59:48 PM UTC-5, echin1999 wrote:

Here's a downsized version of the code (see attachment on original post for complete version except that attachment only has one sort):


qb = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),FilterBuilders.termFilter("cmp", "1"));

        

SearchRequestBuilder srb = client.prepareSearch().setQuery(qb);


I create two sortBuilders...


FieldSortBuilder sortBuilder = new FieldSortBuilder("_score");

sortBuilder.ignoreUnmapped(true);

sortBuilder.order(SortOrder.DESC);

FieldSortBuilder sortBuilder2 = new FieldSortBuilder("ln");

sortBuilder2.ignoreUnmapped(true);

sortBuilder2.order(SortOrder.ASC);


If I only use the first sortBuilder, i get no issues… (score = 1)

but if i use both (uncomment the second line here:)


srb.addSort(sortBuilder);

// srb.addSort(sortBuilder2);


// execute 

SearchResponse response = srb.execute().actionGet();

Then I get NaN for the hit.getScore() - please note, however, the sorting appears to be fine - hard to tell for sure though.

// print response

SearchHits hits = response.getHits();


for(SearchHit hit: hits)

{

    System.out.println(hit.getScore()+":"+hit.getSourceAsString());

}



The document looks like:


{"tgl":["pension","fUNd","funkychickeEN"],"ln":"Chin","gui":[5,6],"d":"my description at Sat Feb 16 18:11:20 EST 2013","dur":180,"nui":[5],"n":"MyPublishTest","pdt":1361056280000,"cmp":1,"fls":{"mvd":1361056280761,"pvd":1361056280762,"ort":3,"ori":1361056280763},"own":5,"by":"Chin Ed","cid":28,"fn":"Ed"}


Thanks

Ed



On Wednesday, February 27, 2013 8:15:04 PM UTC-5, Igor Motov wrote:
Yes, that's correct usage. Could you provide a complete example of the issue with NaN?

On Wednesday, February 27, 2013 6:13:00 PM UTC-5, echin1999 wrote:
Hi Igor,

thanks for the tip..  I added the following code to my test program (essentially wrapping the filter i already had with a custom fiilter score):

qb = QueryBuilders.customFiltersScoreQuery(qb).add(FilterBuilders.termFilter("ln", someValue_2), 100);


and added a sort on the _score, and it seems to work. (boost the score to 100 if field "ln" contained someValue_2, otherwise not).   Is that the correct usage, or is there a better way that doesn't require wrapping the existing filter?   

Another thing i noticed is that when I iterated through the hits, and printed the score, I would get a NaN for the score if I added a secondary sort against another field.  If just a single sort by score, then the score printed fine.   Is that expected behavior?

Thanks
Ed


On Wednesday, February 27, 2013 10:11:52 AM UTC-5, Igor Motov wrote:
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sorting on a custom script field in Java

echin1999-2
I prepared two log files: 1) log file when the NaN occurs: http://abbyandally.com/echin1999/log_with_NaN.log
and 2) log file when the program worked fine: http://abbyandally.com/echin1999/log_when_it_works.log

thanks

On Thursday, February 28, 2013 10:32:17 PM UTC-5, Igor Motov wrote:
The log level can be set in logging.yml file which can be found in the same directory as elasticsearch.yml. Just replace INFO in the first line with TRACE. 

The AbstractNodesTests is part of elasticsearch tests, so it's not included in the release distribution. 

On Thursday, February 28, 2013 7:57:09 PM UTC-5, echin1999 wrote:
For the server, how do i set the log level? I see in the elasticsearch.yml file settings for the slowlog and gc, but nothing for generic logging.

Also, is AbstractNodesTests a class that comes with the release distribution?  I don't see it in any of the jars.  Or do i need to go to github and download the src?

thanks.


On Thursday, February 28, 2013 4:06:49 PM UTC-5, Igor Motov wrote:
You can include the the test that I posted class that I sent you into a class inherited from AbstractNodesTests. Can you enable TRACE logging on your elasticsearch server, run your test and post the log somewhere? 

On Thursday, February 28, 2013 11:05:57 AM UTC-5, echin1999 wrote:
so.. upgrading to 0.20.5 didn't seem to help. However I've observed an unusual pattern.

I am running a single elasticsearch process locally.   If I restart the elastic search process (kill and start), the problem goes away.  If i restart again, the problem comes back.  This   alternating behavior repeats for each restart.   It makes no difference if i run in the foreground (-f) or not.  Also, if i run in the foreground, i don't see any errors. 

Are there specific debug settings to tweak to perhaps get more information as to what is going on?  
   

On Thursday, February 28, 2013 9:44:25 AM UTC-5, Igor Motov wrote:
I cannot reproduce it for some reason. Here's what I tried:

    @Test
    public void searchSort() {
        Node node = startNode("node1");

        try {
            node.client().admin().indices().prepareDelete("test").execute().actionGet();
        } catch (Exception e) {
            // ignore
        }
        node.client().admin().indices().prepareCreate("test")
                .setSettings(settingsBuilder().put("index.number_of_shards", 1)).execute().actionGet();


        for (int i = 0; i < 20; i++) {
            node.client().prepareIndex("test", "type1", Integer.toString(i))
                    .setSource("field", "test " + i, "num", i).execute().actionGet();

        }
        node.client().admin().indices().prepareRefresh().execute().actionGet();

        SearchResponse searchResponse = node.client().prepareSearch("test")
                .setQuery(QueryBuilders.matchQuery("field", "test 10"))
                .addSort(SortBuilders.fieldSort("_score").ignoreUnmapped(true).order(SortOrder.DESC))
                .addSort(SortBuilders.fieldSort("num").ignoreUnmapped(true).order(SortOrder.ASC))
                .execute().actionGet();
        assertThat(searchResponse.getHits().totalHits(), equalTo(20l));

        assertThat(searchResponse.getHits().hits()[0].getScore(), greaterThan(0.0f));

    }

Could you modify this test in such a way that it reproduces the issue? Which version of elasticsearch are you using?


On Wednesday, February 27, 2013 9:59:48 PM UTC-5, echin1999 wrote:

Here's a downsized version of the code (see attachment on original post for complete version except that attachment only has one sort):


qb = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),FilterBuilders.termFilter("cmp", "1"));

        

SearchRequestBuilder srb = client.prepareSearch().setQuery(qb);


I create two sortBuilders...


FieldSortBuilder sortBuilder = new FieldSortBuilder("_score");

sortBuilder.ignoreUnmapped(true);

sortBuilder.order(SortOrder.DESC);

FieldSortBuilder sortBuilder2 = new FieldSortBuilder("ln");

sortBuilder2.ignoreUnmapped(true);

sortBuilder2.order(SortOrder.ASC);


If I only use the first sortBuilder, i get no issues… (score = 1)

but if i use both (uncomment the second line here:)


srb.addSort(sortBuilder);

// srb.addSort(sortBuilder2);


// execute 

SearchResponse response = srb.execute().actionGet();

Then I get NaN for the hit.getScore() - please note, however, the sorting appears to be fine - hard to tell for sure though.

// print response

SearchHits hits = response.getHits();


for(SearchHit hit: hits)

{

    System.out.println(hit.getScore()+":"+hit.getSourceAsString());

}



The document looks like:


{"tgl":["pension","fUNd","funkychickeEN"],"ln":"Chin","gui":[5,6],"d":"my description at Sat Feb 16 18:11:20 EST 2013","dur":180,"nui":[5],"n":"MyPublishTest","pdt":1361056280000,"cmp":1,"fls":{"mvd":1361056280761,"pvd":1361056280762,"ort":3,"ori":1361056280763},"own":5,"by":"Chin Ed","cid":28,"fn":"Ed"}


Thanks

Ed



On Wednesday, February 27, 2013 8:15:04 PM UTC-5, Igor Motov wrote:
Yes, that's correct usage. Could you provide a complete example of the issue with NaN?

On Wednesday, February 27, 2013 6:13:00 PM UTC-5, echin1999 wrote:
Hi Igor,

thanks for the tip..  I added the following code to my test program (essentially wrapping the filter i already had with a custom fiilter score):

qb = QueryBuilders.customFiltersScoreQuery(qb).add(FilterBuilders.termFilter("ln", someValue_2), 100);


and added a sort on the _score, and it seems to work. (boost the score to 100 if field "ln" contained someValue_2, otherwise not).   Is that the correct usage, or is there a better way that doesn't require wrapping the existing filter?   

Another thing i noticed is that when I iterated through the hits, and printed the score, I would get a NaN for the score if I added a secondary sort against another field.  If just a single sort by score, then the score printed fine.   Is that expected behavior?

Thanks
Ed


On Wednesday, February 27, 2013 10:11:52 AM UTC-5, Igor Motov wrote:
You cannot sort on a script field, but you can use a script to provide a value to sort on. For example the following script will bring the record with the value 42 in the field my_field to the first position. 

.addSort(SortBuilders
        .scriptSort("doc[\"my_field\"].value == my_val? 1 : 0", "number")
        .order(SortOrder.DESC)
        .setParams(MapBuilder.<String, Object>newMapBuilder().put("my_val", 42).map()))

While you can do it with a script sort, if all you need is to boost records with certain values, you might achieve much better performance using a Custom Filters Score Query and adding _score as one of the sort value. 

On Tuesday, February 26, 2013 5:30:46 PM UTC-5, echin1999 wrote:

Hi,


I was wondering if someone could point me to some sample Java code that specifically shows how to sort on a custom script field.  


I see that the SearchRequestBuilder has a method called: addScriptField(name, script, params); 


but i'm not sure how to define the "script" argument.   I assume the "name" is the reference to the script field that i would pass into the construction of a FieldSortBuilder()? 


I want to be able to sort my results based on whether or not a property (or multiple properties) of type long[] contains a long value specified at search time (ie. documents where the criteria holds true will be returned ahead of those documents where the criteria is false).  



I've attached my sample code which does basic sorting on an existing field - that seems to work fine.   I  now just want to enhance it to be able to sort on a custom script field coded as i describe above.  Any pointers are greatly appreciated!


thanks

Ed

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Loading...