behavior of scrolling search with "from" parameter in newer ES versions

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

behavior of scrolling search with "from" parameter in newer ES versions

Gordon Tillman
Greetings All,

I ran into an interesting issue when upgrading from ES version 1.0.1 to newer versions.  In particular, I tested the following with versions 1.2.4, 1.3.4 and 1.4.2.

summary

When doing a normal scroll search (not one with a search_type=scan), it appears that the "from" parameter is being ignored.


example 1 (no scrolling, result is correct)

In this simplified example, only one document matches the supplied query, so with from=1 no documents are returned.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4\&from=1

example 2 (with scrolling, incorrect result)

In this example, where we specify the same query and from parameters, but also specify a scroll parameter, we incorrectly get the single document returned from the query.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4\&from=1\&scroll=2s | json

notes

  1. Both of the above test cases work correctly in version 1.0.1
  2. In newer versions (where example 2 fails), I noticed that the from value is not present in the data that was returned from the query.

I understand with with a search_type=scan, this would be expected because sorting is disabled.  Also, please note this is a greatly simplified query just for illustration purposes.  I've attached a sample of an actual query at the bottom of this post, and it does contain sorting specifications.

Is this a known issue or is this being done by design in the newer ES versions.  A quick scan through the release notes was unrevealing.

Many thanks for any insight!

--g

sample of full query


{
    "sort": [
        {
            "_type": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }, 
        {
            "name_lower": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }, 
        {
            "dds_key": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }
    ], 
    "query": {
        "filtered": {
            "filter": {
                "and": [
                    {
                        "or": [
                            {
                                "term": {
                                    "parent": "c10ed0583104012f94e11ad0ac36f2aaf"
                                }
                            }
                        ]
                    }, 
                    {
                        "not": {
                            "term": {
                                "vcn": "DeleteMarker"
                            }
                        }
                    }, 
                    {
                        "not": {
                            "exists": {
                                "field": "notfinalized"
                            }
                        }
                    }
                ]
            }, 
            "query": {
                "match_all": {}
            }
        }
    }, 
    "from": 1, 
    "size": 1000
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: behavior of scrolling search with "from" parameter in newer ES versions

Martijn v Groningen
Hi Gordon,

This `from` is kind of ignored for scroll search. I don't remember why that was the case, but it seems to me that scroll search can/should take into account the `from` option during the first scroll search request.

Martijn

On 20 January 2015 at 23:33, Gordon Tillman <[hidden email]> wrote:
Greetings All,

I ran into an interesting issue when upgrading from ES version 1.0.1 to newer versions.  In particular, I tested the following with versions 1.2.4, 1.3.4 and 1.4.2.

summary

When doing a normal scroll search (not one with a search_type=scan), it appears that the "from" parameter is being ignored.


example 1 (no scrolling, result is correct)

In this simplified example, only one document matches the supplied query, so with from=1 no documents are returned.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4\&from=1

example 2 (with scrolling, incorrect result)

In this example, where we specify the same query and from parameters, but also specify a scroll parameter, we incorrectly get the single document returned from the query.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4\&from=1\&scroll=2s | json

notes

  1. Both of the above test cases work correctly in version 1.0.1
  2. In newer versions (where example 2 fails), I noticed that the from value is not present in the data that was returned from the query.

I understand with with a search_type=scan, this would be expected because sorting is disabled.  Also, please note this is a greatly simplified query just for illustration purposes.  I've attached a sample of an actual query at the bottom of this post, and it does contain sorting specifications.

Is this a known issue or is this being done by design in the newer ES versions.  A quick scan through the release notes was unrevealing.

Many thanks for any insight!

--g

sample of full query


{
    "sort": [
        {
            "_type": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }, 
        {
            "name_lower": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }, 
        {
            "dds_key": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }
    ], 
    "query": {
        "filtered": {
            "filter": {
                "and": [
                    {
                        "or": [
                            {
                                "term": {
                                    "parent": "c10ed0583104012f94e11ad0ac36f2aaf"
                                }
                            }
                        ]
                    }, 
                    {
                        "not": {
                            "term": {
                                "vcn": "DeleteMarker"
                            }
                        }
                    }, 
                    {
                        "not": {
                            "exists": {
                                "field": "notfinalized"
                            }
                        }
                    }
                ]
            }, 
            "query": {
                "match_all": {}
            }
        }
    }, 
    "from": 1, 
    "size": 1000
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2BA76Tw7_MvnXLAD_4LrQUHvxasGkEUL%2BQRBbVv_UDU3UAq1vg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: behavior of scrolling search with "from" parameter in newer ES versions

Gordon Tillman
Thanks Martijn,

That is what I have observed.  But is is a regression from ES version 1.0.1 and before. And I can't find anything that even implies that `from` is not supported for a scroll search, except of course when `search_type=scan`.

I would love to get support for that back into the product.  

--g



On Wednesday, January 21, 2015 at 2:57:29 AM UTC-6, Martijn v Groningen wrote:
Hi Gordon,

This `from` is kind of ignored for scroll search. I don't remember why that was the case, but it seems to me that scroll search can/should take into account the `from` option during the first scroll search request.

Martijn

On 20 January 2015 at 23:33, Gordon Tillman <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="7amLM_h2xoAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">gor...@...> wrote:
Greetings All,

I ran into an interesting issue when upgrading from ES version 1.0.1 to newer versions.  In particular, I tested the following with versions 1.2.4, 1.3.4 and 1.4.2.

summary

When doing a normal scroll search (not one with a search_type=scan), it appears that the "from" parameter is being ignored.


example 1 (no scrolling, result is correct)

In this simplified example, only one document matches the supplied query, so with from=1 no documents are returned.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4\&from=1

example 2 (with scrolling, incorrect result)

In this example, where we specify the same query and from parameters, but also specify a scroll parameter, we incorrectly get the single document returned from the query.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4\&from=1\&scroll=2s | json

notes

  1. Both of the above test cases work correctly in version 1.0.1
  2. In newer versions (where example 2 fails), I noticed that the from value is not present in the data that was returned from the query.

I understand with with a search_type=scan, this would be expected because sorting is disabled.  Also, please note this is a greatly simplified query just for illustration purposes.  I've attached a sample of an actual query at the bottom of this post, and it does contain sorting specifications.

Is this a known issue or is this being done by design in the newer ES versions.  A quick scan through the release notes was unrevealing.

Many thanks for any insight!

--g

sample of full query


{
    "sort": [
        {
            "_type": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }, 
        {
            "name_lower": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }, 
        {
            "dds_key": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }
    ], 
    "query": {
        "filtered": {
            "filter": {
                "and": [
                    {
                        "or": [
                            {
                                "term": {
                                    "parent": "c10ed0583104012f94e11ad0ac36f2aaf"
                                }
                            }
                        ]
                    }, 
                    {
                        "not": {
                            "term": {
                                "vcn": "DeleteMarker"
                            }
                        }
                    }, 
                    {
                        "not": {
                            "exists": {
                                "field": "notfinalized"
                            }
                        }
                    }
                ]
            }, 
            "query": {
                "match_all": {}
            }
        }
    }, 
    "from": 1, 
    "size": 1000
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="7amLM_h2xoAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.



--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a4f5abd9-2f1f-4d1d-8168-260b4769d268%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: behavior of scrolling search with "from" parameter in newer ES versions

Martijn v Groningen

On 21 January 2015 at 14:27, Gordon Tillman <[hidden email]> wrote:
Thanks Martijn,

That is what I have observed.  But is is a regression from ES version 1.0.1 and before. And I can't find anything that even implies that `from` is not supported for a scroll search, except of course when `search_type=scan`.

I would love to get support for that back into the product.  

--g



On Wednesday, January 21, 2015 at 2:57:29 AM UTC-6, Martijn v Groningen wrote:
Hi Gordon,

This `from` is kind of ignored for scroll search. I don't remember why that was the case, but it seems to me that scroll search can/should take into account the `from` option during the first scroll search request.

Martijn

On 20 January 2015 at 23:33, Gordon Tillman <[hidden email]> wrote:
Greetings All,

I ran into an interesting issue when upgrading from ES version 1.0.1 to newer versions.  In particular, I tested the following with versions 1.2.4, 1.3.4 and 1.4.2.

summary

When doing a normal scroll search (not one with a search_type=scan), it appears that the "from" parameter is being ignored.


example 1 (no scrolling, result is correct)

In this simplified example, only one document matches the supplied query, so with from=1 no documents are returned.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4\&from=1

example 2 (with scrolling, incorrect result)

In this example, where we specify the same query and from parameters, but also specify a scroll parameter, we incorrectly get the single document returned from the query.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4\&from=1\&scroll=2s | json

notes

  1. Both of the above test cases work correctly in version 1.0.1
  2. In newer versions (where example 2 fails), I noticed that the from value is not present in the data that was returned from the query.

I understand with with a search_type=scan, this would be expected because sorting is disabled.  Also, please note this is a greatly simplified query just for illustration purposes.  I've attached a sample of an actual query at the bottom of this post, and it does contain sorting specifications.

Is this a known issue or is this being done by design in the newer ES versions.  A quick scan through the release notes was unrevealing.

Many thanks for any insight!

--g

sample of full query


{
    "sort": [
        {
            "_type": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }, 
        {
            "name_lower": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }, 
        {
            "dds_key": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }
    ], 
    "query": {
        "filtered": {
            "filter": {
                "and": [
                    {
                        "or": [
                            {
                                "term": {
                                    "parent": "c10ed0583104012f94e11ad0ac36f2aaf"
                                }
                            }
                        ]
                    }, 
                    {
                        "not": {
                            "term": {
                                "vcn": "DeleteMarker"
                            }
                        }
                    }, 
                    {
                        "not": {
                            "exists": {
                                "field": "notfinalized"
                            }
                        }
                    }
                ]
            }, 
            "query": {
                "match_all": {}
            }
        }
    }, 
    "from": 1, 
    "size": 1000
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Met vriendelijke groet,

Martijn van Groningen



--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2BA76Tx2LjHcxMjmgo5wbFLxUYrzKR2_eHO%2B9vpCV6KGvmKZDA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: behavior of scrolling search with "from" parameter in newer ES versions

Gordon Tillman
Thank you Martijn!

On Wednesday, January 21, 2015 at 8:04:23 AM UTC-6, Martijn v Groningen wrote:
I agree, this should be fixed: <a href="https://github.com/elasticsearch/elasticsearch/issues/9373" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fissues%2F9373\46sa\75D\46sntz\0751\46usg\75AFQjCNGnNyD2xfcF016eGxpwRvOfChotwg';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fissues%2F9373\46sa\75D\46sntz\0751\46usg\75AFQjCNGnNyD2xfcF016eGxpwRvOfChotwg';return true;">https://github.com/elasticsearch/elasticsearch/issues/9373

On 21 January 2015 at 14:27, Gordon Tillman <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="5PJNGJ38oCQJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">gor...@...> wrote:
Thanks Martijn,

That is what I have observed.  But is is a regression from ES version 1.0.1 and before. And I can't find anything that even implies that `from` is not supported for a scroll search, except of course when `search_type=scan`.

I would love to get support for that back into the product.  

--g



On Wednesday, January 21, 2015 at 2:57:29 AM UTC-6, Martijn v Groningen wrote:
Hi Gordon,

This `from` is kind of ignored for scroll search. I don't remember why that was the case, but it seems to me that scroll search can/should take into account the `from` option during the first scroll search request.

Martijn

On 20 January 2015 at 23:33, Gordon Tillman <[hidden email]> wrote:
Greetings All,

I ran into an interesting issue when upgrading from ES version 1.0.1 to newer versions.  In particular, I tested the following with versions 1.2.4, 1.3.4 and 1.4.2.

summary

When doing a normal scroll search (not one with a search_type=scan), it appears that the "from" parameter is being ignored.


example 1 (no scrolling, result is correct)

In this simplified example, only one document matches the supplied query, so with from=1 no documents are returned.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4\&from=1

example 2 (with scrolling, incorrect result)

In this example, where we specify the same query and from parameters, but also specify a scroll parameter, we incorrectly get the single document returned from the query.

curl localhost:9200/hm-community-alias/FileInfo/_search?q=parents:c10ed0583104036a94e110f0a8b5fd7d4\&from=1\&scroll=2s | json

notes

  1. Both of the above test cases work correctly in version 1.0.1
  2. In newer versions (where example 2 fails), I noticed that the from value is not present in the data that was returned from the query.

I understand with with a search_type=scan, this would be expected because sorting is disabled.  Also, please note this is a greatly simplified query just for illustration purposes.  I've attached a sample of an actual query at the bottom of this post, and it does contain sorting specifications.

Is this a known issue or is this being done by design in the newer ES versions.  A quick scan through the release notes was unrevealing.

Many thanks for any insight!

--g

sample of full query


{
    "sort": [
        {
            "_type": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }, 
        {
            "name_lower": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }, 
        {
            "dds_key": {
                "order": "asc", 
                "ignore_unmapped": true
            }
        }
    ], 
    "query": {
        "filtered": {
            "filter": {
                "and": [
                    {
                        "or": [
                            {
                                "term": {
                                    "parent": "c10ed0583104012f94e11ad0ac36f2aaf"
                                }
                            }
                        ]
                    }, 
                    {
                        "not": {
                            "term": {
                                "vcn": "DeleteMarker"
                            }
                        }
                    }, 
                    {
                        "not": {
                            "exists": {
                                "field": "notfinalized"
                            }
                        }
                    }
                ]
            }, 
            "query": {
                "match_all": {}
            }
        }
    }, 
    "from": 1, 
    "size": 1000
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/ce8145ac-59c6-4a22-b4c4-eacb90281c4d%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.



--
Met vriendelijke groet,

Martijn van Groningen



--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c6ce78f2-cfe1-43e6-b759-2cb658881d99%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.