Ridiculously long Scroll_id

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Ridiculously long Scroll_id

Scott Chung

Does Scroll_id get longer with the number of shards? I have 500 shards (for using custom routing and speeding up search queries), and I when I tried to do a scan search, I got back a scroll_id of length 20708.

My guess is that this is because the scroll_id is some id for each shard concatenated together. (Is my guess completely off?) Is there a way to make each shard use the same id, or have a table that translates a public scroll_id that's returned to one that's used internally?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Ridiculously long Scroll_id

Oli McCormack

The scroll id is a base64 encoded string, that I believe does have information relating to your shards.

Decode it to see its contents.

On Jul 23, 2013 5:50 PM, "Scott Chung" <[hidden email]> wrote:

Does Scroll_id get longer with the number of shards? I have 500 shards (for using custom routing and speeding up search queries), and I when I tried to do a scan search, I got back a scroll_id of length 20708.

My guess is that this is because the scroll_id is some id for each shard concatenated together. (Is my guess completely off?) Is there a way to make each shard use the same id, or have a table that translates a public scroll_id that's returned to one that's used internally?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Ridiculously long Scroll_id

Scott Chung
When I decode the scroll id, I get the following:
   "scan;500;1607138:Ah5ZV4RiRRmr3OjrrZB48A;1661193:3FbURZzHT3-XHuYGQCRQ3g;1636318:NbHVKbL2TJedKQI8ejKA2g;1607139:Ah5ZV4RiRRmr3OjrrZB48A;1713931:-1U8JPQ0TvGk8XX_6dGp_A;...."

So, my guess is this is "scan; {# of shards}; {shard_id_1}:{corresponding_scroll_id_1}; {shard_id_2}:{corresponding_scroll_id_2}; ..."
So, the scroll_id grows linearly to the number of shards. In fact, for me, the scroll_id is too large to pass as part of the url.
Can we get a fix for this?

On Tuesday, July 23, 2013 6:20:22 PM UTC-7, Oli wrote:

The scroll id is a base64 encoded string, that I believe does have information relating to your shards.

Decode it to see its contents.

On Jul 23, 2013 5:50 PM, "Scott Chung" <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="w8vqxV2gMpEJ">scot...@...> wrote:

Does Scroll_id get longer with the number of shards? I have 500 shards (for using custom routing and speeding up search queries), and I when I tried to do a scan search, I got back a scroll_id of length 20708.

My guess is that this is because the scroll_id is some id for each shard concatenated together. (Is my guess completely off?) Is there a way to make each shard use the same id, or have a table that translates a public scroll_id that's returned to one that's used internally?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="w8vqxV2gMpEJ">elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Ridiculously long Scroll_id

Igor Motov-3
You don't have to pass it as part of the url. You can simply send it in the request body (http://www.elasticsearch.org/guide/reference/api/search/search-type/).  

On Tuesday, July 23, 2013 9:31:45 PM UTC-4, Scott Chung wrote:
When I decode the scroll id, I get the following:
   "scan;500;1607138:Ah5ZV4RiRRmr3OjrrZB48A;1661193:3FbURZzHT3-XHuYGQCRQ3g;1636318:NbHVKbL2TJedKQI8ejKA2g;1607139:Ah5ZV4RiRRmr3OjrrZB48A;1713931:-1U8JPQ0TvGk8XX_6dGp_A;...."

So, my guess is this is "scan; {# of shards}; {shard_id_1}:{corresponding_scroll_id_1}; {shard_id_2}:{corresponding_scroll_id_2}; ..."
So, the scroll_id grows linearly to the number of shards. In fact, for me, the scroll_id is too large to pass as part of the url.
Can we get a fix for this?

On Tuesday, July 23, 2013 6:20:22 PM UTC-7, Oli wrote:

The scroll id is a base64 encoded string, that I believe does have information relating to your shards.

Decode it to see its contents.

On Jul 23, 2013 5:50 PM, "Scott Chung" <[hidden email]> wrote:

Does Scroll_id get longer with the number of shards? I have 500 shards (for using custom routing and speeding up search queries), and I when I tried to do a scan search, I got back a scroll_id of length 20708.

My guess is that this is because the scroll_id is some id for each shard concatenated together. (Is my guess completely off?) Is there a way to make each shard use the same id, or have a table that translates a public scroll_id that's returned to one that's used internally?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Ridiculously long Scroll_id

Scott Chung
Yes, that is what I ended up doing. (In case anybody's also using Nest and runs into this trouble. You can use the "searchRaw" method to trick the client into parsing the results for you.)
But, it's still not really desirable to be passing really long ID's around.

On Tuesday, July 23, 2013 9:29:16 PM UTC-7, Igor Motov wrote:
You don't have to pass it as part of the url. You can simply send it in the request body (http://www.elasticsearch.org/guide/reference/api/search/search-type/).  

On Tuesday, July 23, 2013 9:31:45 PM UTC-4, Scott Chung wrote:
When I decode the scroll id, I get the following:
   "scan;500;1607138:Ah5ZV4RiRRmr3OjrrZB48A;1661193:3FbURZzHT3-XHuYGQCRQ3g;1636318:NbHVKbL2TJedKQI8ejKA2g;1607139:Ah5ZV4RiRRmr3OjrrZB48A;1713931:-1U8JPQ0TvGk8XX_6dGp_A;...."

So, my guess is this is "scan; {# of shards}; {shard_id_1}:{corresponding_scroll_id_1}; {shard_id_2}:{corresponding_scroll_id_2}; ..."
So, the scroll_id grows linearly to the number of shards. In fact, for me, the scroll_id is too large to pass as part of the url.
Can we get a fix for this?

On Tuesday, July 23, 2013 6:20:22 PM UTC-7, Oli wrote:

The scroll id is a base64 encoded string, that I believe does have information relating to your shards.

Decode it to see its contents.

On Jul 23, 2013 5:50 PM, "Scott Chung" <[hidden email]> wrote:

Does Scroll_id get longer with the number of shards? I have 500 shards (for using custom routing and speeding up search queries), and I when I tried to do a scan search, I got back a scroll_id of length 20708.

My guess is that this is because the scroll_id is some id for each shard concatenated together. (Is my guess completely off?) Is there a way to make each shard use the same id, or have a table that translates a public scroll_id that's returned to one that's used internally?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.