performance hit when sorting on _id

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

performance hit when sorting on _id

zohar
Hi
We are seeing a massive performance hit ( query goes from 200ms to 1min) for a largish index if we sort on
_id.  version is 0.11 we are upgrading as I type. We switch the default sort to something else but was wondering if this was expected ?
Reply | Threaded
Open this post in threaded view
|

Re: performance hit when sorting on _id

kimchy
Administrator

The first query when sorting on a field will cause all the field values to be loaded to memory. Since _id is unique, it has quite a few values, so it takes time to load them. but once its loaded, the next queries will be fast. Also, make sure you are not running into memory problems when doing it.

On Monday, December 6, 2010 at 1:51 PM, zohar wrote:


Hi
We are seeing a massive performance hit ( query goes from 200ms to 1min) for
a largish index if we sort on
_id. version is 0.11 we are upgrading as I type. We switch the default sort
to something else but was wondering if this was expected ?
--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/performance-hit-when-sorting-on-id-tp2026820p2026820.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: performance hit when sorting on _id

zohar

All filed values in the index or all field values matching the query
or filter?

On Dec 6, 4:29 pm, Shay Banon <[hidden email]> wrote:

>  The first query when sorting on a field will cause all the field values to be loaded to memory. Since _id is unique, it has quite a few values, so it takes time to load them. but once its loaded, the next queries will be fast. Also, make sure you are not running into memory problems when doing it.
>
>
>
> On Monday, December 6, 2010 at 1:51 PM, zohar wrote:
>
> > Hi
> > We are seeing a massive performance hit ( query goes from 200ms to 1min) for
> > a largish index if we sort on
> > _id. version is 0.11 we are upgrading as I type. We switch the default sort
> > to something else but was wondering if this was expected ?
> > --
> > View this message in context:http://elasticsearch-users.115913.n3.nabble.com/performance-hit-when-...
> > Sent from the ElasticSearch Users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: performance hit when sorting on _id

kimchy
Administrator

All field values in the index, thats how sorting works...

On Monday, December 6, 2010 at 11:45 PM, Zohar wrote:


All filed values in the index or all field values matching the query
or filter?

On Dec 6, 4:29 pm, Shay Banon <[hidden email]> wrote:
 The first query when sorting on a field will cause all the field values to be loaded to memory. Since _id is unique, it has quite a few values, so it takes time to load them. but once its loaded, the next queries will be fast. Also, make sure you are not running into memory problems when doing it.



On Monday, December 6, 2010 at 1:51 PM, zohar wrote:

> Hi
> We are seeing a massive performance hit ( query goes from 200ms to 1min) for
> a largish index if we sort on
> _id. version is 0.11 we are upgrading as I type. We switch the default sort
> to something else but was wondering if this was expected ?
> --
> View this message in context:http://elasticsearch-users.115913.n3.nabble.com/performance-hit-when-...
> Sent from the ElasticSearch Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: performance hit when sorting on _id

zohar
Ouch. Good to know.
On Dec 6, 10:06 pm, Shay Banon <[hidden email]> wrote:

>  All field values in the index, thats how sorting works...
>
>
>
> On Monday, December 6, 2010 at 11:45 PM, Zohar wrote:
>
> > All filed values in the index or all field values matching the query
> > or filter?
>
> > On Dec 6, 4:29 pm, Shay Banon <[hidden email]> wrote:
>
> > > The first query when sorting on a field will cause all the field values to be loaded to memory. Since _id is unique, it has quite a few values, so it takes time to load them. but once its loaded, the next queries will be fast. Also, make sure you are not running into memory problems when doing it.
>
> > >  On Monday, December 6, 2010 at 1:51 PM, zohar wrote:
>
> > > > Hi
> > > > We are seeing a massive performance hit ( query goes from 200ms to 1min) for
> > > > a largish index if we sort on
> > > > _id. version is 0.11 we are upgrading as I type. We switch the default sort
> > > > to something else but was wondering if this was expected ?
> > > > --
> > > > View this message in context:http://elasticsearch-users.115913.n3.nabble.com/performance-hit-when-...
> > > > Sent from the ElasticSearch Users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: performance hit when sorting on _id

kimchy
Administrator

Nothing new for Lucene users..., though I have ideas on how to reduce the memory implications of it.

On Wednesday, December 8, 2010 at 12:01 AM, Zohar wrote:

Ouch. Good to know.
On Dec 6, 10:06 pm, Shay Banon <[hidden email]> wrote:
 All field values in the index, thats how sorting works...



On Monday, December 6, 2010 at 11:45 PM, Zohar wrote:

> All filed values in the index or all field values matching the query
> or filter?

> On Dec 6, 4:29 pm, Shay Banon wrote:

> > The first query when sorting on a field will cause all the field values to be loaded to memory. Since _id is unique, it has quite a few values, so it takes time to load them. but once its loaded, the next queries will be fast. Also, make sure you are not running into memory problems when doing it.

> >  On Monday, December 6, 2010 at 1:51 PM, zohar wrote:

> > > Hi
> > > We are seeing a massive performance hit ( query goes from 200ms to 1min) for
> > > a largish index if we sort on
> > > _id. version is 0.11 we are upgrading as I type. We switch the default sort
> > > to something else but was wondering if this was expected ?
> > > --
> > > View this message in context:http://elasticsearch-users.115913.n3.nabble.com/performance-hit-when-...
> > > Sent from the ElasticSearch Users mailing list archive at Nabble.com.