Garbage Collection and Intermittant Search Performance issues

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Garbage Collection and Intermittant Search Performance issues

davrob
Hi,

Yesterday my users were reporting intermittant poor response times for
searches, at the same time garbage collection seems to be running more
often than usual in the index (https://gist.github.com/1031079).

JVM params in elasticsearch.in.sh are ES_MAX_MEM=2g.

My first instinct was to increase the memory to 4g, any other tips on
what I should be doing to prevent this?

Best Regards,

David.
Reply | Threaded
Open this post in threaded view
|

Re: Garbage Collection and Intermittant Search Performance issues

Enrique Medina Montenegro
Use mlockall in your configuration to see whether it helps:


(see Memory Settings section)

On Fri, Jun 17, 2011 at 10:43 AM, davrob2 <[hidden email]> wrote:
Hi,

Yesterday my users were reporting intermittant poor response times for
searches, at the same time garbage collection seems to be running more
often than usual in the index (https://gist.github.com/1031079).

JVM params in elasticsearch.in.sh are ES_MAX_MEM=2g.

My first instinct was to increase the memory to 4g, any other tips on
what I should be doing to prevent this?

Best Regards,

David.

Reply | Threaded
Open this post in threaded view
|

Re: Garbage Collection and Intermittant Search Performance issues

davrob
mlocakall seems quite dangerous:

       Note, this is experimental feature, and might cause the JVM or
shell session to exit if failing to allocate the memory (because not
enough memory is available on the machine).

Can you tell me any more about what it does and how it works?

On Jun 17, 9:45 am, Enrique Medina Montenegro <[hidden email]>
wrote:

> Use mlockall in your configuration to see whether it helps:
>
> http://www.elasticsearch.org/guide/reference/setup/installation.html
>
> (see Memory Settings section)
>
>
>
>
>
>
>
> On Fri, Jun 17, 2011 at 10:43 AM, davrob2 <[hidden email]> wrote:
> > Hi,
>
> > Yesterday my users were reporting intermittant poor response times for
> > searches, at the same time garbage collection seems to be running more
> > often than usual in the index (https://gist.github.com/1031079).
>
> > JVM params in elasticsearch.in.sh are ES_MAX_MEM=2g.
>
> > My first instinct was to increase the memory to 4g, any other tips on
> > what I should be doing to prevent this?
>
> > Best Regards,
>
> > David.
Reply | Threaded
Open this post in threaded view
|

Re: Garbage Collection and Intermittant Search Performance issues

Colin Surprenant
afaik this should only happen at startup when ES tries to allocate its
memory no?

On Fri, Jun 17, 2011 at 10:53 AM, davrob2 <[hidden email]> wrote:

> mlocakall seems quite dangerous:
>
>       Note, this is experimental feature, and might cause the JVM or
> shell session to exit if failing to allocate the memory (because not
> enough memory is available on the machine).
>
> Can you tell me any more about what it does and how it works?
>
> On Jun 17, 9:45 am, Enrique Medina Montenegro <[hidden email]>
> wrote:
>> Use mlockall in your configuration to see whether it helps:
>>
>> http://www.elasticsearch.org/guide/reference/setup/installation.html
>>
>> (see Memory Settings section)
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Jun 17, 2011 at 10:43 AM, davrob2 <[hidden email]> wrote:
>> > Hi,
>>
>> > Yesterday my users were reporting intermittant poor response times for
>> > searches, at the same time garbage collection seems to be running more
>> > often than usual in the index (https://gist.github.com/1031079).
>>
>> > JVM params in elasticsearch.in.sh are ES_MAX_MEM=2g.
>>
>> > My first instinct was to increase the memory to 4g, any other tips on
>> > what I should be doing to prevent this?
>>
>> > Best Regards,
>>
>> > David.
>
Reply | Threaded
Open this post in threaded view
|

Re: Garbage Collection and Intermittant Search Performance issues

David Williams
when using mlockall, you should also used a fixed size heap (min and
max are equal) so it either allocates all of it, or won't start.

An alternative is to just run without swap. On a dedicated elastic
search server anything that gets moved to swap is a bad thing.


On Fri, Jun 17, 2011 at 8:07 AM, Colin Surprenant
<[hidden email]> wrote:

> afaik this should only happen at startup when ES tries to allocate its
> memory no?
>
> On Fri, Jun 17, 2011 at 10:53 AM, davrob2 <[hidden email]> wrote:
>> mlocakall seems quite dangerous:
>>
>>       Note, this is experimental feature, and might cause the JVM or
>> shell session to exit if failing to allocate the memory (because not
>> enough memory is available on the machine).
>>
>> Can you tell me any more about what it does and how it works?
>>
>> On Jun 17, 9:45 am, Enrique Medina Montenegro <[hidden email]>
>> wrote:
>>> Use mlockall in your configuration to see whether it helps:
>>>
>>> http://www.elasticsearch.org/guide/reference/setup/installation.html
>>>
>>> (see Memory Settings section)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jun 17, 2011 at 10:43 AM, davrob2 <[hidden email]> wrote:
>>> > Hi,
>>>
>>> > Yesterday my users were reporting intermittant poor response times for
>>> > searches, at the same time garbage collection seems to be running more
>>> > often than usual in the index (https://gist.github.com/1031079).
>>>
>>> > JVM params in elasticsearch.in.sh are ES_MAX_MEM=2g.
>>>
>>> > My first instinct was to increase the memory to 4g, any other tips on
>>> > what I should be doing to prevent this?
>>>
>>> > Best Regards,
>>>
>>> > David.
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Garbage Collection and Intermittant Search Performance issues

davrob
Actually, I did miss this big post on dealing with these issues:

http://elasticsearch-users.115913.n3.nabble.com/0-16-Release-Plan-td2831600.html#a2833637

So I will take this approach and see what happens, including the
mlockall option.

Thanks for all your suggestions.

David.

On Jun 17, 5:55 pm, David Williams <[hidden email]> wrote:

> when using mlockall, you should also used a fixed size heap (min and
> max are equal) so it either allocates all of it, or won't start.
>
> An alternative is to just run without swap. On a dedicated elastic
> search server anything that gets moved to swap is a bad thing.
>
> On Fri, Jun 17, 2011 at 8:07 AM, Colin Surprenant
>
>
>
>
>
>
>
> <[hidden email]> wrote:
> > afaik this should only happen at startup when ES tries to allocate its
> > memory no?
>
> > On Fri, Jun 17, 2011 at 10:53 AM, davrob2 <[hidden email]> wrote:
> >> mlocakall seems quite dangerous:
>
> >>       Note, this is experimental feature, and might cause the JVM or
> >> shell session to exit if failing to allocate the memory (because not
> >> enough memory is available on the machine).
>
> >> Can you tell me any more about what it does and how it works?
>
> >> On Jun 17, 9:45 am, Enrique Medina Montenegro <[hidden email]>
> >> wrote:
> >>> Use mlockall in your configuration to see whether it helps:
>
> >>>http://www.elasticsearch.org/guide/reference/setup/installation.html
>
> >>> (see Memory Settings section)
>
> >>> On Fri, Jun 17, 2011 at 10:43 AM, davrob2 <[hidden email]> wrote:
> >>> > Hi,
>
> >>> > Yesterday my users were reporting intermittant poor response times for
> >>> > searches, at the same time garbage collection seems to be running more
> >>> > often than usual in the index (https://gist.github.com/1031079).
>
> >>> > JVM params in elasticsearch.in.sh are ES_MAX_MEM=2g.
>
> >>> > My first instinct was to increase the memory to 4g, any other tips on
> >>> > what I should be doing to prevent this?
>
> >>> > Best Regards,
>
> >>> > David.
Reply | Threaded
Open this post in threaded view
|

Re: Garbage Collection and Intermittant Search Performance issues

davrob
Regarding the memory settings here:  http://www.elasticsearch.org/guide/reference/setup/installation.html

Can anyone recommend the equivalent of ulimit -l for Solaris?

Is it necessary / possible to apply this setting to lock the memory in
Solaris?

When I execute the ulimit command it returns "unlimited", but I'm not
sure if that has anything to do with the "max locked memory" setting.



On Jun 18, 4:06 pm, davrob2 <[hidden email]> wrote:

> Actually, I did miss this big post on dealing with these issues:
>
> http://elasticsearch-users.115913.n3.nabble.com/0-16-Release-Plan-td2...
>
> So I will take this approach and see what happens, including the
> mlockall option.
>
> Thanks for all your suggestions.
>
> David.
>
> On Jun 17, 5:55 pm, David Williams <[hidden email]> wrote:
>
>
>
>
>
>
>
> > when using mlockall, you should also used a fixed size heap (min and
> > max are equal) so it either allocates all of it, or won't start.
>
> > An alternative is to just run without swap. On a dedicated elastic
> > search server anything that gets moved to swap is a bad thing.
>
> > On Fri, Jun 17, 2011 at 8:07 AM, Colin Surprenant
>
> > <[hidden email]> wrote:
> > > afaik this should only happen at startup when ES tries to allocate its
> > > memory no?
>
> > > On Fri, Jun 17, 2011 at 10:53 AM, davrob2 <[hidden email]> wrote:
> > >> mlocakall seems quite dangerous:
>
> > >>       Note, this is experimental feature, and might cause the JVM or
> > >> shell session to exit if failing to allocate the memory (because not
> > >> enough memory is available on the machine).
>
> > >> Can you tell me any more about what it does and how it works?
>
> > >> On Jun 17, 9:45 am, Enrique Medina Montenegro <[hidden email]>
> > >> wrote:
> > >>> Use mlockall in your configuration to see whether it helps:
>
> > >>>http://www.elasticsearch.org/guide/reference/setup/installation.html
>
> > >>> (see Memory Settings section)
>
> > >>> On Fri, Jun 17, 2011 at 10:43 AM, davrob2 <[hidden email]> wrote:
> > >>> > Hi,
>
> > >>> > Yesterday my users were reporting intermittant poor response times for
> > >>> > searches, at the same time garbage collection seems to be running more
> > >>> > often than usual in the index (https://gist.github.com/1031079).
>
> > >>> > JVM params in elasticsearch.in.sh are ES_MAX_MEM=2g.
>
> > >>> > My first instinct was to increase the memory to 4g, any other tips on
> > >>> > what I should be doing to prevent this?
>
> > >>> > Best Regards,
>
> > >>> > David.