$ES_HEAP_SIZE

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

$ES_HEAP_SIZE

kayngee
Hi Guys,

If i Set $ES_HEAP_SIZE to 64g should i see that the system is reserving 64GB of memory using htop?

Regards
Shawn

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

Clinton Gormley-2
On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote:
> Hi Guys,
>
>
> If i Set $ES_HEAP_SIZE to 64g should i see that the system is
> reserving 64GB of memory using htop?

you should.  note that using more than 30GB for the heap will prevent
the jvm from using compressed pointers, and will increase GC load. you
probably don't want to do this

clint

>
>
> Regards
> Shawn
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

kayngee
What is i set it exactly to 30GB do i still get this problem.

On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote:
> Hi Guys,
>
>
> If i Set $ES_HEAP_SIZE to 64g should i see that the system is
> reserving 64GB of memory using htop?

you should.  note that using more than 30GB for the heap will prevent
the jvm from using compressed pointers, and will increase GC load. you
probably don't want to do this

clint

>
>
> Regards
> Shawn
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="tobs0ljvDDgJ">elasticsearc...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

Clinton Gormley-2
On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote:
> What is i set it exactly to 30GB do i still get this problem.

?

>
> On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley wrote:
>         On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote:
>         > Hi Guys,
>         >
>         >
>         > If i Set $ES_HEAP_SIZE to 64g should i see that the system
>         is
>         > reserving 64GB of memory using htop?
>        
>         you should.  note that using more than 30GB for the heap will
>         prevent
>         the jvm from using compressed pointers, and will increase GC
>         load. you
>         probably don't want to do this
>        
>         clint
>        
>         >
>         >
>         > Regards
>         > Shawn
>         >
>         > --
>         > You received this message because you are subscribed to the
>         Google
>         > Groups "elasticsearch" group.
>         > To unsubscribe from this group and stop receiving emails
>         from it, send
>         > an email to [hidden email].
>         > For more options, visit
>         https://groups.google.com/groups/opt_out.
>         >  
>         >  
>        
>        
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

kayngee
If i set the ES_HEAP_SIZE to 30GB would i avoid the compressed pointers problem?

On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote:
> What is i set it exactly to 30GB do i still get this problem.

?

>
> On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley wrote:
>         On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote:
>         > Hi Guys,
>         >
>         >
>         > If i Set $ES_HEAP_SIZE to 64g should i see that the system
>         is
>         > reserving 64GB of memory using htop?
>        
>         you should.  note that using more than 30GB for the heap will
>         prevent
>         the jvm from using compressed pointers, and will increase GC
>         load. you
>         probably don't want to do this
>        
>         clint
>        
>         >
>         >
>         > Regards
>         > Shawn
>         >
>         > --
>         > You received this message because you are subscribed to the
>         Google
>         > Groups "elasticsearch" group.
>         > To unsubscribe from this group and stop receiving emails
>         from it, send
>         > an email to elasticsearc...@googlegroups.com.
>         > For more options, visit
>         https://groups.google.com/groups/opt_out.
>         >  
>         >  
>        
>        
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="u2fHh-o8bMQJ">elasticsearc...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

Clinton Gormley-2
On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote:
> If i set the ES_HEAP_SIZE to 30GB would i avoid the compressed
> pointers problem?

yes :)

don't forget that you should leave about 50% of your RAM for kernel
filesystem cache as well

clint

>
> On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley wrote:
>         On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote:
>         > What is i set it exactly to 30GB do i still get this
>         problem.
>        
>         ?
>        
>         >
>         > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley
>         wrote:
>         >         On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie
>         wrote:
>         >         > Hi Guys,
>         >         >
>         >         >
>         >         > If i Set $ES_HEAP_SIZE to 64g should i see that
>         the system
>         >         is
>         >         > reserving 64GB of memory using htop?
>         >        
>         >         you should.  note that using more than 30GB for the
>         heap will
>         >         prevent
>         >         the jvm from using compressed pointers, and will
>         increase GC
>         >         load. you
>         >         probably don't want to do this
>         >        
>         >         clint
>         >        
>         >         >
>         >         >
>         >         > Regards
>         >         > Shawn
>         >         >
>         >         > --
>         >         > You received this message because you are
>         subscribed to the
>         >         Google
>         >         > Groups "elasticsearch" group.
>         >         > To unsubscribe from this group and stop receiving
>         emails
>         >         from it, send
>         >         > an email to [hidden email].
>         >         > For more options, visit
>         >         https://groups.google.com/groups/opt_out.
>         >         >  
>         >         >  
>         >        
>         >        
>         >
>         > --
>         > You received this message because you are subscribed to the
>         Google
>         > Groups "elasticsearch" group.
>         > To unsubscribe from this group and stop receiving emails
>         from it, send
>         > an email to [hidden email].
>         > For more options, visit
>         https://groups.google.com/groups/opt_out.
>         >  
>         >  
>        
>        
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

kayngee
Shouldn't be problematic as the server has 128GB of RAM. So if i set that ES_HEAP_SIZE to 30GB and run elastic search using top should i see elastic search using up 30GB of ram?

Regards
Shawn

On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote:
> If i set the ES_HEAP_SIZE to 30GB would i avoid the compressed
> pointers problem?

yes :)

don't forget that you should leave about 50% of your RAM for kernel
filesystem cache as well

clint

>
> On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley wrote:
>         On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote:
>         > What is i set it exactly to 30GB do i still get this
>         problem.
>        
>         ?
>        
>         >
>         > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley
>         wrote:
>         >         On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie
>         wrote:
>         >         > Hi Guys,
>         >         >
>         >         >
>         >         > If i Set $ES_HEAP_SIZE to 64g should i see that
>         the system
>         >         is
>         >         > reserving 64GB of memory using htop?
>         >        
>         >         you should.  note that using more than 30GB for the
>         heap will
>         >         prevent
>         >         the jvm from using compressed pointers, and will
>         increase GC
>         >         load. you
>         >         probably don't want to do this
>         >        
>         >         clint
>         >        
>         >         >
>         >         >
>         >         > Regards
>         >         > Shawn
>         >         >
>         >         > --
>         >         > You received this message because you are
>         subscribed to the
>         >         Google
>         >         > Groups "elasticsearch" group.
>         >         > To unsubscribe from this group and stop receiving
>         emails
>         >         from it, send
>         >         > an email to elasticsearc...@googlegroups.com.
>         >         > For more options, visit
>         >         https://groups.google.com/groups/opt_out.
>         >         >  
>         >         >  
>         >        
>         >        
>         >
>         > --
>         > You received this message because you are subscribed to the
>         Google
>         > Groups "elasticsearch" group.
>         > To unsubscribe from this group and stop receiving emails
>         from it, send
>         > an email to elasticsearc...@googlegroups.com.
>         > For more options, visit
>         https://groups.google.com/groups/opt_out.
>         >  
>         >  
>        
>        
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="T_kO58aF56IJ">elasticsearc...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

Clinton Gormley-2
On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote:
> Shouldn't be problematic as the server has 128GB of RAM. So if i set
> that ES_HEAP_SIZE to 30GB and run elastic search using top should i
> see elastic search using up 30GB of ram?

yes

with 128GB of RAM, you may consider running two instance of ES on the
same machine, but just make sure that you use "awareness" to ensure that
you don't have the same primary and replica shards on a single box.

http://www.elasticsearch.org/guide/reference/modules/cluster.html

clint

>
>
> Regards
> Shawn
>
> On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley wrote:
>         On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote:
>         > If i set the ES_HEAP_SIZE to 30GB would i avoid the
>         compressed
>         > pointers problem?
>        
>         yes :)
>        
>         don't forget that you should leave about 50% of your RAM for
>         kernel
>         filesystem cache as well
>        
>         clint
>        
>         >
>         > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley
>         wrote:
>         >         On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie
>         wrote:
>         >         > What is i set it exactly to 30GB do i still get
>         this
>         >         problem.
>         >        
>         >         ?
>         >        
>         >         >
>         >         > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton
>         Gormley
>         >         wrote:
>         >         >         On Fri, 2013-02-08 at 01:08 -0800, Shawn
>         Ritchie
>         >         wrote:
>         >         >         > Hi Guys,
>         >         >         >
>         >         >         >
>         >         >         > If i Set $ES_HEAP_SIZE to 64g should i
>         see that
>         >         the system
>         >         >         is
>         >         >         > reserving 64GB of memory using htop?
>         >         >        
>         >         >         you should.  note that using more than
>         30GB for the
>         >         heap will
>         >         >         prevent
>         >         >         the jvm from using compressed pointers,
>         and will
>         >         increase GC
>         >         >         load. you
>         >         >         probably don't want to do this
>         >         >        
>         >         >         clint
>         >         >        
>         >         >         >
>         >         >         >
>         >         >         > Regards
>         >         >         > Shawn
>         >         >         >
>         >         >         > --
>         >         >         > You received this message because you
>         are
>         >         subscribed to the
>         >         >         Google
>         >         >         > Groups "elasticsearch" group.
>         >         >         > To unsubscribe from this group and stop
>         receiving
>         >         emails
>         >         >         from it, send
>         >         >         > an email to
>         [hidden email].
>         >         >         > For more options, visit
>         >         >         https://groups.google.com/groups/opt_out.
>         >         >         >  
>         >         >         >  
>         >         >        
>         >         >        
>         >         >
>         >         > --
>         >         > You received this message because you are
>         subscribed to the
>         >         Google
>         >         > Groups "elasticsearch" group.
>         >         > To unsubscribe from this group and stop receiving
>         emails
>         >         from it, send
>         >         > an email to [hidden email].
>         >         > For more options, visit
>         >         https://groups.google.com/groups/opt_out.
>         >         >  
>         >         >  
>         >        
>         >        
>         >
>         > --
>         > You received this message because you are subscribed to the
>         Google
>         > Groups "elasticsearch" group.
>         > To unsubscribe from this group and stop receiving emails
>         from it, send
>         > an email to [hidden email].
>         > For more options, visit
>         https://groups.google.com/groups/opt_out.
>         >  
>         >  
>        
>        
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

kayngee
Would I get greater throughput though using 2 instances?


Regards
Shawn

On Friday, 8 February 2013 12:11:24 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote:
> Shouldn't be problematic as the server has 128GB of RAM. So if i set
> that ES_HEAP_SIZE to 30GB and run elastic search using top should i
> see elastic search using up 30GB of ram?

yes

with 128GB of RAM, you may consider running two instance of ES on the
same machine, but just make sure that you use "awareness" to ensure that
you don't have the same primary and replica shards on a single box.

http://www.elasticsearch.org/guide/reference/modules/cluster.html

clint

>
>
> Regards
> Shawn
>
> On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley wrote:
>         On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote:
>         > If i set the ES_HEAP_SIZE to 30GB would i avoid the
>         compressed
>         > pointers problem?
>        
>         yes :)
>        
>         don't forget that you should leave about 50% of your RAM for
>         kernel
>         filesystem cache as well
>        
>         clint
>        
>         >
>         > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley
>         wrote:
>         >         On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie
>         wrote:
>         >         > What is i set it exactly to 30GB do i still get
>         this
>         >         problem.
>         >        
>         >         ?
>         >        
>         >         >
>         >         > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton
>         Gormley
>         >         wrote:
>         >         >         On Fri, 2013-02-08 at 01:08 -0800, Shawn
>         Ritchie
>         >         wrote:
>         >         >         > Hi Guys,
>         >         >         >
>         >         >         >
>         >         >         > If i Set $ES_HEAP_SIZE to 64g should i
>         see that
>         >         the system
>         >         >         is
>         >         >         > reserving 64GB of memory using htop?
>         >         >        
>         >         >         you should.  note that using more than
>         30GB for the
>         >         heap will
>         >         >         prevent
>         >         >         the jvm from using compressed pointers,
>         and will
>         >         increase GC
>         >         >         load. you
>         >         >         probably don't want to do this
>         >         >        
>         >         >         clint
>         >         >        
>         >         >         >
>         >         >         >
>         >         >         > Regards
>         >         >         > Shawn
>         >         >         >
>         >         >         > --
>         >         >         > You received this message because you
>         are
>         >         subscribed to the
>         >         >         Google
>         >         >         > Groups "elasticsearch" group.
>         >         >         > To unsubscribe from this group and stop
>         receiving
>         >         emails
>         >         >         from it, send
>         >         >         > an email to
>         elasticsearc...@googlegroups.com.
>         >         >         > For more options, visit
>         >         >         https://groups.google.com/groups/opt_out.
>         >         >         >  
>         >         >         >  
>         >         >        
>         >         >        
>         >         >
>         >         > --
>         >         > You received this message because you are
>         subscribed to the
>         >         Google
>         >         > Groups "elasticsearch" group.
>         >         > To unsubscribe from this group and stop receiving
>         emails
>         >         from it, send
>         >         > an email to elasticsearc...@googlegroups.com.
>         >         > For more options, visit
>         >         https://groups.google.com/groups/opt_out.
>         >         >  
>         >         >  
>         >        
>         >        
>         >
>         > --
>         > You received this message because you are subscribed to the
>         Google
>         > Groups "elasticsearch" group.
>         > To unsubscribe from this group and stop receiving emails
>         from it, send
>         > an email to elasticsearc...@googlegroups.com.
>         > For more options, visit
>         https://groups.google.com/groups/opt_out.
>         >  
>         >  
>        
>        
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="uvBou5PL8ToJ">elasticsearc...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

Clinton Gormley-2
On Fri, 2013-02-08 at 03:14 -0800, Shawn Ritchie wrote:
> Would I get greater throughput though using 2 instances?

they'd be sharing CPU and IO but you'd be making use of all of your RAM
(as opposed to wasting half of it).

the answer is: I think so, but not entirely sure :)

try it and see - let us know what you find

clint

>
>
> Regards
> Shawn
>
> On Friday, 8 February 2013 12:11:24 UTC+1, Clinton Gormley wrote:
>         On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote:
>         > Shouldn't be problematic as the server has 128GB of RAM. So
>         if i set
>         > that ES_HEAP_SIZE to 30GB and run elastic search using top
>         should i
>         > see elastic search using up 30GB of ram?
>        
>         yes
>        
>         with 128GB of RAM, you may consider running two instance of ES
>         on the
>         same machine, but just make sure that you use "awareness" to
>         ensure that
>         you don't have the same primary and replica shards on a single
>         box.
>        
>         http://www.elasticsearch.org/guide/reference/modules/cluster.html 
>        
>         clint
>        
>         >
>         >
>         > Regards
>         > Shawn
>         >
>         > On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley
>         wrote:
>         >         On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie
>         wrote:
>         >         > If i set the ES_HEAP_SIZE to 30GB would i avoid
>         the
>         >         compressed
>         >         > pointers problem?
>         >        
>         >         yes :)
>         >        
>         >         don't forget that you should leave about 50% of your
>         RAM for
>         >         kernel
>         >         filesystem cache as well
>         >        
>         >         clint
>         >        
>         >         >
>         >         > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton
>         Gormley
>         >         wrote:
>         >         >         On Fri, 2013-02-08 at 02:27 -0800, Shawn
>         Ritchie
>         >         wrote:
>         >         >         > What is i set it exactly to 30GB do i
>         still get
>         >         this
>         >         >         problem.
>         >         >        
>         >         >         ?
>         >         >        
>         >         >         >
>         >         >         > On Friday, 8 February 2013 11:22:36 UTC
>         +1, Clinton
>         >         Gormley
>         >         >         wrote:
>         >         >         >         On Fri, 2013-02-08 at 01:08
>         -0800, Shawn
>         >         Ritchie
>         >         >         wrote:
>         >         >         >         > Hi Guys,
>         >         >         >         >
>         >         >         >         >
>         >         >         >         > If i Set $ES_HEAP_SIZE to 64g
>         should i
>         >         see that
>         >         >         the system
>         >         >         >         is
>         >         >         >         > reserving 64GB of memory using
>         htop?
>         >         >         >        
>         >         >         >         you should.  note that using
>         more than
>         >         30GB for the
>         >         >         heap will
>         >         >         >         prevent
>         >         >         >         the jvm from using compressed
>         pointers,
>         >         and will
>         >         >         increase GC
>         >         >         >         load. you
>         >         >         >         probably don't want to do this
>         >         >         >        
>         >         >         >         clint
>         >         >         >        
>         >         >         >         >
>         >         >         >         >
>         >         >         >         > Regards
>         >         >         >         > Shawn
>         >         >         >         >
>         >         >         >         > --
>         >         >         >         > You received this message
>         because you
>         >         are
>         >         >         subscribed to the
>         >         >         >         Google
>         >         >         >         > Groups "elasticsearch" group.
>         >         >         >         > To unsubscribe from this group
>         and stop
>         >         receiving
>         >         >         emails
>         >         >         >         from it, send
>         >         >         >         > an email to
>         >         [hidden email].
>         >         >         >         > For more options, visit
>         >         >         >
>         https://groups.google.com/groups/opt_out.
>         >         >         >         >  
>         >         >         >         >  
>         >         >         >        
>         >         >         >        
>         >         >         >
>         >         >         > --
>         >         >         > You received this message because you
>         are
>         >         subscribed to the
>         >         >         Google
>         >         >         > Groups "elasticsearch" group.
>         >         >         > To unsubscribe from this group and stop
>         receiving
>         >         emails
>         >         >         from it, send
>         >         >         > an email to
>         [hidden email].
>         >         >         > For more options, visit
>         >         >         https://groups.google.com/groups/opt_out.
>         >         >         >  
>         >         >         >  
>         >         >        
>         >         >        
>         >         >
>         >         > --
>         >         > You received this message because you are
>         subscribed to the
>         >         Google
>         >         > Groups "elasticsearch" group.
>         >         > To unsubscribe from this group and stop receiving
>         emails
>         >         from it, send
>         >         > an email to [hidden email].
>         >         > For more options, visit
>         >         https://groups.google.com/groups/opt_out.
>         >         >  
>         >         >  
>         >        
>         >        
>         >
>         > --
>         > You received this message because you are subscribed to the
>         Google
>         > Groups "elasticsearch" group.
>         > To unsubscribe from this group and stop receiving emails
>         from it, send
>         > an email to [hidden email].
>         > For more options, visit
>         https://groups.google.com/groups/opt_out.
>         >  
>         >  
>        
>        
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

kayngee
Hmm, i will need some help with this,

so what you are implying i run 2 instances of ElasticSearch with different configs, 

Set one instance with 
node.rack_id: rack_one
cluster.routing.allocation.awareness.attributes: rack_id
node.master: true node.data: true
index.number_of_shards: 4
index.number_of_replicas: 0

set another instance with 

node.rack_id: rack_two
cluster.routing.allocation.awareness.attributes: rack_two
node.master: false node.data: true
index.number_of_shards: 4
index.number_of_replicas: 0

and this will automatically 2 shards on each instance?

Regards
Shawn


On Friday, 8 February 2013 12:21:06 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 03:14 -0800, Shawn Ritchie wrote:
> Would I get greater throughput though using 2 instances?

they'd be sharing CPU and IO but you'd be making use of all of your RAM
(as opposed to wasting half of it).

the answer is: I think so, but not entirely sure :)

try it and see - let us know what you find

clint

>
>
> Regards
> Shawn
>
> On Friday, 8 February 2013 12:11:24 UTC+1, Clinton Gormley wrote:
>         On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote:
>         > Shouldn't be problematic as the server has 128GB of RAM. So
>         if i set
>         > that ES_HEAP_SIZE to 30GB and run elastic search using top
>         should i
>         > see elastic search using up 30GB of ram?
>        
>         yes
>        
>         with 128GB of RAM, you may consider running two instance of ES
>         on the
>         same machine, but just make sure that you use "awareness" to
>         ensure that
>         you don't have the same primary and replica shards on a single
>         box.
>        
>         http://www.elasticsearch.org/guide/reference/modules/cluster.html
>        
>         clint
>        
>         >
>         >
>         > Regards
>         > Shawn
>         >
>         > On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley
>         wrote:
>         >         On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie
>         wrote:
>         >         > If i set the ES_HEAP_SIZE to 30GB would i avoid
>         the
>         >         compressed
>         >         > pointers problem?
>         >        
>         >         yes :)
>         >        
>         >         don't forget that you should leave about 50% of your
>         RAM for
>         >         kernel
>         >         filesystem cache as well
>         >        
>         >         clint
>         >        
>         >         >
>         >         > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton
>         Gormley
>         >         wrote:
>         >         >         On Fri, 2013-02-08 at 02:27 -0800, Shawn
>         Ritchie
>         >         wrote:
>         >         >         > What is i set it exactly to 30GB do i
>         still get
>         >         this
>         >         >         problem.
>         >         >        
>         >         >         ?
>         >         >        
>         >         >         >
>         >         >         > On Friday, 8 February 2013 11:22:36 UTC
>         +1, Clinton
>         >         Gormley
>         >         >         wrote:
>         >         >         >         On Fri, 2013-02-08 at 01:08
>         -0800, Shawn
>         >         Ritchie
>         >         >         wrote:
>         >         >         >         > Hi Guys,
>         >         >         >         >
>         >         >         >         >
>         >         >         >         > If i Set $ES_HEAP_SIZE to 64g
>         should i
>         >         see that
>         >         >         the system
>         >         >         >         is
>         >         >         >         > reserving 64GB of memory using
>         htop?
>         >         >         >        
>         >         >         >         you should.  note that using
>         more than
>         >         30GB for the
>         >         >         heap will
>         >         >         >         prevent
>         >         >         >         the jvm from using compressed
>         pointers,
>         >         and will
>         >         >         increase GC
>         >         >         >         load. you
>         >         >         >         probably don't want to do this
>         >         >         >        
>         >         >         >         clint
>         >         >         >        
>         >         >         >         >
>         >         >         >         >
>         >         >         >         > Regards
>         >         >         >         > Shawn
>         >         >         >         >
>         >         >         >         > --
>         >         >         >         > You received this message
>         because you
>         >         are
>         >         >         subscribed to the
>         >         >         >         Google
>         >         >         >         > Groups "elasticsearch" group.
>         >         >         >         > To unsubscribe from this group
>         and stop
>         >         receiving
>         >         >         emails
>         >         >         >         from it, send
>         >         >         >         > an email to
>         >         elasticsearc...@googlegroups.com.
>         >         >         >         > For more options, visit
>         >         >         >
>         https://groups.google.com/groups/opt_out.
>         >         >         >         >  
>         >         >         >         >  
>         >         >         >        
>         >         >         >        
>         >         >         >
>         >         >         > --
>         >         >         > You received this message because you
>         are
>         >         subscribed to the
>         >         >         Google
>         >         >         > Groups "elasticsearch" group.
>         >         >         > To unsubscribe from this group and stop
>         receiving
>         >         emails
>         >         >         from it, send
>         >         >         > an email to
>         elasticsearc...@googlegroups.com.
>         >         >         > For more options, visit
>         >         >         https://groups.google.com/groups/opt_out.
>         >         >         >  
>         >         >         >  
>         >         >        
>         >         >        
>         >         >
>         >         > --
>         >         > You received this message because you are
>         subscribed to the
>         >         Google
>         >         > Groups "elasticsearch" group.
>         >         > To unsubscribe from this group and stop receiving
>         emails
>         >         from it, send
>         >         > an email to elasticsearc...@googlegroups.com.
>         >         > For more options, visit
>         >         https://groups.google.com/groups/opt_out.
>         >         >  
>         >         >  
>         >        
>         >        
>         >
>         > --
>         > You received this message because you are subscribed to the
>         Google
>         > Groups "elasticsearch" group.
>         > To unsubscribe from this group and stop receiving emails
>         from it, send
>         > an email to elasticsearc...@googlegroups.com.
>         > For more options, visit
>         https://groups.google.com/groups/opt_out.
>         >  
>         >  
>        
>        
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="5s0hOBYfkdsJ">elasticsearc...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

Clinton Gormley-2
On Fri, 2013-02-08 at 03:27 -0800, Shawn Ritchie wrote:
> Hmm, i will need some help with this,
>
>
> so what you are implying i run 2 instances of ElasticSearch with
> different configs,

OK, first question is: do you have just one server or more? If you have
just one server then you are at risk. If that server dies, you will lose
your cluster and (eg if the disks die) lose your data.

With just one server, the "awareness" doesn't matter - if the server
dies, you're screwed.

If you have more than one server then Elasticsearch puts a primary shard
on one server and a replica of the primary on the other server. That
way, if you lose one server, all of your data is safe and the cluster
can keep on running.

If you are running two nodes on one server and two nodes on another
server, then you need to use awareness to ensure that the primary and
replica shards are not placed on the same server.

So the two nodes running on the same server would have the same rack ID.
That way Elasticsearch knows to place the primary shards on one rack,
and the replica shards on the other rack.

clint


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

kayngee
Hi, 

Basically we only have 1 machine for Elastic Search, and for now are not interested in replication, for now we are only interested in maximising performance.

So if I understood we do not need the awareness field! and the 2 elastic search instances will take care of 2 shards each automatically?

Also each instance i'm  guessing needs to be configured with different ports? and which instance will i be querying?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

Clinton Gormley-2

>
> Basically we only have 1 machine for Elastic Search, and for now are
> not interested in replication, for now we are only interested in
> maximising performance.

OK
>
> So if I understood we do not need the awareness field! and the 2
> elastic search instances will take care of 2 shards each
> automatically?

Yes

>
> Also each instance i'm  guessing needs to be configured with different
> ports? and which instance will i be querying?

Each node will try to connect to port 9300 and if that is not available,
try 9301, 9302 etc.  This is the port for communication between nodes
(and for the java clients), so make sure that you have 9300 and 9301
open in your firewall.

Similarly, each node will bind to 9200, 9201.... for the HTTP
interface.

This is automatic - you don't need to configure anything except for
opening the ports on your firewall.

You can talk to either node (http on 9200 or 9201, java on 9300, 9301)

clint


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

kayngee
In reply to this post by kayngee
or do i only need to set node.max_local_storage_nodes: 2 and it will automatically setup 2 instances?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

Clinton Gormley-2
On Fri, 2013-02-08 at 04:07 -0800, Shawn Ritchie wrote:
> or do i only need to set node.max_local_storage_nodes: 2 and it will
> automatically setup 2 instances?

No, you need to start two instances.  the above setting just stops you
from starting more than X instances on the same server (or at least from
the same directory)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

kayngee
Last question :D

How do i lunch and instance with a specific config file? As i find it quite ironic that you specify the config file directory in the config file itself?

On Friday, 8 February 2013 13:09:10 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 04:07 -0800, Shawn Ritchie wrote:
> or do i only need to set node.max_local_storage_nodes: 2 and it will
> automatically setup 2 instances?

No, you need to start two instances.  the above setting just stops you
from starting more than X instances on the same server (or at least from
the same directory)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

Clinton Gormley-2
On Fri, 2013-02-08 at 04:14 -0800, Shawn Ritchie wrote:
> Last question :D
>
> How do i lunch and instance with a specific config file? As i find it
> quite ironic that you specify the config file directory in the config
> file itself?

http://www.elasticsearch.org/guide/reference/setup/configuration.html


>
> On Friday, 8 February 2013 13:09:10 UTC+1, Clinton Gormley wrote:
>         On Fri, 2013-02-08 at 04:07 -0800, Shawn Ritchie wrote:
>         > or do i only need to set node.max_local_storage_nodes: 2 and
>         it will
>         > automatically setup 2 instances?
>        
>         No, you need to start two instances.  the above setting just
>         stops you
>         from starting more than X instances on the same server (or at
>         least from
>         the same directory)
>        
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

kayngee
Already read that post, but from what i understood or misunderstood, is its making the assumption you will have 1 instance of elastic search running on a machine.

What i'd like to do is with 1 elastic search installation is lunch 2 instance of elastic search with different /config /data /log node name.

Or is it that multiple versions of elastic search on the same machine run @ a directory level, that is 2 instances which are sharing the /config /data and /log directries together with the node name?


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: $ES_HEAP_SIZE

Clinton Gormley-2
On Fri, 2013-02-08 at 04:27 -0800, Shawn Ritchie wrote:

> Already read that post, but from what i understood or misunderstood,
> is its making the assumption you will have 1 instance of elastic
> search running on a machine.
>
> What i'd like to do is with 1 elastic search installation is lunch 2
> instance of elastic search with different /config /data /log node
> name.
>
>
> Or is it that multiple versions of elastic search on the same machine
> run @ a directory level, that is 2 instances which are sharing
> the /config /data and /log directries together with the node name?

You can run multiple instances with the same paths (including logging
and data).

If you just want to specify a different node name, then you could do so
on the command line:

./bin/elasticsearch -Des.node.name=node_1
./bin/elasticsearch -Des.node.name=node_2

If you want to change more than that, you could specify a specific
config file:

./bin/elasticsearch -Des.config=/path/to/config/file_1
./bin/elasticsearch -Des.config=/path/to/config/file_2

clint


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


12