elasticsearch hangs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

elasticsearch hangs

Weiwei Wang
hi,all,
    I have 4 es node with 32 shard 1 replica, each with 8G memory. my elasticsearch version is elasticsearch-0.18.5.
I start each node with command:
nohup bin/elasticsearch   -Xms8g -Xmx8g -Des.max-open-files=true -Dbootstrap.mlockall=true -p es.pid

and current the index size is:
size: 35.3gb (70.7gb)
docs: 15938184 (17506608)

from elasticsearch-head i noticed that:
jvm{
  • pid17608
  • version1.6.0_13
  • vm_nameJava HotSpot(TM) 64-Bit Server VM
  • vm_version11.3-b02
  • vm_vendorSun Microsystems Inc.
  • start_time1337151263158
  • mem: {
    • heap_init2gb
    • heap_init_in_bytes2147483648
    • heap_max5.3gb
    • heap_max_in_bytes5726666752
    • non_heap_init130.4mb
    • non_heap_init_in_bytes136773632
    • non_heap_max560mb
    • non_heap_max_in_bytes587202560
    }
}

from top command:
21003 work      18   0 8879m 8.3g 9796 S 257.0 13.2  48:45.98 java                                      

21003 is elasticsearch's pid and we can see that cpu usage is 257.0% and keeps around that, some time even 300+%.

every one month I have to restarted all my es nodes otherwise  it will responses very slow and dumps a *.hprof file. 

from the index size we can see that we have only 30G around index(70 total with replica) and each node has 8G memeory, but es still performs very poorly.

how should i do to achieve better performance and avoid restarting es nodes(which is very frustrating)


Reply | Threaded
Open this post in threaded view
|

Re: elasticsearch hangs

Daniel Schnell
It might be worthwhile to upgrade to the latest version 0.19.7. We had a very similar problem with 0.19.3 & 0.19.4 and upgrading to 0.19.7 fixed this issue for us. Take a look at the upgrade notes, concerning upgrading from 0.18.X versions.


Regards,

Daniel.



Am 28.06.2012 um 11:37 schrieb Weiwei Wang:

hi,all,
    I have 4 es node with 32 shard 1 replica, each with 8G memory. my elasticsearch version is elasticsearch-0.18.5.
I start each node with command:
nohup bin/elasticsearch   -Xms8g -Xmx8g -Des.max-open-files=true -Dbootstrap.mlockall=true -p es.pid

and current the index size is:
size: 35.3gb (70.7gb)
docs: 15938184 (17506608)

from elasticsearch-head i noticed that:
jvm{
  • pid17608
  • version1.6.0_13
  • vm_nameJava HotSpot(TM) 64-Bit Server VM
  • vm_version11.3-b02
  • vm_vendorSun Microsystems Inc.
  • start_time1337151263158
  • mem: {
    • heap_init2gb
    • heap_init_in_bytes2147483648
    • heap_max5.3gb
    • heap_max_in_bytes5726666752
    • non_heap_init130.4mb
    • non_heap_init_in_bytes136773632
    • non_heap_max560mb
    • non_heap_max_in_bytes587202560
    }
}

from top command:
21003 work      18   0 8879m 8.3g 9796 S 257.0 13.2  48:45.98 java                                      

21003 is elasticsearch's pid and we can see that cpu usage is 257.0% and keeps around that, some time even 300+%.

every one month I have to restarted all my es nodes otherwise  it will responses very slow and dumps a *.hprof file. 

from the index size we can see that we have only 30G around index(70 total with replica) and each node has 8G memeory, but es still performs very poorly.

how should i do to achieve better performance and avoid restarting es nodes(which is very frustrating)



Reply | Threaded
Open this post in threaded view
|

Re: elasticsearch hangs

Weiwei Wang
thanks daniel, i'm planning to do that. I'm testing my client which use client lib's TransportClient to connect to es cluster and find that client lib also need to upgrade, so I need to do a full test because we use it in our online product.

On Thursday, June 28, 2012 7:20:53 PM UTC+8, Daniel Schnell wrote:
It might be worthwhile to upgrade to the latest version 0.19.7. We had a very similar problem with 0.19.3 & 0.19.4 and upgrading to 0.19.7 fixed this issue for us. Take a look at the upgrade notes, concerning upgrading from 0.18.X versions.


Regards,

Daniel.



Am 28.06.2012 um 11:37 schrieb Weiwei Wang:

hi,all,
    I have 4 es node with 32 shard 1 replica, each with 8G memory. my elasticsearch version is elasticsearch-0.18.5.
I start each node with command:
nohup bin/elasticsearch   -Xms8g -Xmx8g -Des.max-open-files=true -Dbootstrap.mlockall=true -p es.pid

and current the index size is:
size: 35.3gb (70.7gb)
docs: 15938184 (17506608)

from elasticsearch-head i noticed that:
jvm{
  • pid17608
  • version1.6.0_13
  • vm_nameJava HotSpot(TM) 64-Bit Server VM
  • vm_version11.3-b02
  • vm_vendorSun Microsystems Inc.
  • start_time1337151263158
  • mem: {
    • heap_init2gb
    • heap_init_in_bytes2147483648
    • heap_max5.3gb
    • heap_max_in_bytes5726666752
    • non_heap_init130.4mb
    • non_heap_init_in_bytes136773632
    • non_heap_max560mb
    • non_heap_max_in_bytes587202560
    }
}

from top command:
21003 work      18   0 8879m 8.3g 9796 S 257.0 13.2  48:45.98 java                                      

21003 is elasticsearch's pid and we can see that cpu usage is 257.0% and keeps around that, some time even 300+%.

every one month I have to restarted all my es nodes otherwise  it will responses very slow and dumps a *.hprof file. 

from the index size we can see that we have only 30G around index(70 total with replica) and each node has 8G memeory, but es still performs very poorly.

how should i do to achieve better performance and avoid restarting es nodes(which is very frustrating)