Lots of segments per index

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Lots of segments per index

mike.giardinelli
Hi All,

We have monthly indexes (currently 11 months) with 22 shards per index.  I am seeing what seems to be a lot of segments per index (roughly 1200 to 1380 segments).  The older indexes should have very little, if any updates occurring on them.  From everything I have read it sounds like ES should be automatically merging segments, but now I am a bit concerned that this is not occurring.  I know we can manually run an optimize, but will need to allocate another resource to do that work (as to not impact current system).  I am fairly new to ES (if that isn't obvious) and am really trying to understand we have an issue or not. 

Rough index stats: 

11 indexes
22 shards per index
65 million docs per index
350 GB per index

Any info, suggestions, etc. are greatly appreciated. 

Thanks!

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c6bedad4-9c24-4e92-94ef-c4fc0e51a7ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Lots of segments per index

Nikolas Everett

It merges segments in response to indexes and updates so an index that doesn't change will not have merges. You can manually optimize the index once, when it is mostly done with updates. Once the index is optimized further calls to optimize with the same parameters are noops.

You can't really ads more resources to optimize though, its something done on each shard.

On May 20, 2015 7:23 PM, <[hidden email]> wrote:
Hi All,

We have monthly indexes (currently 11 months) with 22 shards per index.  I am seeing what seems to be a lot of segments per index (roughly 1200 to 1380 segments).  The older indexes should have very little, if any updates occurring on them.  From everything I have read it sounds like ES should be automatically merging segments, but now I am a bit concerned that this is not occurring.  I know we can manually run an optimize, but will need to allocate another resource to do that work (as to not impact current system).  I am fairly new to ES (if that isn't obvious) and am really trying to understand we have an issue or not. 

Rough index stats: 

11 indexes
22 shards per index
65 million docs per index
350 GB per index

Any info, suggestions, etc. are greatly appreciated. 

Thanks!

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c6bedad4-9c24-4e92-94ef-c4fc0e51a7ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0QO%3DQ2T4c6-tZQH6x7YjW3eLZEeF1K_Zz3XSP-dTK7HQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.