[ANN] SegmentSpy - Site Plugin to watch segments in realtime

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] SegmentSpy - Site Plugin to watch segments in realtime

Zachary Tong
A while ago, Lukas showed me an article titled "Visualizing Lucene's Segment Merges", written by Mike McCandless on his blog.  Since reading that article I've been enamored with the segment visualizations that he created to demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the Index Segments API, so I created a fairly simple site plugin to graph this data: ElasticSearch-SegmentSpy

The plugin is still very rough around the edges, and has only been tested on 0.20.2 with Firefox/Chrome.  Let me know if you run into any errors.  In Mike's article, he graphed segments by physical size, while my plugin graphs by doc count.  ES provides the physical size of a segment but it doesn't specify how big the deleted_docs section is (just number of deleted docs).  ES also doesn't expose which segments are merging, so I can't color actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is fun to watch segments merge as you index/delete from your data.  =)
-Zach


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] SegmentSpy - Site Plugin to watch segments in realtime

Michael McCandless
Very cool!

For the deleted docs in the videos I think I just pro-rated according to %tg deleted docs.



On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong <[hidden email]> wrote:
A while ago, Lukas showed me an article titled "Visualizing Lucene's Segment Merges", written by Mike McCandless on his blog.  Since reading that article I've been enamored with the segment visualizations that he created to demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the Index Segments API, so I created a fairly simple site plugin to graph this data: ElasticSearch-SegmentSpy

The plugin is still very rough around the edges, and has only been tested on 0.20.2 with Firefox/Chrome.  Let me know if you run into any errors.  In Mike's article, he graphed segments by physical size, while my plugin graphs by doc count.  ES provides the physical size of a segment but it doesn't specify how big the deleted_docs section is (just number of deleted docs).  ES also doesn't expose which segments are merging, so I can't color actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is fun to watch segments merge as you index/delete from your data.  =)
-Zach


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] SegmentSpy - Site Plugin to watch segments in realtime

Zachary Tong
Ahh, that makes sense.  I'll make some changes when I get a chance to account for that...I'd be happier if the segments represented size instead of docs. =)

Thanks for taking a look at the plugin!
-Zach



On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote:
Very cool!

For the deleted docs in the videos I think I just pro-rated according to %tg deleted docs.



On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="pu7dR3utFj0J">zachar...@...> wrote:
A while ago, Lukas showed me an article titled "Visualizing Lucene's Segment Merges", written by Mike McCandless on his blog.  Since reading that article I've been enamored with the segment visualizations that he created to demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the Index Segments API, so I created a fairly simple site plugin to graph this data: ElasticSearch-SegmentSpy

The plugin is still very rough around the edges, and has only been tested on 0.20.2 with Firefox/Chrome.  Let me know if you run into any errors.  In Mike's article, he graphed segments by physical size, while my plugin graphs by doc count.  ES provides the physical size of a segment but it doesn't specify how big the deleted_docs section is (just number of deleted docs).  ES also doesn't expose which segments are merging, so I can't color actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is fun to watch segments merge as you index/delete from your data.  =)
-Zach


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="pu7dR3utFj0J">elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] SegmentSpy - Site Plugin to watch segments in realtime

kimchy
Administrator
Very nice!. We will check if we can add a flag to the segments API to indicate that its begin merged. Can you open an issue?

On Feb 4, 2013, at 1:51 PM, Zachary Tong <[hidden email]> wrote:

Ahh, that makes sense.  I'll make some changes when I get a chance to account for that...I'd be happier if the segments represented size instead of docs. =)

Thanks for taking a look at the plugin!
-Zach



On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote:
Very cool!

For the deleted docs in the videos I think I just pro-rated according to %tg deleted docs.



On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="pu7dR3utFj0J">zachar...@...> wrote:
A while ago, Lukas showed me an article titled "Visualizing Lucene's Segment Merges", written by Mike McCandless on his blog.  Since reading that article I've been enamored with the segment visualizations that he created to demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the Index Segments API, so I created a fairly simple site plugin to graph this data: ElasticSearch-SegmentSpy

The plugin is still very rough around the edges, and has only been tested on 0.20.2 with Firefox/Chrome.  Let me know if you run into any errors.  In Mike's article, he graphed segments by physical size, while my plugin graphs by doc count.  ES provides the physical size of a segment but it doesn't specify how big the deleted_docs section is (just number of deleted docs).  ES also doesn't expose which segments are merging, so I can't color actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is fun to watch segments merge as you index/delete from your data.  =)
-Zach



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="pu7dR3utFj0J">elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] SegmentSpy - Site Plugin to watch segments in realtime

Paul Smith
I've really enjoyed this plugin, it makes watching a large Bulk load interesting.  Even with our simple 3-node, 5 shard cluster though one can't see the 'whole picture'. This is where a 3-D grid column chart would look very cool.

Each Index could be a grid, with extra z-axis used to display the other shards.  So X-axis is each segment as it is now, and the Y-axis is the logarithmic size as it is now, but then plot each other shard via the z-axis.  This way one chart for each index can be seen on one page.

Obviously your JS chart library may not have that chart feature though, but I think it would look awesome, and be more practical for even relatively large indices?

I tried mocking this up in Excel but sadly it thinks it's too complex to display even just 2 series with 5 rows.. Stupid Excel... ?  something looking like this anyway: http://www.agocg.ac.uk/reports/graphics/34/appii97/image18.gif as an example.

cheers,

Paul


On 13 February 2013 09:29, <[hidden email]> wrote:
Very nice!. We will check if we can add a flag to the segments API to indicate that its begin merged. Can you open an issue?

On Feb 4, 2013, at 1:51 PM, Zachary Tong <[hidden email]> wrote:

Ahh, that makes sense.  I'll make some changes when I get a chance to account for that...I'd be happier if the segments represented size instead of docs. =)

Thanks for taking a look at the plugin!
-Zach



On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote:
Very cool!

For the deleted docs in the videos I think I just pro-rated according to %tg deleted docs.



On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong <[hidden email]> wrote:
A while ago, Lukas showed me an article titled "Visualizing Lucene's Segment Merges", written by Mike McCandless on his blog.  Since reading that article I've been enamored with the segment visualizations that he created to demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the Index Segments API, so I created a fairly simple site plugin to graph this data: ElasticSearch-SegmentSpy

The plugin is still very rough around the edges, and has only been tested on 0.20.2 with Firefox/Chrome.  Let me know if you run into any errors.  In Mike's article, he graphed segments by physical size, while my plugin graphs by doc count.  ES provides the physical size of a segment but it doesn't specify how big the deleted_docs section is (just number of deleted docs).  ES also doesn't expose which segments are merging, so I can't color actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is fun to watch segments merge as you index/delete from your data.  =)
-Zach



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] SegmentSpy - Site Plugin to watch segments in realtime

Zachary Tong
In reply to this post by kimchy
Very nice!. We will check if we can add a flag to the segments API to indicate that its begin merged. Can you open an issue?
Thanks!  I'll go open an issue right now...that'd be a cool feature to include in the plugin.

Even with our simple 3-node, 5 shard cluster though one can't see the 'whole picture'. This is where a 3-D grid column chart would look very cool.
 Yeah, I totally agree.  My cluster has 3 nodes as well, and it quickly becomes unmanageable to watch all the indices.  I experimented with making all the graphs smaller (ala BigDesk), and it sorta works, but is still difficult to get everything on screen at a reasonable resolution.

I'll play around with 3D charts, that's an excellent idea.  Karmi has convinced me to switch to D3, so more advanced visualizations should definitely be possible.

Thanks for the comments!

-Zach



On Tuesday, February 12, 2013 5:29:40 PM UTC-5, kimchy wrote:
Very nice!. We will check if we can add a flag to the segments API to indicate that its begin merged. Can you open an issue?

On Feb 4, 2013, at 1:51 PM, Zachary Tong <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="hAF0e9aP8Q8J">zachar...@...> wrote:

Ahh, that makes sense.  I'll make some changes when I get a chance to account for that...I'd be happier if the segments represented size instead of docs. =)

Thanks for taking a look at the plugin!
-Zach



On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote:
Very cool!

For the deleted docs in the videos I think I just pro-rated according to %tg deleted docs.



On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong <[hidden email]> wrote:
A while ago, Lukas showed me an article titled "Visualizing Lucene's Segment Merges", written by Mike McCandless on his blog.  Since reading that article I've been enamored with the segment visualizations that he created to demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the Index Segments API, so I created a fairly simple site plugin to graph this data: ElasticSearch-SegmentSpy

The plugin is still very rough around the edges, and has only been tested on 0.20.2 with Firefox/Chrome.  Let me know if you run into any errors.  In Mike's article, he graphed segments by physical size, while my plugin graphs by doc count.  ES provides the physical size of a segment but it doesn't specify how big the deleted_docs section is (just number of deleted docs).  ES also doesn't expose which segments are merging, so I can't color actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is fun to watch segments merge as you index/delete from your data.  =)
-Zach



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="hAF0e9aP8Q8J">elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.