JMX

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

JMX

Nicolas Labrot
Hello,

JMX is a common way to monitor a lot of Java Apps and a lot a monitor apps have JMX connector. I read on this group that Shay does not recommend JMX for monitoring ES. Stats API is prefered. Is there any reason for that ?

I know JMX has a cost. But with low frequency queries, it become unnoticeable.

Regards,

Nicolas


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: JMX

Otis Gospodnetic
Hi,

Not only is JMX not recommended, but it's been completely removed from ES as far as I understand.  Our SPM tool for monitoring ES uses the stats API.

Otis
--
ELASTICSEARCH Performance Monitoring - http://sematext.com/spm
Search Analytics - http://sematext.com/search-analytics


On Friday, May 24, 2013 5:23:38 AM UTC-4, Nicolas Labrot wrote:
Hello,

JMX is a common way to monitor a lot of Java Apps and a lot a monitor apps have JMX connector. I read on this group that Shay does not recommend JMX for monitoring ES. Stats API is prefered. Is there any reason for that ?

I know JMX has a cost. But with low frequency queries, it become unnoticeable.

Regards,

Nicolas


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: JMX

Nicolas Labrot
But any idea why JMX is not recommanded?

Your SPM tool seems great, but ops will see it as another tool to monitor just one app.


Le vendredi 24 mai 2013 18:15:27 UTC+2, Otis Gospodnetic a écrit :
Hi,

Not only is JMX not recommended, but it's been completely removed from ES as far as I understand.  Our SPM tool for monitoring ES uses the stats API.

Otis
--
ELASTICSEARCH Performance Monitoring - http://sematext.com/spm
Search Analytics - http://sematext.com/search-analytics


On Friday, May 24, 2013 5:23:38 AM UTC-4, Nicolas Labrot wrote:
Hello,

JMX is a common way to monitor a lot of Java Apps and a lot a monitor apps have JMX connector. I read on this group that Shay does not recommend JMX for monitoring ES. Stats API is prefered. Is there any reason for that ?

I know JMX has a cost. But with low frequency queries, it become unnoticeable.

Regards,

Nicolas


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: JMX

Dan Everton
The JSON metrics API exported by Elasticsearch is far simpler to get data out of than JMX for a lot of monitoring systems. Talking to JMX generally requires a Java based agent for the metrics gathering system or using something like jmxtrans to get the data out.

Regardless, Elasticsearch has only removed support for creating its own JMX connector. You can still use the JVM command line flags like this

  -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.port=9400
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false

To have the JVM create a remote connector for JMX access. If you add these to the JAVA_OPTS environment variable before invoking Elasticsearch they'll get automatically added to the Elasticsearch command line. It's essentially equivalent to what you got before with jmx.create_connector: true

Cheers,
Dan



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: JMX

Otis Gospodnetic
In reply to this post by Nicolas Labrot
Hi,

On Fri, May 24, 2013 at 1:04 PM, Nicolas Labrot <[hidden email]> wrote:
> But any idea why JMX is not recommanded?

I suspect it was removed because it meant more maintenance work, and
metrics can be retrieved via HTTP API.

> Your SPM tool seems great, but ops will see it as another tool to monitor
> just one app.

Ah, but that's not really true.  SPM monitors not just ElasticSearch,
but also Hadoop, HBase, Solr, Sensei, JVM, and servers.

Tell me what else you want SPM to monitor?
https://docs.google.com/a/sematext.com/spreadsheet/viewform?formkey=dFlVbUNxOHR6UFlQem5XeGIzTjV6Qmc6MQ

Thanks,
Otis
--
Solr & ElasticSearch Support
http://sematext.com/



> Le vendredi 24 mai 2013 18:15:27 UTC+2, Otis Gospodnetic a écrit :
>>
>> Hi,
>>
>> Not only is JMX not recommended, but it's been completely removed from ES
>> as far as I understand.  Our SPM tool for monitoring ES uses the stats API.
>>
>> Otis
>> --
>> ELASTICSEARCH Performance Monitoring - http://sematext.com/spm
>> Search Analytics - http://sematext.com/search-analytics
>>
>>
>> On Friday, May 24, 2013 5:23:38 AM UTC-4, Nicolas Labrot wrote:
>>>
>>> Hello,
>>>
>>> JMX is a common way to monitor a lot of Java Apps and a lot a monitor
>>> apps have JMX connector. I read on this group that Shay does not recommend
>>> JMX for monitoring ES. Stats API is prefered. Is there any reason for that ?
>>>
>>> I know JMX has a cost. But with low frequency queries, it become
>>> unnoticeable.
>>>
>>> Regards,
>>>
>>> Nicolas
>>>
>>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/tUBRE0UotCU/unsubscribe?hl=en-US.
> To unsubscribe from this group and all its topics, send an email to
> [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: JMX

Nicolas Labrot
In reply to this post by Dan Everton

Hello Dan,

Le lundi 27 mai 2013 01:01:05 UTC+2, Dan Everton a écrit :
The JSON metrics API exported by Elasticsearch is far simpler to get data out of than JMX for a lot of monitoring systems. Talking to JMX generally requires a Java based agent for the metrics gathering system or using something like jmxtrans to get the data out.

I agree the JSON API is simpler to query. But this is a specific API which need custom connector. JMX is de facto a standard for monitoring Java apps. There are many tools/connectors/monitoring apps that can query mbeans.

 

Regardless, Elasticsearch has only removed support for creating its own JMX connector. You can still use the JVM command line flags like this

  -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.port=9400
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false

To have the JVM create a remote connector for JMX access. If you add these to the JAVA_OPTS environment variable before invoking Elasticsearch they'll get automatically added to the Elasticsearch command line. It's essentially equivalent to what you got before with jmx.create_connector: true


My message is about ES mbeans which have been removed.


Regards,

Nicolas

 

Cheers,
Dan



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: JMX

Dan Everton

My message is about ES mbeans which have been removed.

I missed that the actual MBeans had been removed along with the connector. That is a bit tricky to deal with if you need to stick with JMX and can't adjust your monitoring tools to consume JSON. One option would be to write a plugin for Elasticsearch that contains the MBeans for the metrics you need and just wraps the existing statistics.

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: JMX

Nicolas Labrot






Le mardi 28 mai 2013 00:55:51 UTC+2, Dan Everton a écrit :

My message is about ES mbeans which have been removed.

I missed that the actual MBeans had been removed along with the connector. That is a bit tricky to deal with if you need to stick with JMX and can't adjust your monitoring tools to consume JSON. One option would be to write a plugin for Elasticsearch that contains the MBeans for the metrics you need and just wraps the existing statistics.

Absolutely, a plugin should not be so hard to do.

 

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: JMX

joergprante@gmail.com
Have you started work on an JMX plugin?

Note, the overhead comes form RMI. As long as JMX is not running, it does not matter, but in the long run, the RMI and GC overhead are significant, as still noted in http://www.elasticsearch.org/guide/reference/modules/jmx/

There is a glitch which makes a JMX plugin implementation not very handy. Either you add a static set of MBeans. Or a dynamic one, but for this, you have to add a JMX MBeans annotation jar to the main ES classpath in $ES_HOME/lib (JSR 255 like, for an example https://github.com/wjw465150/jmx-annotations).  Otherwise, classes annotated for being MBeans won't get recognized at ES node start time by the Guice plugin initiator.

Jörg

On Tuesday, May 28, 2013 9:11:22 AM UTC+2, Nicolas Labrot wrote:






Le mardi 28 mai 2013 00:55:51 UTC+2, Dan Everton a écrit :

My message is about ES mbeans which have been removed.

I missed that the actual MBeans had been removed along with the connector. That is a bit tricky to deal with if you need to stick with JMX and can't adjust your monitoring tools to consume JSON. One option would be to write a plugin for Elasticsearch that contains the MBeans for the metrics you need and just wraps the existing statistics.

Absolutely, a plugin should not be so hard to do.

 

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: JMX

kimchy
Administrator
I guess people know here I am not a fan of JMX, specifically because of the problem to present complex data structures (Ike size per shard on a node) and not being able to easily display stats across nodes (like the size of an index and all its shards across a cluster). also, it does add an annoying overhead. 

always thought that stats in Java products tended to be quite useless because of the limiting factors of JMX...  (either in interaction or exposing it)

we removed the auto connector support and the few mbeans we had (they didnt really expose that much data). the effort involved in trying to support the breadth of stats ES supports would not have even been challenging using JMX, but basically impossible... 


On Fri, May 31, 2013 at 4:49 PM, Jörg Prante <[hidden email]> wrote:

Have you started work on an JMX plugin?


Note, the overhead comes form RMI. As long as JMX is not running, it does not matter, but in the long run, the RMI and GC overhead are significant, as still noted in http://www.elasticsearch.org/guide/reference/modules/jmx/

There is a glitch which makes a JMX plugin implementation not very handy. Either you add a static set of MBeans. Or a dynamic one, but for this, you have to add a JMX MBeans annotation jar to the main ES classpath in $ES_HOME/lib (JSR 255 like, for an example https://github.com/wjw465150/jmx-annotations).  Otherwise, classes annotated for being MBeans won't get recognized at ES node start time by the Guice plugin initiator.

Jörg

On Tuesday, May 28, 2013 9:11:22 AM UTC+2, Nicolas Labrot wrote:






Le mardi 28 mai 2013 00:55:51 UTC+2, Dan Everton a écrit :

My message is about ES mbeans which have been removed.

I missed that the actual MBeans had been removed along with the connector. That is a bit tricky to deal with if you need to stick with JMX and can't adjust your monitoring tools to consume JSON. One option would be to write a plugin for Elasticsearch that contains the MBeans for the metrics you need and just wraps the existing statistics.

Absolutely, a plugin should not be so hard to do.

 

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: JMX

Nicolas Labrot
Thanks for your reply.

@Jörg: I will choose the shortest path  and JMX seems to be actually not the shortest one. A logstash input plugin may be more appropriate


On Sat, Jun 1, 2013 at 3:14 AM, Shay Banon <[hidden email]> wrote:
I guess people know here I am not a fan of JMX, specifically because of the problem to present complex data structures (Ike size per shard on a node) and not being able to easily display stats across nodes (like the size of an index and all its shards across a cluster). also, it does add an annoying overhead. 

always thought that stats in Java products tended to be quite useless because of the limiting factors of JMX...  (either in interaction or exposing it)

we removed the auto connector support and the few mbeans we had (they didnt really expose that much data). the effort involved in trying to support the breadth of stats ES supports would not have even been challenging using JMX, but basically impossible... 


On Fri, May 31, 2013 at 4:49 PM, Jörg Prante <[hidden email]> wrote:

Have you started work on an JMX plugin?


Note, the overhead comes form RMI. As long as JMX is not running, it does not matter, but in the long run, the RMI and GC overhead are significant, as still noted in http://www.elasticsearch.org/guide/reference/modules/jmx/

There is a glitch which makes a JMX plugin implementation not very handy. Either you add a static set of MBeans. Or a dynamic one, but for this, you have to add a JMX MBeans annotation jar to the main ES classpath in $ES_HOME/lib (JSR 255 like, for an example https://github.com/wjw465150/jmx-annotations).  Otherwise, classes annotated for being MBeans won't get recognized at ES node start time by the Guice plugin initiator.

Jörg

On Tuesday, May 28, 2013 9:11:22 AM UTC+2, Nicolas Labrot wrote:






Le mardi 28 mai 2013 00:55:51 UTC+2, Dan Everton a écrit :

My message is about ES mbeans which have been removed.

I missed that the actual MBeans had been removed along with the connector. That is a bit tricky to deal with if you need to stick with JMX and can't adjust your monitoring tools to consume JSON. One option would be to write a plugin for Elasticsearch that contains the MBeans for the metrics you need and just wraps the existing statistics.

Absolutely, a plugin should not be so hard to do.

 

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: JMX

Charles Moulliard
In reply to this post by Nicolas Labrot
I have done a test using last snapshot 1.x but that does not work 

Add to the elasticsearch script -->

JAVA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9400 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

After starting jconsole and connect to elasticsearch bootstrap, then no ES MBeans appear

I see from the code that we have commented those lines in Bootstrap. This is perhaps the explanation

    private static Tuple<Settings, Environment> setupJmx(Tuple<Settings, Environment> tuple) {
        // We disable JMX on by default, since we don't really want the overhead of RMI (and RMI GC...)
//        if (tuple.v1().get(JmxService.SettingsConstants.CREATE_CONNECTOR) == null) {
//            // automatically create the connector if we are bootstrapping
//            Settings updated = settingsBuilder().put(tuple.v1()).put(JmxService.SettingsConstants.CREATE_CONNECTOR, true).build();
//            tuple = new Tuple<Settings, Environment>(updated, tuple.v2());
//        }
        return tuple;
    }



On Friday, May 24, 2013 11:23:38 AM UTC+2, Nicolas Labrot wrote:
Hello,

JMX is a common way to monitor a lot of Java Apps and a lot a monitor apps have JMX connector. I read on this group that Shay does not recommend JMX for monitoring ES. Stats API is prefered. Is there any reason for that ?

I know JMX has a cost. But with low frequency queries, it become unnoticeable.

Regards,

Nicolas


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.