Same query, different CPU util when run with Java API versus REST

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Same query, different CPU util when run with Java API versus REST

Jeff Potts
I have two JMeter tests. Let's call them "Straight Elasticsearch Test" and "Service Test". Both tests use the same settings in terms of number of threads (users), number of times the URL is invoked (loops), and randomized CSV-based data set.

The difference between the two tests is that the Straight Elasticsearch Test just makes HTTP requests against my cluster whereas the Service Test makes HTTP requests against my Java service layer.

Both tests invoke the same Elasticsearch query. I've verified this by comparing what the Straight Elasticsearch Test posts to what the Java service creates using the SearchRequestBuilder. Here is the query and the search invocation via the Java API:

https://gist.github.com/jpotts/feaf0cedc2eea29a8aa7

I'm running both tests with 10 concurrent users and 2000 iterations for a total of 20,000 hits against either Elasticsearch directly or the Java service depending on the test.

Here's the puzzling thing:
The Straight Elasticsearch Test causes the CPU on my single-node cluster to go to around 50% for the duration of the test. But the Service Test causes it to go to twice that.

So the same number of concurrent users, the same number of total hits, and the same query are causing drastically different CPU utilization rates.

What could be causing this?

I've tested with the Java service using the Transport Client as well as the Node Client, the results are the same.

I'm thinking it must be something different in how the server handles HTTP request versus Node/Transport Client requests.

This is Elasticsearch 1.3.4 and the index is a single shard with no replicas and about 100k docs.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/63fd9d8e-ec67-4c6c-817f-4201d5f18ea9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Same query, different CPU util when run with Java API versus REST

Jeff Potts
I should mention that the Elasticsearch node, the Java service, and the JMeter test client are all on different machines.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Same query, different CPU util when run with Java API versus REST

joergprante@gmail.com
Can you post the full query code for better recreation?

Jörg

On Fri, Dec 12, 2014 at 6:44 PM, Jeff Potts <[hidden email]> wrote:
I should mention that the Elasticsearch node, the Java service, and the JMeter test client are all on different machines.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvO8iOu0%3DqfmNegSuB2Pv%3DD8y2pQuXem9HUj4%2BV6vO1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Same query, different CPU util when run with Java API versus REST

Jeff Potts
Yes, updated the gist. Thanks for taking a look at this.

Jeff

On Sunday, December 14, 2014 11:52:35 AM UTC-6, Jörg Prante wrote:
Can you post the full query code for better recreation?

Jörg

On Fri, Dec 12, 2014 at 6:44 PM, Jeff Potts <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="05BlNpWX-lYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">jeffp...@...> wrote:
I should mention that the Elasticsearch node, the Java service, and the JMeter test client are all on different machines.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="05BlNpWX-lYJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c312478a-e5c5-49aa-abd8-88eeac2c2051%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Same query, different CPU util when run with Java API versus REST

Marie Jacob
I have this same issue, except that CPU utilization is approximately the same, but response times are very different. I'm using ES 1.3.4. I have two JMeter tests simulating concurrent tests with the exact same configuration. The test using the Java Sampler with the Transport Client are showing response times that are twice as much as the test with HTTP samplers. Are the connection channels in the ES client maybe overloaded? 

Any insight on this would definitely be great.

Thanks,
Marie.
 

On Monday, December 15, 2014 9:27:18 AM UTC-5, Jeff Potts wrote:
Yes, updated the gist. Thanks for taking a look at this.

Jeff

On Sunday, December 14, 2014 11:52:35 AM UTC-6, Jörg Prante wrote:
Can you post the full query code for better recreation?

Jörg

On Fri, Dec 12, 2014 at 6:44 PM, Jeff Potts <[hidden email]> wrote:
I should mention that the Elasticsearch node, the Java service, and the JMeter test client are all on different machines.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7ff35b47-9e32-4f85-8899-2bbef4e632e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Same query, different CPU util when run with Java API versus REST

InquiringMind
In reply to this post by Jeff Potts
Jeff,

Does getReadClient() get a reference to a previously created singleton TransportClient (or NodeClient, as the case may be)? I am guessing yes, but just asking to be sure.

In my own set-up, I have created a thin HTTP REST layer (also using Netty, with the LMAX Disruptor to minimize thread usage while maximizing throughput). It contains our business logic, and issues queries and updates to Elasticsearch via the TransportClient. It creates a singleton instance of the TransportClient, and shares this singleton's reference throughout the service. Using seige, I have hammered the combination of my thin server plus Elasticsearch and don't see any performance issues as you describe.

Just a thought...

Regards,
Brian

On Monday, December 15, 2014 9:27:18 AM UTC-5, Jeff Potts wrote:
Yes, updated the gist. Thanks for taking a look at this.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e0f039df-95fc-4fb9-a787-84ee56d77e84%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Same query, different CPU util when run with Java API versus REST

Jeff Potts
Yes, getReadClient() gets a Node Client that is instantiated by Spring and then injected as a dependency. I have tried the Transport Client as well and it makes no difference.

An interesting finding is that I have isolated the performance degradation to the Range filter. When my service has these filters the test using REST causes the ES node to use 9% CPU while the Java service uses 22% CPU:

filtersToApply.add(FilterBuilders.termFilter(Constants.PROP_PAGE_ID, pageId));
filtersToApply.add(FilterBuilders.termFilter(Constants.PROP_GEO_CODE, meta.getGeoCode()));
filtersToApply.add(FilterBuilders.termFilter(Constants.PROP_PLACEMENT_ID, meta.getPlacementId()));

When I add a date range, the difference in performance increases dramatically with the REST call staying at around 9% and the Java service driving the CPU util to 81%:

filtersToApply.add(FilterBuilders.rangeFilter(Constants.PROP_PUB_DATE).lte(curDate.getTime()));

I've also noted that while the index search shard query rate is identical between the two tests, the total shard search query time is dramatically different with the Java service driving a much higher query time compared to the REST-based test.

Jeff

On Tuesday, December 16, 2014 2:38:51 PM UTC-6, Brian wrote:
Jeff,

Does getReadClient() get a reference to a previously created singleton TransportClient (or NodeClient, as the case may be)? I am guessing yes, but just asking to be sure.

In my own set-up, I have created a thin HTTP REST layer (also using Netty, with the LMAX Disruptor to minimize thread usage while maximizing throughput). It contains our business logic, and issues queries and updates to Elasticsearch via the TransportClient. It creates a singleton instance of the TransportClient, and shares this singleton's reference throughout the service. Using seige, I have hammered the combination of my thin server plus Elasticsearch and don't see any performance issues as you describe.

Just a thought...

Regards,
Brian

On Monday, December 15, 2014 9:27:18 AM UTC-5, Jeff Potts wrote:
Yes, updated the gist. Thanks for taking a look at this.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dd88c5a7-28ec-4aa2-ad1f-2974a5ed1978%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Same query, different CPU util when run with Java API versus REST

Jeff Potts
Scratch what I said about total shard search query time. I had started a client node and was pointing my REST test at that.

CPU difference still there.

Also, I've upgraded to 1.3.7.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e429e400-2be3-4481-8218-c3a0e3d76308%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Same query, different CPU util when run with Java API versus REST

joergprante@gmail.com
In reply to this post by Jeff Potts
The difference between DSL and Java is that in Java you use search type QUERY_AND_FETCH which is slow.

Jörg

On Mon, Dec 15, 2014 at 3:27 PM, Jeff Potts <[hidden email]> wrote:
Yes, updated the gist. Thanks for taking a look at this.

Jeff

On Sunday, December 14, 2014 11:52:35 AM UTC-6, Jörg Prante wrote:
Can you post the full query code for better recreation?

Jörg

On Fri, Dec 12, 2014 at 6:44 PM, Jeff Potts <[hidden email]> wrote:
I should mention that the Elasticsearch node, the Java service, and the JMeter test client are all on different machines.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c312478a-e5c5-49aa-abd8-88eeac2c2051%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGE8DGWF-b1o9StQkY9dSFDDLvtyXGifAGtAx%2BC7Jx96Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Same query, different CPU util when run with Java API versus REST

Jeff Potts
I figured it out. The queries executed by the straight Elasticsearch REST test and the Java test are not exactly identical. The difference is that the "current date" used by the straight Elasticsearch test is pulled from a randomized CSV file whereas the Java service always injects the actual current date down to the millisecond. So, even though I switched the query and the Java to leverage filters, the straight Elasticsearch REST test is the only test that takes advantage of the cache. The Java service query doesn't take advantage of the cache because the current date changes every millisecond.

To confirm this I hardcoded the milliseconds in both the REST test and the Java service. Once I did that, the CPU rates (and everything else) were identical.

The other telltale sign that there was a problem was the indices filter cache graph in Marvel. Once straight REST test ran once, it could be re-run and never cause an increase in what is in the filter cache. The Java service, on the other hand, was populating the filter cache every time.

Thanks to everyone that weighed in on this.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4a3e2981-7a3b-4e6e-a2f0-ed65fddc9150%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.