How to improve search performance in ES ?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

How to improve search performance in ES ?

dark_shadow
Hi,

I'm working on improving the search response of ES but not able to do anything. My scenario is something like this:

I'm using 3 ES queries to get relevant results for my autocompleter.

1. A function score query with a match query  ( To get a correct match if user typed query is available in documents based on popularity)

2. A multi match query  (To handle those scenarios in which a user types some text which is present in different fields in a document since my documents are multi fields like name, address, city, state, country )

3. A query string (In order to ensure if I missed user query by the above type I'll be able to search using more powerful but less accurate query string)

Along with all the 3 queries, I'm using 4 filters (clubbed using AND filter).

My performance is really bad and I want to improve it along with delivering relevat results in my autocompleter.

Can anyone help me how can I improve this ? Any way I can club the queries for better performance ? 

I have read that I BOOL filters should be used instead of AND filter since they use bitset which are cached internally. I think this makes one improvement because if in the first query ES stores the information of filters in bitset, it can reuse it in other two queries. That will make the thigs a little fast but based on queries, I'm not able to do any improvement ?

Is there any way by which I can combine match and multi-match queries ( 1 and 2) into a single effective query.

Also, in place of query_string should I use some other query for faster execution.

Any suggestions are welcome. 
Thanks


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5d99495b-20ef-46b6-a069-365574fdc0a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to improve search performance in ES ?

joergprante@gmail.com


On Fri, Jul 11, 2014 at 8:11 PM, coder <[hidden email]> wrote:
Hi,

I'm working on improving the search response of ES but not able to do anything. My scenario is something like this:

I'm using 3 ES queries to get relevant results for my autocompleter.

1. A function score query with a match query  ( To get a correct match if user typed query is available in documents based on popularity)

2. A multi match query  (To handle those scenarios in which a user types some text which is present in different fields in a document since my documents are multi fields like name, address, city, state, country )

3. A query string (In order to ensure if I missed user query by the above type I'll be able to search using more powerful but less accurate query string)

Along with all the 3 queries, I'm using 4 filters (clubbed using AND filter).

My performance is really bad and I want to improve it along with delivering relevat results in my autocompleter.

Can anyone help me how can I improve this ? Any way I can club the queries for better performance ? 

I have read that I BOOL filters should be used instead of AND filter since they use bitset which are cached internally. I think this makes one improvement because if in the first query ES stores the information of filters in bitset, it can reuse it in other two queries. That will make the thigs a little fast but based on queries, I'm not able to do any improvement ?

Is there any way by which I can combine match and multi-match queries ( 1 and 2) into a single effective query.

Also, in place of query_string should I use some other query for faster execution.

Any suggestions are welcome. 
Thanks


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5d99495b-20ef-46b6-a069-365574fdc0a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBchn0AKB_heFFjr%2B%3Df1X_CzfJBGFnFQf_rEpgAiHUvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to improve search performance in ES ?

dark_shadow
Hi Jörg,

I have seen these links. I'm using ngram tokenizer. Issue which I'm facing is slow response time. For that I need some suggestions, how can I improve it ? Is there anyway by which I can query in a better way ? Also, I'm using a match query in a field in one of my filters but I have read that term filters are more effective. Can you give me some insight how can I use term filter in this case even if the field on which I want to apply the filter is not present in all the documents.

Thanks

On Saturday, 12 July 2014 00:09:50 UTC+5:30, Jörg Prante wrote:
For autocompletion, you should use the completion suggester 

<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-suggesters-completion.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsearch-suggesters-completion.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFDmSngwsefS5p-Kj8yUre86kLvDw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fsearch-suggesters-completion.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFDmSngwsefS5p-Kj8yUre86kLvDw';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-suggesters-completion.html

or edge ngram tokenizer

<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fanalysis-edgengram-tokenizer.html\46sa\75D\46sntz\0751\46usg\75AFQjCNE8yu7dpnli3ACHcka9Y3hpigFdGQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fanalysis-edgengram-tokenizer.html\46sa\75D\46sntz\0751\46usg\75AFQjCNE8yu7dpnli3ACHcka9Y3hpigFdGQ';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html

Jörg


On Fri, Jul 11, 2014 at 8:11 PM, coder <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="eeotsrgyBl8J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">mukul...@...> wrote:
Hi,

I'm working on improving the search response of ES but not able to do anything. My scenario is something like this:

I'm using 3 ES queries to get relevant results for my autocompleter.

1. A function score query with a match query  ( To get a correct match if user typed query is available in documents based on popularity)

2. A multi match query  (To handle those scenarios in which a user types some text which is present in different fields in a document since my documents are multi fields like name, address, city, state, country )

3. A query string (In order to ensure if I missed user query by the above type I'll be able to search using more powerful but less accurate query string)

Along with all the 3 queries, I'm using 4 filters (clubbed using AND filter).

My performance is really bad and I want to improve it along with delivering relevat results in my autocompleter.

Can anyone help me how can I improve this ? Any way I can club the queries for better performance ? 

I have read that I BOOL filters should be used instead of AND filter since they use bitset which are cached internally. I think this makes one improvement because if in the first query ES stores the information of filters in bitset, it can reuse it in other two queries. That will make the thigs a little fast but based on queries, I'm not able to do any improvement ?

Is there any way by which I can combine match and multi-match queries ( 1 and 2) into a single effective query.

Also, in place of query_string should I use some other query for faster execution.

Any suggestions are welcome. 
Thanks


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="eeotsrgyBl8J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/5d99495b-20ef-46b6-a069-365574fdc0a9%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/5d99495b-20ef-46b6-a069-365574fdc0a9%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/5d99495b-20ef-46b6-a069-365574fdc0a9%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/5d99495b-20ef-46b6-a069-365574fdc0a9%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/741a7bc5-ffd7-4ba7-9296-ff6fff8f559f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to improve search performance in ES ?

Mark Walkom
Can you elaborate on how you're measuring and comparing these response times and why you feel they are slow?
It might also help if you can put a sample query and document into a gist/pastebin

Also, is your cluster under load when you run these queries? What metrics are you gathering around that side?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email]
web: www.campaignmonitor.com


On 12 July 2014 05:01, coder <[hidden email]> wrote:
Hi Jörg,

I have seen these links. I'm using ngram tokenizer. Issue which I'm facing is slow response time. For that I need some suggestions, how can I improve it ? Is there anyway by which I can query in a better way ? Also, I'm using a match query in a field in one of my filters but I have read that term filters are more effective. Can you give me some insight how can I use term filter in this case even if the field on which I want to apply the filter is not present in all the documents.

Thanks


On Saturday, 12 July 2014 00:09:50 UTC+5:30, Jörg Prante wrote:


On Fri, Jul 11, 2014 at 8:11 PM, coder <[hidden email]> wrote:
Hi,

I'm working on improving the search response of ES but not able to do anything. My scenario is something like this:

I'm using 3 ES queries to get relevant results for my autocompleter.

1. A function score query with a match query  ( To get a correct match if user typed query is available in documents based on popularity)

2. A multi match query  (To handle those scenarios in which a user types some text which is present in different fields in a document since my documents are multi fields like name, address, city, state, country )

3. A query string (In order to ensure if I missed user query by the above type I'll be able to search using more powerful but less accurate query string)

Along with all the 3 queries, I'm using 4 filters (clubbed using AND filter).

My performance is really bad and I want to improve it along with delivering relevat results in my autocompleter.

Can anyone help me how can I improve this ? Any way I can club the queries for better performance ? 

I have read that I BOOL filters should be used instead of AND filter since they use bitset which are cached internally. I think this makes one improvement because if in the first query ES stores the information of filters in bitset, it can reuse it in other two queries. That will make the thigs a little fast but based on queries, I'm not able to do any improvement ?

Is there any way by which I can combine match and multi-match queries ( 1 and 2) into a single effective query.

Also, in place of query_string should I use some other query for faster execution.

Any suggestions are welcome. 
Thanks


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/741a7bc5-ffd7-4ba7-9296-ff6fff8f559f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZJUXjDVyXFtQSB0-0Mghf5FHC%2Bmm%2BiJUiSCjKJtPDCbg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.