Is Bulk Percolation possible?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Is Bulk Percolation possible?

Mohammady Mahdy
I have a use case were I am receiving a constant stream of documents (on the scale of a thousand document every couple of seconds). and I have a functionality were certain users are interested in only a subset of this and wants to be notified if any new documents of the kind are added.


I was wondering if it's possible to percolate in batch as well. i.e. after submitting my batch, the response that I get can include queries and the id's of documents that matched them or a certain field of my choice.

I was also wondering about the scale and validity of such a solution, i.e. how many percolators can be present and how does the growth of such number affect perf. in general.

Say I have a 1000 users, and each will have 2-3 queries that I need to percolate against. Is this valid? is this too much? if this is not the way to go. What would be the best way to go given that ES is the only place this data lives permanently now.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Is Bulk Percolation possible?

Igor Motov-3
Bulk requests support percolation. You just need to set percolate field on your IndexRequest. As far as performance is concerned, I would suggest testing it. 

On Monday, April 8, 2013 6:25:38 AM UTC-4, Mo wrote:
I have a use case were I am receiving a constant stream of documents (on the scale of a thousand document every couple of seconds). and I have a functionality were certain users are interested in only a subset of this and wants to be notified if any new documents of the kind are added.


I was wondering if it's possible to percolate in batch as well. i.e. after submitting my batch, the response that I get can include queries and the id's of documents that matched them or a certain field of my choice.

I was also wondering about the scale and validity of such a solution, i.e. how many percolators can be present and how does the growth of such number affect perf. in general.

Say I have a 1000 users, and each will have 2-3 queries that I need to percolate against. Is this valid? is this too much? if this is not the way to go. What would be the best way to go given that ES is the only place this data lives permanently now.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Is Bulk Percolation possible?

Mohammady Mahdy
Perfect! Many thanks.

On Monday, April 8, 2013 10:04:09 PM UTC+4, Igor Motov wrote:
Bulk requests support percolation. You just need to set percolate field on your IndexRequest. As far as performance is concerned, I would suggest testing it. 

On Monday, April 8, 2013 6:25:38 AM UTC-4, Mo wrote:
I have a use case were I am receiving a constant stream of documents (on the scale of a thousand document every couple of seconds). and I have a functionality were certain users are interested in only a subset of this and wants to be notified if any new documents of the kind are added.


I was wondering if it's possible to percolate in batch as well. i.e. after submitting my batch, the response that I get can include queries and the id's of documents that matched them or a certain field of my choice.

I was also wondering about the scale and validity of such a solution, i.e. how many percolators can be present and how does the growth of such number affect perf. in general.

Say I have a 1000 users, and each will have 2-3 queries that I need to percolate against. Is this valid? is this too much? if this is not the way to go. What would be the best way to go given that ES is the only place this data lives permanently now.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Is Bulk Percolation possible?

webmstr
This post has NOT been accepted by the mailing list yet.
In reply to this post by Igor Motov-3
Might you have a sample command?  I've tried:

curl -XPOST 'http://localhost:9200/_bulk?percolate=*' --data-binary @data.json

but I don't get any percolate-related info back in the reply.