Percolator for alerts between specific time intervals?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Percolator for alerts between specific time intervals?

vibin
Hi,

I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).

We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.

We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.

Please let me know, if I want to give any more information.

Thanks in Advance

vibin/manoj
Reply | Threaded
Open this post in threaded view
|

Re: Percolator for alerts between specific time intervals?

Otis Gospodnetic
Hi,

Percolator runs incoming documents (documents getting indexed) against
stored Percolator queries as documents are being indexed (i.e. not
periodically).  However, the "interval aspect" of your requirement
could be implemented in your application, which can use Percolator to
find out when documents match queries, and then buffer them in
appropriate buckets.

Otis
--
http://sematext.com/


On Jan 26, 10:37 pm, vibin dhas <[hidden email]> wrote:

> Hi,
>
> I have been researching the Percolator feature of the ES, from which I
> understood that, this is useful for dynamic alerts or Streams(for
> every indexing of document or bulk indexing).
>
> We have a requirement to create alerts between specific time
> intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
> really interested to know if the Percolator could be used for this
> purpose.
>
> We have a 24/7 live Indexing scheduler, indexing documents in multiple
> indices. Almost 18 million documents are created per day.
>
> Please let me know, if I want to give any more information.
>
> Thanks in Advance
>
> vibin/manoj
Reply | Threaded
Open this post in threaded view
|

Re: Percolator for alerts between specific time intervals?

Michael Sick
In reply to this post by vibin
Hi Vibin,

I've implemented a similar system in the past using custom code. We chose to:
  1) Generate the percolation matches as documents came in
  2) Persist the matches consisting of something like:
    document_id, percolation(subscription)_id, time_of_match, delivery_status
  3a) Deliver real-time events immediately
  3b) Invoke a query for any matches for the given subscription_id from a scheduler (quartz or whatever you prefer)

I believe I'd do the same thing with ES if I was using percolation. I've been rethinking it as we may implement one in the coming weeks.

--Mike

On Thu, Jan 26, 2012 at 10:37 PM, vibin dhas <[hidden email]> wrote:
Hi,

I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).

We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.

We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.

Please let me know, if I want to give any more information.

Thanks in Advance

vibin/manoj

Reply | Threaded
Open this post in threaded view
|

Re: Percolator for alerts between specific time intervals?

kimchy
Administrator
Another option might be the simplest one, which is to poll for changes that happened in the given time span you are interested in. Filters can make this part really fast as they are nicely cached.

On Friday, January 27, 2012 at 9:03 AM, Michael Sick wrote:

Hi Vibin,

I've implemented a similar system in the past using custom code. We chose to:
  1) Generate the percolation matches as documents came in
  2) Persist the matches consisting of something like:
    document_id, percolation(subscription)_id, time_of_match, delivery_status
  3a) Deliver real-time events immediately
  3b) Invoke a query for any matches for the given subscription_id from a scheduler (quartz or whatever you prefer)

I believe I'd do the same thing with ES if I was using percolation. I've been rethinking it as we may implement one in the coming weeks.

--Mike

On Thu, Jan 26, 2012 at 10:37 PM, vibin dhas <[hidden email]> wrote:
Hi,

I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).

We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.

We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.

Please let me know, if I want to give any more information.

Thanks in Advance

vibin/manoj


Reply | Threaded
Open this post in threaded view
|

Re: Percolator for alerts between specific time intervals?

vibin
Great got it...Still I have a doubt

Does Polling for changes in the indices and Hitting the indices with
search(with filter) at specific time intervals, will have same
performance?


On Jan 28, 1:53 am, Shay Banon <[hidden email]> wrote:

> Another option might be the simplest one, which is to poll for changes that happened in the given time span you are interested in. Filters can make this part really fast as they are nicely cached.
>
>
>
>
>
>
>
> On Friday, January 27, 2012 at 9:03 AM, Michael Sick wrote:
> > Hi Vibin,
>
> > I've implemented a similar system in the past using custom code. We chose to:
> >   1) Generate the percolation matches as documents came in
> >   2) Persist the matches consisting of something like:
> >     document_id, percolation(subscription)_id, time_of_match, delivery_status
> >   3a) Deliver real-time events immediately
> >   3b) Invoke a query for any matches for the given subscription_id from a scheduler (quartz or whatever you prefer)
>
> > I believe I'd do the same thing with ES if I was using percolation. I've been rethinking it as we may implement one in the coming weeks.
>
> > --Mike
>
> > On Thu, Jan 26, 2012 at 10:37 PM, vibin dhas <[hidden email] (mailto:[hidden email])> wrote:
> > > Hi,
>
> > > I have been researching the Percolator feature of the ES, from which I
> > > understood that, this is useful for dynamic alerts or Streams(for
> > > every indexing of document or bulk indexing).
>
> > > We have a requirement to create alerts between specific time
> > > intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
> > > really interested to know if the Percolator could be used for this
> > > purpose.
>
> > > We have a 24/7 live Indexing scheduler, indexing documents in multiple
> > > indices. Almost 18 million documents are created per day.
>
> > > Please let me know, if I want to give any more information.
>
> > > Thanks in Advance
>
> > > vibin/manoj
Reply | Threaded
Open this post in threaded view
|

Re: Percolator for alerts between specific time intervals?

kimchy
Administrator
Searching (with a filter) is what I meant by polling for changes.

On Tuesday, January 31, 2012 at 6:27 AM, vibin dhas wrote:

Great got it...Still I have a doubt

Does Polling for changes in the indices and Hitting the indices with
search(with filter) at specific time intervals, will have same
performance?


On Jan 28, 1:53 am, Shay Banon <kim...@gmail.com> wrote:
Another option might be the simplest one, which is to poll for changes that happened in the given time span you are interested in. Filters can make this part really fast as they are nicely cached.







On Friday, January 27, 2012 at 9:03 AM, Michael Sick wrote:
Hi Vibin,

I've implemented a similar system in the past using custom code. We chose to:
  1) Generate the percolation matches as documents came in
  2) Persist the matches consisting of something like:
    document_id, percolation(subscription)_id, time_of_match, delivery_status
  3a) Deliver real-time events immediately
  3b) Invoke a query for any matches for the given subscription_id from a scheduler (quartz or whatever you prefer)

I believe I'd do the same thing with ES if I was using percolation. I've been rethinking it as we may implement one in the coming weeks.

--Mike

On Thu, Jan 26, 2012 at 10:37 PM, vibin dhas <[hidden email] ([hidden email]...@gmail.com)> wrote:
Hi,

I have been researching the Percolator feature of the ES, from which I
understood that, this is useful for dynamic alerts or Streams(for
every indexing of document or bulk indexing).

We have a requirement to create alerts between specific time
intervals. For Eg: Once in 10mins, Once in an Hr, Once in a day. I am
really interested to know if the Percolator could be used for this
purpose.

We have a 24/7 live Indexing scheduler, indexing documents in multiple
indices. Almost 18 million documents are created per day.

Please let me know, if I want to give any more information.

Thanks in Advance

vibin/manoj