NRT search with elastic search

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

NRT search with elastic search

rcch
Hi all,

1) I'd like to get some feedback/hear experiences from people who've
used elastic search for near-real-time search applications.

2) How does elastic search compare to using near-real time search in
lucene ?

3) I believe there's a 1 second delay before documents can be made
searchable (due to how the index gets updated asynchronously). Is this
a parameter that can be tuned ?

Our application requires an ingestion of content of ~1000s of
documents/second, but they need to be searchable right-away..

Thanks for your input.

Cheers,

Vijay
Reply | Threaded
Open this post in threaded view
|

Re: NRT search with elastic search

kimchy
Administrator
On Thu, Aug 11, 2011 at 5:44 AM, rcch <[hidden email]> wrote:
Hi all,

1) I'd like to get some feedback/hear experiences from people who've
used elastic search for near-real-time search applications.

2) How does elastic search compare to using near-real time search in
lucene ?

elasticsearch uses Lucene NRT.
 

3) I believe there's a 1 second delay before documents can be made
searchable (due to how the index gets updated asynchronously). Is this
a parameter that can be tuned ?

The index is not updated asynchronously, when you index data its applied to the shard and the replicas in a sync manner including writing to a transaction log. A "fresh" view of the content to be searchable is opened every 1 second by default (thats NRT).
 

Our application requires an ingestion of content of ~1000s of
documents/second, but they need to be searchable right-away..

Thanks for your input.

Cheers,

Vijay

Reply | Threaded
Open this post in threaded view
|

Re: NRT search with elastic search

Pavel Penchev
Hi

On 11.08.2011 11:25, Shay Banon wrote:
On Thu, Aug 11, 2011 at 5:44 AM, rcch <[hidden email]> wrote:
Hi all,

1) I'd like to get some feedback/hear experiences from people who've
used elastic search for near-real-time search applications.

2) How does elastic search compare to using near-real time search in
lucene ?

elasticsearch uses Lucene NRT.
 

3) I believe there's a 1 second delay before documents can be made
searchable (due to how the index gets updated asynchronously). Is this
a parameter that can be tuned ?

The index is not updated asynchronously, when you index data its applied to the shard and the replicas in a sync manner including writing to a transaction log. A "fresh" view of the content to be searchable is opened every 1 second by default (thats NRT).
 

Our application requires an ingestion of content of ~1000s of
documents/second, but they need to be searchable right-away..
In case you only need them "get-able" right-away - ES has this since 0.17.0, see https://github.com/elasticsearch/elasticsearch/issues/1060

Regards,
Pavel


Thanks for your input.

Cheers,

Vijay


Reply | Threaded
Open this post in threaded view
|

Re: NRT search with elastic search

rcch
Thanks for your replies, Shay, Pavel.

Can you please tell me

0) Is it true that there will always be a 1 second delay before
content can be searchable?

1) How does ES compare to Zoie (from Linkedin) for real-time-search
applications ?

2) What kind of ingestion rates can ES support comfortably ? Is 10s/
second ?  100s/second ? or 1000s/second ?  Can you suggest where you
think the breakdown is?

Thanks,

Cheers,

Vijay

On Aug 11, 8:13 am, Pavel Penchev <[hidden email]> wrote:

> Hi
>
> On 11.08.2011 11:25, Shay Banon wrote:
>
>
>
>
>
>
>
> > On Thu, Aug 11, 2011 at 5:44 AM, rcch <[hidden email]
> > <mailto:[hidden email]>> wrote:
>
> >     Hi all,
>
> >     1) I'd like to get some feedback/hear experiences from people who've
> >     used elastic search for near-real-time search applications.
>
> >     2) How does elastic search compare to using near-real time search in
> >     lucene ?
>
> > elasticsearch uses Lucene NRT.
>
> >     3) I believe there's a 1 second delay before documents can be made
> >     searchable (due to how the index gets updated asynchronously). Is this
> >     a parameter that can be tuned ?
>
> > The index is not updated asynchronously, when you index data its
> > applied to the shard and the replicas in a sync manner including
> > writing to a transaction log. A "fresh" view of the content to be
> > searchable is opened every 1 second by default (thats NRT).
>
> >     Our application requires an ingestion of content of ~1000s of
> >     documents/second, but they need to be searchable right-away..
>
> In case you only need them "get-able" right-away - ES has this since
> 0.17.0, seehttps://github.com/elasticsearch/elasticsearch/issues/1060
>
> Regards,
> Pavel
>
>
>
>
>
>
>
>
>
> >     Thanks for your input.
>
> >     Cheers,
>
> >     Vijay
Reply | Threaded
Open this post in threaded view
|

Re: NRT search with elastic search

Berkay Mollamustafaoglu-2

On Fri, Aug 12, 2011 at 3:02 PM, rcch <[hidden email]> wrote:
Thanks for your replies, Shay, Pavel.

Can you please tell me

0) Is it true that there will always be a 1 second delay before
content can be searchable?
By default, refresh is called every 1 sec, so there would be delay up to 1 sec. However, refresh interval is configurable and can also be called programmatically. Technically you can call it after indexing every document but this would have significant performance impact. 
 
1) How does ES compare to Zoie (from Linkedin) for real-time-search
applications ?

2) What kind of ingestion rates can ES support comfortably ? Is 10s/
second ?  100s/second ? or 1000s/second ?  Can you suggest where you
think the breakdown is?

Hard to tell. What is the document size? How many servers (CPU, Memory, Disks) in the cluster? ES is horizontally scalable so event 1000s/sec is achievable with multiple servers. No better way to find it out is to test with your own docs.


Thanks,

Cheers,

Vijay

On Aug 11, 8:13 am, Pavel Penchev <[hidden email]> wrote:
> Hi
>
> On 11.08.2011 11:25, Shay Banon wrote:
>
>
>
>
>
>
>
> > On Thu, Aug 11, 2011 at 5:44 AM, rcch <[hidden email]
> > <mailto:[hidden email]>> wrote:
>
> >     Hi all,
>
> >     1) I'd like to get some feedback/hear experiences from people who've
> >     used elastic search for near-real-time search applications.
>
> >     2) How does elastic search compare to using near-real time search in
> >     lucene ?
>
> > elasticsearch uses Lucene NRT.
>
> >     3) I believe there's a 1 second delay before documents can be made
> >     searchable (due to how the index gets updated asynchronously). Is this
> >     a parameter that can be tuned ?
>
> > The index is not updated asynchronously, when you index data its
> > applied to the shard and the replicas in a sync manner including
> > writing to a transaction log. A "fresh" view of the content to be
> > searchable is opened every 1 second by default (thats NRT).
>
> >     Our application requires an ingestion of content of ~1000s of
> >     documents/second, but they need to be searchable right-away..
>
> In case you only need them "get-able" right-away - ES has this since
> 0.17.0, seehttps://github.com/elasticsearch/elasticsearch/issues/1060
>
> Regards,
> Pavel
>
>
>
>
>
>
>
>
>
> >     Thanks for your input.
>
> >     Cheers,
>
> >     Vijay