What sort of write frequency does elasticsearch support?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

What sort of write frequency does elasticsearch support?

doug livesey
Hi -- we're about to replace a lot of our search and index queries with elasticsearch, but were also thinking about having it serve get requests by id on items.
To do this, we'd need to keep the search index totally up-to-date with the database, which may involve a lot of writes.
We estimate these (research ongoing) to be < 5000 per day, but as we mainly operate in the UK, those will cluster around UK daylight hours.
Would elasticsearch comfortably handle that amount of index writing?
And at what point do people think we'd need to start worrying about pulling the updates to one side to perform as a bulk operation?
Thanks for any & all advice,
   Doug.
Reply | Threaded
Open this post in threaded view
|

Re: What sort of write frequency does elasticsearch support?

Ivan Brusic
5000 is not a lot for any configuration.  In one of my projects, elasticsearch continuously indexes (using a bulk indexer for performance reasons, but not really necessary) around 75K documents a day on an EC2 microinstance (613MBs total of memory/300MBs allocated toward the JVM).  For another project, I was incorrectly not using a bulk indexer when batch updating, resulting in 50K+ continuous inserts.  Once again, no issues.

-- 
Ivan

On Fri, Aug 12, 2011 at 7:01 AM, doug livesey <[hidden email]> wrote:
Hi -- we're about to replace a lot of our search and index queries with elasticsearch, but were also thinking about having it serve get requests by id on items.
To do this, we'd need to keep the search index totally up-to-date with the database, which may involve a lot of writes.
We estimate these (research ongoing) to be < 5000 per day, but as we mainly operate in the UK, those will cluster around UK daylight hours.
Would elasticsearch comfortably handle that amount of index writing?
And at what point do people think we'd need to start worrying about pulling the updates to one side to perform as a bulk operation?
Thanks for any & all advice,
   Doug.

Reply | Threaded
Open this post in threaded view
|

Re: What sort of write frequency does elasticsearch support?

doug livesey
That's excellent news to hear, thankyou! :)

On 12 August 2011 14:59, Ivan Brusic <[hidden email]> wrote:
5000 is not a lot for any configuration.  In one of my projects, elasticsearch continuously indexes (using a bulk indexer for performance reasons, but not really necessary) around 75K documents a day on an EC2 microinstance (613MBs total of memory/300MBs allocated toward the JVM).  For another project, I was incorrectly not using a bulk indexer when batch updating, resulting in 50K+ continuous inserts.  Once again, no issues.

-- 
Ivan


On Fri, Aug 12, 2011 at 7:01 AM, doug livesey <[hidden email]> wrote:
Hi -- we're about to replace a lot of our search and index queries with elasticsearch, but were also thinking about having it serve get requests by id on items.
To do this, we'd need to keep the search index totally up-to-date with the database, which may involve a lot of writes.
We estimate these (research ongoing) to be < 5000 per day, but as we mainly operate in the UK, those will cluster around UK daylight hours.
Would elasticsearch comfortably handle that amount of index writing?
And at what point do people think we'd need to start worrying about pulling the updates to one side to perform as a bulk operation?
Thanks for any & all advice,
   Doug.