Who uses ES in production ?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Who uses ES in production ?

dpilato

Hello « the list » ;-)

 

I would like to know if someone uses Elastic Search in production environment. If possible also to give me a feedback about the number of document indexed…

 

It will be very useful for IT managers to know that…

Shay ? Do you have somewhere ES testimonials or guestbook ?

 

Thanks

David.

Reply | Threaded
Open this post in threaded view
|

Re: Who uses ES in production ?

kimchy
Administrator
Regarding usage in production, I will let the list to answer. I know of several quite big production uses of elasticsearch (TBs of data, billions of documents).

What I would love to do is to have a "usage" section in the site. With user stories on using elasticsearch. So, if someone is interested in that, it would be great. We can collaborate on creating the first several user stories.

-shay.banon

On Tuesday, March 15, 2011 at 10:52 AM, David Pilato wrote:

Hello « the list » ;-)

 

I would like to know if someone uses Elastic Search in production environment. If possible also to give me a feedback about the number of document indexed…

 

It will be very useful for IT managers to know that…

Shay ? Do you have somewhere ES testimonials or guestbook ?

 

Thanks

David.


Reply | Threaded
Open this post in threaded view
|

Re: Who uses ES in production ?

Alexandre Heimburger
I plan to use ES in production in a few weeks.

We are a french software company building a social network software for big companies.

In a first step, ES is used for a super fast auto completion engine.



On Tue, Mar 15, 2011 at 10:05 AM, Shay Banon <[hidden email]> wrote:
Regarding usage in production, I will let the list to answer. I know of several quite big production uses of elasticsearch (TBs of data, billions of documents).

What I would love to do is to have a "usage" section in the site. With user stories on using elasticsearch. So, if someone is interested in that, it would be great. We can collaborate on creating the first several user stories.

-shay.banon

On Tuesday, March 15, 2011 at 10:52 AM, David Pilato wrote:

Hello « the list » ;-)

 

I would like to know if someone uses Elastic Search in production environment. If possible also to give me a feedback about the number of document indexed…

 

It will be very useful for IT managers to know that…

Shay ? Do you have somewhere ES testimonials or guestbook ?

 

Thanks

David.





--
Alexandre Heimburger
R&D Manager
blueKiwi Software
tel : +33687880997
email : [hidden email]
adress : 93 rue Vieille du Temple, 75003 Paris

What is blueKiwi? blueKiwi - the first Enterprise Social Software Suite in the world building professional networks on conversations and relationships - helps large organizations increase their productivity, foster innovations and boost people satisfaction.
Reply | Threaded
Open this post in threaded view
|

Re: Who uses ES in production ?

Enrique Medina Montenegro
We also plan to start using ES as a product search engine in our website cuestamenos.com

This product search engine will also combine search of stores and faceting by categories and prices.

We're talking here about 5 million products distributed in around 500 stores and 30 categories.


On Tue, Mar 15, 2011 at 10:11 AM, Alexandre Heimburger <[hidden email]> wrote:
I plan to use ES in production in a few weeks.

We are a french software company building a social network software for big companies.

In a first step, ES is used for a super fast auto completion engine.



On Tue, Mar 15, 2011 at 10:05 AM, Shay Banon <[hidden email]> wrote:
Regarding usage in production, I will let the list to answer. I know of several quite big production uses of elasticsearch (TBs of data, billions of documents).

What I would love to do is to have a "usage" section in the site. With user stories on using elasticsearch. So, if someone is interested in that, it would be great. We can collaborate on creating the first several user stories.

-shay.banon

On Tuesday, March 15, 2011 at 10:52 AM, David Pilato wrote:

Hello « the list » ;-)

 

I would like to know if someone uses Elastic Search in production environment. If possible also to give me a feedback about the number of document indexed…

 

It will be very useful for IT managers to know that…

Shay ? Do you have somewhere ES testimonials or guestbook ?

 

Thanks

David.





--
Alexandre Heimburger
R&D Manager
blueKiwi Software
tel : +33687880997
email : [hidden email]
adress : 93 rue Vieille du Temple, 75003 Paris

What is blueKiwi? blueKiwi - the first Enterprise Social Software Suite in the world building professional networks on conversations and relationships - helps large organizations increase their productivity, foster innovations and boost people satisfaction.

Reply | Threaded
Open this post in threaded view
|

Re: Who uses ES in production ?

Karel Minařík
In reply to this post by dpilato
We are in the process of migrating a social monitoring tool [http://
ataxosocialinsider.cz/; case study here: http://www.couchbase.com/case-studies/ataxo]
from CouchDB-Lucene to ElasticSearch, after couple of weeks of
research and testing. We plan to use ES both on the "frontend" part
(which the customer uses to filter, search, mine mentions) as well on
the "backend" part (the actual crawlers, aggregators, etc).

So far, we've been testing with document counts in several hundred of
thousands only, and the ES performance was more then satisfactory.
Given it's internal design, we're confident it can deal with couple of
terabytes of data and millions of documents on the backend. If we
would be proved wrong, we'll let everybody know :)

Best!,

Karel

On Mar 15, 9:52 am, David Pilato
<[hidden email]> wrote:

> Hello < the list > ;-)
>
> I would like to know if someone uses Elastic Search in production
> environment. If possible also to give me a feedback about the number of
> document indexed.
>
> It will be very useful for IT managers to know that.
>
> Shay ? Do you have somewhere ES testimonials or guestbook ?
>
> Thanks
>
> David.
Reply | Threaded
Open this post in threaded view
|

Re: Who uses ES in production ?

Joaquin Cuenca Abela
We're using it in production at http://www.presspeople.com/ where
we're indexing mainly press offices and press releases in spanish. We
have only around 100.000 items that we sync from a mongodb database in
near real time, through a queue managed by celeryd. As mongodb and ES
are not updated atomically, we bulk index everything every night
(stopping the update queues before the bulk index and restarting them
after the bulk index finishes). These nightly indexes have a timestamp
in their name, and they're aliased to a canonical name (presspeople).

We have been moving some queries from mongodb to ES, but we don't yet
trust it as the main repository of content. For example a few days ago
I had to move ES directory, and after I could access the last index
(presspeople-2011-3-5), but I could not access it by its aliased name
(presspeople). Being able to wipe old data and rebuild it whenever I
want helps me sleep better.

We love the design and API of ES, and all the features that keep
coming, all in all, I'm a very happy user.

Cheers,

On Tue, Mar 15, 2011 at 10:40 AM, Karel Minarik <[hidden email]> wrote:

> We are in the process of migrating a social monitoring tool [http://
> ataxosocialinsider.cz/; case study here: http://www.couchbase.com/case-studies/ataxo]
> from CouchDB-Lucene to ElasticSearch, after couple of weeks of
> research and testing. We plan to use ES both on the "frontend" part
> (which the customer uses to filter, search, mine mentions) as well on
> the "backend" part (the actual crawlers, aggregators, etc).
>
> So far, we've been testing with document counts in several hundred of
> thousands only, and the ES performance was more then satisfactory.
> Given it's internal design, we're confident it can deal with couple of
> terabytes of data and millions of documents on the backend. If we
> would be proved wrong, we'll let everybody know :)
>
> Best!,
>
> Karel
>
> On Mar 15, 9:52 am, David Pilato
> <[hidden email]> wrote:
>> Hello < the list > ;-)
>>
>> I would like to know if someone uses Elastic Search in production
>> environment. If possible also to give me a feedback about the number of
>> document indexed.
>>
>> It will be very useful for IT managers to know that.
>>
>> Shay ? Do you have somewhere ES testimonials or guestbook ?
>>
>> Thanks
>>
>> David.
>



--
Joaquin Cuenca Abela -- presspeople.com: Fuentes de prensa y comunicados
Reply | Threaded
Open this post in threaded view
|

Re: Who uses ES in production ?

IvanBrusic
In reply to this post by dpilato
I have been using ElasticSearch for a trivial application at work.
Nothing too exciting, the volume of data is rather low (for now), but
the REST interface allows querying from different applications.

In the meantime, I built a websocket river for ElasticSearch.  I am
currently indexing all RSVPs from Meetup.com. Why? Because. :)  I have
been doing some rudimentary data mining using only ES and JSON calls!
Top programming language (by the amount of RSVPs)?  PHP.  Top state
for development meetups?  NY.  My plan is to have an HTML5 interface
to present the data, but I am not a front-end guy and I have yet to
start the work.

Ivan

On Mar 15, 4:52 am, David Pilato
<[hidden email]> wrote:

> Hello < the list > ;-)
>
> I would like to know if someone uses Elastic Search in production
> environment. If possible also to give me a feedback about the number of
> document indexed.
>
> It will be very useful for IT managers to know that.
>
> Shay ? Do you have somewhere ES testimonials or guestbook ?
>
> Thanks
>
> David.
Reply | Threaded
Open this post in threaded view
|

RE: SPAM-LOW: Re: Who uses ES in production ?

Administrator-2
Ivan,

        Actually, I'd be interested in the meetup river, as I would like to
find events of a particular nature (video game tournaments/casuals for
Street Fighter 4, Marvel vs Capcom 3, etc).

        Care to share?

                - Nick

-----Original Message-----
From: Ivan Brusic [mailto:[hidden email]]
Sent: Tuesday, March 15, 2011 8:05 AM
To: users
Subject: SPAM-LOW: Re: Who uses ES in production ?

I have been using ElasticSearch for a trivial application at work.
Nothing too exciting, the volume of data is rather low (for now), but
the REST interface allows querying from different applications.

In the meantime, I built a websocket river for ElasticSearch.  I am
currently indexing all RSVPs from Meetup.com. Why? Because. :)  I have
been doing some rudimentary data mining using only ES and JSON calls!
Top programming language (by the amount of RSVPs)?  PHP.  Top state
for development meetups?  NY.  My plan is to have an HTML5 interface
to present the data, but I am not a front-end guy and I have yet to
start the work.

Ivan

On Mar 15, 4:52 am, David Pilato
<[hidden email]> wrote:

> Hello < the list > ;-)
>
> I would like to know if someone uses Elastic Search in production
> environment. If possible also to give me a feedback about the number of
> document indexed.
>
> It will be very useful for IT managers to know that.
>
> Shay ? Do you have somewhere ES testimonials or guestbook ?
>
> Thanks
>
> David.



Reply | Threaded
Open this post in threaded view
|

Re: SPAM-LOW: Re: Who uses ES in production ?

Ivan Porto Carrero
We're using elasticsearch and definitely will put it in production. we're a company that is streaming the web to you.


We're in limited beta but we do things like make a http stream/websocket stream of web api's/RSS/email/sms. we're looking for more private beta testers soon. 
We allow for filters to be specified on the stream and this all happens through elasticsearch/lucene. so you have a real-time filtered stream to everything on the web.

If you would be interested in beta testing this send me your email address and I'll add you to our invite list.

Met vriendelijke groeten - Best regards - Salutations

Ivan Porto Carrero
Co-founder & Developer at Mojolly
Mojolly
Phone: +44.7513.233.465



On Tue, Mar 15, 2011 at 1:14 PM, Administrator <[hidden email]> wrote:
Ivan,

       Actually, I'd be interested in the meetup river, as I would like to
find events of a particular nature (video game tournaments/casuals for
Street Fighter 4, Marvel vs Capcom 3, etc).

       Care to share?

               - Nick

-----Original Message-----
From: Ivan Brusic [mailto:[hidden email]]
Sent: Tuesday, March 15, 2011 8:05 AM
To: users
Subject: SPAM-LOW: Re: Who uses ES in production ?

I have been using ElasticSearch for a trivial application at work.
Nothing too exciting, the volume of data is rather low (for now), but
the REST interface allows querying from different applications.

In the meantime, I built a websocket river for ElasticSearch.  I am
currently indexing all RSVPs from Meetup.com. Why? Because. :)  I have
been doing some rudimentary data mining using only ES and JSON calls!
Top programming language (by the amount of RSVPs)?  PHP.  Top state
for development meetups?  NY.  My plan is to have an HTML5 interface
to present the data, but I am not a front-end guy and I have yet to
start the work.

Ivan

On Mar 15, 4:52 am, David Pilato
<[hidden email]> wrote:
> Hello < the list > ;-)
>
> I would like to know if someone uses Elastic Search in production
> environment. If possible also to give me a feedback about the number of
> document indexed.
>
> It will be very useful for IT managers to know that.
>
> Shay ? Do you have somewhere ES testimonials or guestbook ?
>
> Thanks
>
> David.




Reply | Threaded
Open this post in threaded view
|

Re: SPAM-LOW: Re: Who uses ES in production ?

ppearcy
We've been using elasticsearch in production since last December.
Performance (search/index), flexibility, and APIs are all awesome.

~33 million docs w/ millions of queries per day with avg response
times at ~40ms and max times ~2 secs. Our query strings have a lot of
filters in them, too. Unfortunately, I still can't go into exact
details on where it is being used due to some legal issues.

Our current pain point is that there is a bug around network
partitioning that can cause data loss that we are paranoid about.
Also, when data is re-balancing performance can be impacted, but we
have mostly mitigated that by only allowing a single index cover and
single stream per index.


On Mar 15, 12:34 pm, Ivan Porto Carrero <[hidden email]> wrote:

> We're using elasticsearch and definitely will put it in production. we're a
> company that is streaming the web to you.
>
> http://backchat.io
>
> We're in limited beta but we do things like make a http stream/websocket
> stream of web api's/RSS/email/sms. we're looking for more private beta
> testers soon.
> We allow for filters to be specified on the stream and this all happens
> through elasticsearch/lucene. so you have a real-time filtered stream to
> everything on the web.
>
> If you would be interested in beta testing this send me your email address
> and I'll add you to our invite list.
>
> *Met vriendelijke groeten - Best regards - Salutations*
>
> *Ivan Porto Carrero*
> Co-founder & Developer at Mojolly
> Mojolly
> Visit us at *www.mojolly.com, flanders.co.nz*
> We tweet at @mojolly <http://twitter.com/mojolly>,
> @casualjim<http://twitter.com/casualjim>
>
> Phone: +44.7513.233.465
>
>
>
>
>
>
>
> On Tue, Mar 15, 2011 at 1:14 PM, Administrator <[hidden email]> wrote:
> > Ivan,
>
> >        Actually, I'd be interested in the meetup river, as I would like to
> > find events of a particular nature (video game tournaments/casuals for
> > Street Fighter 4, Marvel vs Capcom 3, etc).
>
> >        Care to share?
>
> >                - Nick
>
> > -----Original Message-----
> > From: Ivan Brusic [mailto:[hidden email]]
> > Sent: Tuesday, March 15, 2011 8:05 AM
> > To: users
> > Subject: SPAM-LOW: Re: Who uses ES in production ?
>
> > I have been using ElasticSearch for a trivial application at work.
> > Nothing too exciting, the volume of data is rather low (for now), but
> > the REST interface allows querying from different applications.
>
> > In the meantime, I built a websocket river for ElasticSearch.  I am
> > currently indexing all RSVPs from Meetup.com. Why? Because. :)  I have
> > been doing some rudimentary data mining using only ES and JSON calls!
> > Top programming language (by the amount of RSVPs)?  PHP.  Top state
> > for development meetups?  NY.  My plan is to have an HTML5 interface
> > to present the data, but I am not a front-end guy and I have yet to
> > start the work.
>
> > Ivan
>
> > On Mar 15, 4:52 am, David Pilato
> > <[hidden email]> wrote:
> > > Hello < the list > ;-)
>
> > > I would like to know if someone uses Elastic Search in production
> > > environment. If possible also to give me a feedback about the number of
> > > document indexed.
>
> > > It will be very useful for IT managers to know that.
>
> > > Shay ? Do you have somewhere ES testimonials or guestbook ?
>
> > > Thanks
>
> > > David.
Reply | Threaded
Open this post in threaded view
|

Re: SPAM-LOW: Re: Who uses ES in production ?

IvanBrusic
In reply to this post by Administrator-2
Collecting data from any websocket into ElasticSearch is just a silly
pet project, which means I have little time to work on it. Here is
some code so far:

https://github.com/ibrusic/simple-websocket-river

It is not complete, but functional.  Almost finished with a
configurable index builder (bulk or not).  There is no testing since
testing for a plugin seems tricky.  The code needs to be abstracted
first.  Also, ES plugins need to be compiled for a specific version,
so I need to create a configurable build script instead of changing
the project file, which might be confusing for non-SBT users.

I will probably work on it more after 0.16 is officially released.
0.16 is required, so I am working of maven snapshots for the plugin
build and compiling from master for the actual server.

If you plan on working with Meetup data, the example mapping is far
from complete.  The default analyzer and no storage does not work for
most "interesting" fields.

Ivan


On Mar 15, 9:14 am, "Administrator" <[hidden email]> wrote:

> Ivan,
>
>         Actually, I'd be interested in the meetup river, as I would like to
> find events of a particular nature (video game tournaments/casuals for
> Street Fighter 4, Marvel vs Capcom 3, etc).
>
>         Care to share?
>
>                 - Nick
>
>
>
>
>
>
>
> -----Original Message-----
> From: Ivan Brusic [mailto:[hidden email]]
> Sent: Tuesday, March 15, 2011 8:05 AM
> To: users
> Subject: SPAM-LOW: Re: Who uses ES in production ?
>
> I have been using ElasticSearch for a trivial application at work.
> Nothing too exciting, the volume of data is rather low (for now), but
> the REST interface allows querying from different applications.
>
> In the meantime, I built a websocket river for ElasticSearch.  I am
> currently indexing all RSVPs from Meetup.com. Why? Because. :)  I have
> been doing some rudimentary data mining using only ES and JSON calls!
> Top programming language (by the amount of RSVPs)?  PHP.  Top state
> for development meetups?  NY.  My plan is to have an HTML5 interface
> to present the data, but I am not a front-end guy and I have yet to
> start the work.
>
> Ivan
>
> On Mar 15, 4:52 am, David Pilato
> <[hidden email]> wrote:
> > Hello < the list > ;-)
>
> > I would like to know if someone uses Elastic Search in production
> > environment. If possible also to give me a feedback about the number of
> > document indexed.
>
> > It will be very useful for IT managers to know that.
>
> > Shay ? Do you have somewhere ES testimonials or guestbook ?
>
> > Thanks
>
> > David.
Reply | Threaded
Open this post in threaded view
|

Re: SPAM-LOW: Re: Who uses ES in production ?

IvanBrusic
In reply to this post by Administrator-2
Not sure why my previous response has yet to appear.

Indexing any content from a websocket into ElasticSearch is just a pet
project, which means I do not have much time to work on it!  Here is
some code so far:

https://github.com/ibrusic/simple-websocket-river

It is still not complete, but it is functional.  I have some code for
configurable index builders (bulk or not) that is not checked in.
Also missing are unit tests since testing for a plugin is tricky.
Need to abstract my classes a bit more first.  The main limitation is
the build system.  ElasticSearch plugins must be compiled against the
version you intend to run with and currently the version is hardcoded
inside the SBT project file.  Once I finished with the code, I will
create pre-built jar files.

I will probably work on it more once 0.16 is released.  0.16 is a
requirement due to changes in the river api, so for now you must build
against jars from the snapshot repo and deploy with a server built
from sources.

The mapping for the meetup example is far from complete.  I actually
set a few fields to be stored/non-indexded.

Ivan

On Mar 15, 9:14 am, "Administrator" <[hidden email]> wrote:

> Ivan,
>
>         Actually, I'd be interested in the meetup river, as I would like to
> find events of a particular nature (video game tournaments/casuals for
> Street Fighter 4, Marvel vs Capcom 3, etc).
>
>         Care to share?
>
>                 - Nick
>
>
>
>
>
>
>
> -----Original Message-----
> From: Ivan Brusic [mailto:[hidden email]]
> Sent: Tuesday, March 15, 2011 8:05 AM
> To: users
> Subject: SPAM-LOW: Re: Who uses ES in production ?
>
> I have been using ElasticSearch for a trivial application at work.
> Nothing too exciting, the volume of data is rather low (for now), but
> the REST interface allows querying from different applications.
>
> In the meantime, I built a websocket river for ElasticSearch.  I am
> currently indexing all RSVPs from Meetup.com. Why? Because. :)  I have
> been doing some rudimentary data mining using only ES and JSON calls!
> Top programming language (by the amount of RSVPs)?  PHP.  Top state
> for development meetups?  NY.  My plan is to have an HTML5 interface
> to present the data, but I am not a front-end guy and I have yet to
> start the work.
>
> Ivan
>
> On Mar 15, 4:52 am, David Pilato
> <[hidden email]> wrote:
> > Hello < the list > ;-)
>
> > I would like to know if someone uses Elastic Search in production
> > environment. If possible also to give me a feedback about the number of
> > document indexed.
>
> > It will be very useful for IT managers to know that.
>
> > Shay ? Do you have somewhere ES testimonials or guestbook ?
>
> > Thanks
>
> > David.
Reply | Threaded
Open this post in threaded view
|

Re: SPAM-LOW: Re: Who uses ES in production ?

kimchy
Administrator
i think google groups is having issues now..., seem to be getting the message through mail, but some messages are not showing up in the web google groups interface.

On Wednesday, March 16, 2011 at 8:27 PM, Ivan Brusic wrote:

Not sure why my previous response has yet to appear.

Indexing any content from a websocket into ElasticSearch is just a pet
project, which means I do not have much time to work on it! Here is
some code so far:

https://github.com/ibrusic/simple-websocket-river

It is still not complete, but it is functional. I have some code for
configurable index builders (bulk or not) that is not checked in.
Also missing are unit tests since testing for a plugin is tricky.
Need to abstract my classes a bit more first. The main limitation is
the build system. ElasticSearch plugins must be compiled against the
version you intend to run with and currently the version is hardcoded
inside the SBT project file. Once I finished with the code, I will
create pre-built jar files.

I will probably work on it more once 0.16 is released. 0.16 is a
requirement due to changes in the river api, so for now you must build
against jars from the snapshot repo and deploy with a server built
from sources.

The mapping for the meetup example is far from complete. I actually
set a few fields to be stored/non-indexded.

Ivan

On Mar 15, 9:14 am, "Administrator" <ad...@sf4answers.com> wrote:
Ivan,

        Actually, I'd be interested in the meetup river, as I would like to
find events of a particular nature (video game tournaments/casuals for
Street Fighter 4, Marvel vs Capcom 3, etc).

        Care to share?

                - Nick







-----Original Message-----
From: Ivan Brusic [[hidden email]_bru...@yahoo.com]
Sent: Tuesday, March 15, 2011 8:05 AM
To: users
Subject: SPAM-LOW: Re: Who uses ES in production ?

I have been using ElasticSearch for a trivial application at work.
Nothing too exciting, the volume of data is rather low (for now), but
the REST interface allows querying from different applications.

In the meantime, I built a websocket river for ElasticSearch.  I am
currently indexing all RSVPs from Meetup.com. Why? Because. :)  I have
been doing some rudimentary data mining using only ES and JSON calls!
Top programming language (by the amount of RSVPs)?  PHP.  Top state
for development meetups?  NY.  My plan is to have an HTML5 interface
to present the data, but I am not a front-end guy and I have yet to
start the work.

Ivan

On Mar 15, 4:52 am, David Pilato
<david.pil...@douane.finances.gouv.fr> wrote:
Hello < the list > ;-)

I would like to know if someone uses Elastic Search in production
environment. If possible also to give me a feedback about the number of
document indexed.

It will be very useful for IT managers to know that.

Shay ? Do you have somewhere ES testimonials or guestbook ?

Thanks

David.