Messages getting lost while insertion

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Messages getting lost while insertion

spokarna
Hi All,
I am writing messages into elasticsearch using a single thread, and using the call:
                transportClient.prepareIndex(index, type, id)
                    .setSource(node)
                    .execute()
                    .actionGet();
If I write messages continuously, I observe that some messages are lost somewhere in transit, and my query returns less entries than I write.

If I use elasticsearchhead UI, I see the status of the index as
test
size: 68.9mb (68.9mb)
docs: 42795 (42820)

My index status returns:
docs: {
  • num_docs: 42795
  • max_doc: 42820
  • deleted_docs: 25
}

The max_docs is less than the number of entries that I pushed into ElasticSearch. And why are some messages marked as deleted?
I am using ElasticSearch 0.16.2.


Regards,
Sameer



Reply | Threaded
Open this post in threaded view
|

Re: Messages getting lost while insertion

Ivan Brusic
Sameer,

Are you updating documents that already exist in the index? When you
update a document (insert a document with an existing id), then the
old document gets deleted and the new one inserted. The deleted
document will disappear once a merge occurs.

Ivan

On Tue, Sep 13, 2011 at 5:18 AM, Sameer Pokarna <[hidden email]> wrote:

> Hi All,
> I am writing messages into elasticsearch using a single thread, and using
> the call:
>                 transportClient.prepareIndex(index, type, id)
>                     .setSource(node)
>                     .execute()
>                     .actionGet();
> If I write messages continuously, I observe that some messages are lost
> somewhere in transit, and my query returns less entries than I write.
> If I use elasticsearchhead UI, I see the status of the index as
> test
> size: 68.9mb (68.9mb)
> docs: 42795 (42820)
> My index status returns:
> docs: {
>
> num_docs: 42795
> max_doc: 42820
> deleted_docs: 25
>
> }
> The max_docs is less than the number of entries that I pushed into
> ElasticSearch. And why are some messages marked as deleted?
> I am using ElasticSearch 0.16.2.
>
> Regards,
> Sameer
>
>
>