How to increase indexing speed after configuring routing on an index?

[eldilibra]

I have a cluster with 8 nodes, and an index with 8 shards and 2 replica. After using this setup in production for a few days, we had a huge traffic spike, and CPU increased dangerously for all nodes. We were not using custom routing yet, however, so once I used a field that most of the queries were targeting as the routing field, CPU went way down, since the nodes weren't chattering as much, I think. Shortly after making this change, we realized that indexing a single document was taking a full 60 seconds. I understand that the node which holds the shard that stores a bunch of documents with the same value in their routing field will get "hot", but is this really going to make indexing that slow? If so, is their any setting or configuration within the cluster I can change in order to speed things up? Any help is appreciated, and I'm happy to answer any questions about our configuration.

[Jörg Prante]

Did you notice anything in the logs? Exceptions?

 From what it seems, your cluster ran into a critical low memory
condition, and could not continue.  So, operations tend to receive
timeouts.

In such cases, I restart nodes. Be sure to take precautions that you
won't get hit again by huge traffic (this is usually a traffic shaping
thing outside the scope of Elasticsearch)

Best regards,

Jörg

Am 14.02.13 20:53, schrieb eldilibra:

> I have a cluster with 8 nodes, and an index with 8 shards and 2 replica.
> After using this setup in production for a few days, we had a huge traffic
> spike, and CPU increased dangerously for all nodes. We were not using custom
> routing yet, however, so once I used a field that most of the queries were
> targeting as the routing field, CPU went way down, since the nodes weren't
> chattering as much, I think. Shortly after making this change, we realized
> that indexing a single document was taking a full 60 seconds. I understand
> that the node which holds the shard that stores a bunch of documents with
> the same value in their routing field will get "hot", but is this really
> going to make indexing that slow? If so, is their any setting or
> configuration within the cluster I can change in order to speed things up?
> Any help is appreciated, and I'm happy to answer any questions about our
> configuration.
>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.

[eldilibra]

Thanks for your response Jörg! We actually had to create an entirely new cluster, since the traffic spike ultimately led to us losing data (our fault), so the logs aren't very helpful in this case. Right now, I'm seeing no errors related to indexing in any of the nodes' logs.  

However, I've restarted a few suspicious nodes, and things are humming along nicely :~).

Cheers,

-- 

L.D Libra

Sent with Sparrow

On Thursday, February 14, 2013 at 4:27 PM, Jörg Prante [via ElasticSearch Users] wrote:

Did you notice anything in the logs? Exceptions?

 From what it seems, your cluster ran into a critical low memory
condition, and could not continue.  So, operations tend to receive
timeouts.

In such cases, I restart nodes. Be sure to take precautions that you
won't get hit again by huge traffic (this is usually a traffic shaping
thing outside the scope of Elasticsearch)

Best regards,

Jörg

Am 14.02.13 20:53, schrieb eldilibra:

> I have a cluster with 8 nodes, and an index with 8 shards and 2 replica.
> After using this setup in production for a few days, we had a huge traffic
> spike, and CPU increased dangerously for all nodes. We were not using custom
> routing yet, however, so once I used a field that most of the queries were
> targeting as the routing field, CPU went way down, since the nodes weren't
> chattering as much, I think. Shortly after making this change, we realized
> that indexing a single document was taking a full 60 seconds. I understand
> that the node which holds the shard that stores a bunch of documents with
> the same value in their routing field will get "hot", but is this really
> going to make indexing that slow? If so, is their any setting or
> configuration within the cluster I can change in order to speed things up?
> Any help is appreciated, and I'm happy to answer any questions about our
> configuration.
>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.

---

If you reply to this email, your message will be added to the discussion below:

http://elasticsearch-users.115913.n3.nabble.com/How-to-increase-indexing-speed-after-configuring-routing-on-an-index-tp4029846p4029848.html

To unsubscribe from How to increase indexing speed after configuring routing on an index?, click here.
NAML