problem with bulk insertion

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

problem with bulk insertion

vinod eligeti
I have an index which is of the format:

lk<UUID>:<timestamp>  where lk is a prefix and timestamp is stripped to a month

The embedded node i created has 1 shard and no replication which i gave something like this:

NodeBuilder nb = new NodeBuilder().clusterName("log-store");
nb.settings().put("node.data", true);
nb.settings().put("index.number_of_shards", 1);
nb.settings().put("index.number_of_replicas", 0);

If i create an index, below for example:

lk039dbfc1-95cd-42d2-8e9c-9799423466de:1372723200000

Then bulk insertion happens fine and happens quickly without much delay. However if the create an index without the hyphen like:

lk039dbfc195cd42d28e9c9799423466de:1372723200000

The bulk insertion throws an exception after waiting 1m. 

0]: index [lk0c646caeba0d4d2d84d9f3dbb0e018e3:1372723200000], type [docs], id [PPAfs1r0QMeBr7PFH0wsBQ], message [UnavailableShardsException[[lk0c646caeba0d4d2d84d9f3dbb0e018e3:1372723200000][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@b25f027]]

Can you explain why is that?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: problem with bulk insertion

joergprante@gmail.com
This does not work. Use Cluster Update API after creating index for
changing index.number_of_shards and index.number_of_replicas

Example:

         ImmutableSettings.Builder settingsBuilder =
ImmutableSettings.settingsBuilder();
         settingsBuilder.put("number_of_shards", 1);
         settingsBuilder.put("number_of_replicas", 0);
         UpdateSettingsRequest updateSettingsRequest = new
UpdateSettingsRequest("myindex")
                 .settings(settingsBuilder);
client.admin().indices().updateSettings(updateSettingsRequest).actionGet();

Where do you get your example code from?

Jörg

Am 03.07.13 00:21, schrieb vinod eligeti:
> NodeBuilder nb = new NodeBuilder().clusterName("log-store");
> nb.settings().put("node.data", true);
> nb.settings().put("index.number_of_shards", 1);
> nb.settings().put("index.number_of_replicas", 0);

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: problem with bulk insertion

vinod eligeti
Thanks however, I donot want to update the settings for each index but rather all indices. And another question I have is when I start the embedded ES node I want to start with some settings and not like update the settings as you have showed in the example.

BTW can you also explain why the bulk insertion is failing in the second approach?


On Wed, Jul 3, 2013 at 10:17 AM, Jörg Prante <[hidden email]> wrote:
This does not work. Use Cluster Update API after creating index for changing index.number_of_shards and index.number_of_replicas

Example:

        ImmutableSettings.Builder settingsBuilder = ImmutableSettings.settingsBuilder();
        settingsBuilder.put("number_of_shards", 1);
        settingsBuilder.put("number_of_replicas", 0);
        UpdateSettingsRequest updateSettingsRequest = new UpdateSettingsRequest("myindex")
                .settings(settingsBuilder);
client.admin().indices().updateSettings(updateSettingsRequest).actionGet();

Where do you get your example code from?

Jörg

Am 03.07.13 00:21, schrieb vinod eligeti:

NodeBuilder nb = new NodeBuilder().clusterName("log-store");
nb.settings().put("node.data", true);
nb.settings().put("index.number_of_shards", 1);
nb.settings().put("index.number_of_replicas", 0);

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/2oqe8KhrGeM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: problem with bulk insertion

joergprante@gmail.com
The settings for a node can be given like this:

Node node = NodeBuilder.nodeBuilder()
     .settings(ImmutableSettings.settingsBuilder()
         .put("index.number_of_shards",1)
         .put("index.number_of_replicas", 0)
         .build())
     .build()
     .clusterName("firstCluster")
     .node();

More flexible is ImmutableSettings in the index creation request.

CreateIndexRequest indexRequest = new CreateIndexRequest("myindex");
indexRequest.settings(ImmutableSettings.settingsBuilder()
     .put("index.number_of_shards",1)
     .put("index.number_of_replicas", 0)
     .build());
client.admin().indices().create(indexRequest).actionGet();

Another method is using config files.

Jörg

Am 03.07.13 23:36, schrieb vinod eligeti:
> Thanks however, I donot want to update the settings for each index but
> rather all indices. And another question I have is when I start the
> embedded ES node I want to start with some settings and not like
> update the settings as you have showed in the example.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: problem with bulk insertion

vinod eligeti
Thanks,

However I still see the same problem as I mentioned earlier in my first post.


On Thu, Jul 4, 2013 at 8:11 AM, Jörg Prante <[hidden email]> wrote:
The settings for a node can be given like this:

Node node = NodeBuilder.nodeBuilder()
    .settings(ImmutableSettings.settingsBuilder()
        .put("index.number_of_shards",1)
        .put("index.number_of_replicas", 0)
        .build())
    .build()
    .clusterName("firstCluster")
    .node();

More flexible is ImmutableSettings in the index creation request.

CreateIndexRequest indexRequest = new CreateIndexRequest("myindex");
indexRequest.settings(ImmutableSettings.settingsBuilder()
    .put("index.number_of_shards",1)
    .put("index.number_of_replicas", 0)
    .build());
client.admin().indices().create(indexRequest).actionGet();

Another method is using config files.

Jörg

Am 03.07.13 23:36, schrieb vinod eligeti:

Thanks however, I donot want to update the settings for each index but rather all indices. And another question I have is when I start the embedded ES node I want to start with some settings and not like update the settings as you have showed in the example.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/2oqe8KhrGeM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.