Not getting Good Write Performance

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Not getting Good Write Performance

Ankit Jain
Hi All,

I am creating index using elasticsearch API but not getting good write speed.

Below is code that I used for creating client and record dump.
//creates Client in our case
if (localMode != null && localMode) {
             Settings settings = ImmutableSettings.settingsBuilder().put("node.local",true).build();
             client = nodeBuilder().client(true).node().client();            
        } else {
             Settings settings = ImmutableSettings.settingsBuilder()
                     .build();
             client = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress(
                     elasticSearchHost, elasticSearchPort));
        }
//Record Dumping Part
       Long MobileNumber= new Long(0);
                System.out.println(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes().length);
                 Long k=null,f=null;
                 k=new Date().getTime();
                
                for(Long i=(long) 0;i<10000;i++)
                {
                IndexResponse response2 = this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();            
              MobileNumber++;  
                }
                f=new Date().getTime();
                System.out.println("Indexing Speed = "+((240)/(f-k))+"  Time   "+ (f-k));


Guys, required you help to increase the write performance.

Thanks,
Regards,
Ankit Jain
Reply | Threaded
Open this post in threaded view
|

Re: Not getting Good Write Performance

dadoonet
Use the Bulk API: https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113





Le 28 janv. 2013 à 17:54, Ankit Jain <[hidden email]> a écrit :

Hi All,

I am creating index using elasticsearch API but not getting good write speed.

Below is code that I used for creating client and record dump.
//creates Client in our case
if (localMode != null && localMode) {
             Settings settings = ImmutableSettings.settingsBuilder().put("node.local",true).build();
             client = nodeBuilder().client(true).node().client();            
        } else {
             Settings settings = ImmutableSettings.settingsBuilder()
                     .build();
             client = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress(
                     elasticSearchHost, elasticSearchPort));
        }
//Record Dumping Part
       Long MobileNumber= new Long(0);
                System.out.println(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes().length);
                 Long k=null,f=null;
                 k=new Date().getTime();
                
                for(Long i=(long) 0;i<10000;i++)
                {
                IndexResponse response2 = this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();            
              MobileNumber++;  
                }
                f=new Date().getTime();
                System.out.println("Indexing Speed = "+((240)/(f-k))+"  Time   "+ (f-k));


Guys, required you help to increase the write performance.

Thanks,
Regards,
Ankit Jain

--
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Not getting Good Write Performance

ppearcy
I had heard at one point that using the Java API per document vs the bulk APIs there isn't much difference, perhaps that is (or never was the case).

Anyways, for that specific code chunk, you're calling actionGet on every call, which makes the call synchronous. Instead you should save off the action futures and then call the actionGet after they have all been submitted. 

I'd be curious to know how that performs compared to the bulk API.

Best Regards,
Paul

On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote:
Use the Bulk API: https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113





Le 28 janv. 2013 à 17:54, Ankit Jain <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ki0OmL5CwXMJ">ankitj...@...> a écrit :

Hi All,

I am creating index using elasticsearch API but not getting good write speed.

Below is code that I used for creating client and record dump.
//creates Client in our case
if (localMode != null && localMode) {
             Settings settings = ImmutableSettings.settingsBuilder().put("node.local",true).build();
             client = nodeBuilder().client(true).node().client();            
        } else {
             Settings settings = ImmutableSettings.settingsBuilder()
                     .build();
             client = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress(
                     elasticSearchHost, elasticSearchPort));
        }
//Record Dumping Part
       Long MobileNumber= new Long(0);
                System.out.println(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes().length);
                 Long k=null,f=null;
                 k=new Date().getTime();
                
                for(Long i=(long) 0;i<10000;i++)
                {
                IndexResponse response2 = this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();            
              MobileNumber++;  
                }
                f=new Date().getTime();
                System.out.println("Indexing Speed = "+((240)/(f-k))+"  Time   "+ (f-k));


Guys, required you help to increase the write performance.

Thanks,
Regards,
Ankit Jain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Not getting Good Write Performance

joergprante@gmail.com
There is a difference:

- writing 100 docs per single doc request: 100 packets to the cluster,
cluster must acknowledge the 100 packets on the wire = 100 roundtrips.

- writing 100 doc in a bulk request: 1 packet to the cluster and 1 back
to the client (with reasonable doc size), 1 roundtrip.

So you save huge communication overhead.

Saving actionGet's only creates thread congestion, each will still end
up in a single request and response on the wire.

Jörg

Am 28.01.13 21:23, schrieb ppearcy:

> I had heard at one point that using the Java API per document vs the
> bulk APIs there isn't much difference, perhaps that is (or never was
> the case).
>
> Anyways, for that specific code chunk, you're calling actionGet on
> every call, which makes the call synchronous. Instead you should save
> off the action futures and then call the actionGet after they have all
> been submitted.
>
> I'd be curious to know how that performs compared to the bulk API.
>
> Best Regards,
> Paul
>
> On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote:
>
>     Use the Bulk API:
>     https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113
>     <https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113>
>
>
>
>
>
>
>     Le 28 janv. 2013 à 17:54, Ankit Jain <[hidden email]
>     <javascript:>> a écrit :
>
>>     Hi All,
>>
>>     I am creating index using elasticsearch API but not getting good
>>     write speed.
>>
>>     Below is code that I used for creating client and record dump.
>>     //creates Client in our case
>>     if (localMode != null && localMode) {
>>     Settings settings =
>>     ImmutableSettings.settingsBuilder().put("node.local",true).build();
>>                  client = nodeBuilder().client(true).node().client();
>>             } else {
>>                  Settings settings = ImmutableSettings.settingsBuilder()
>>                          .build();
>>                  client = new
>>     TransportClient(settings).addTransportAddress(new
>>     InetSocketTransportAddress(
>>                          elasticSearchHost, elasticSearchPort));
>>             }
>>     //Record Dumping Part
>>     **Long MobileNumber= new Long(0);
>>                    
>>     System.out.println(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes().length);
>>                      Long k=null,f=null;
>>                      k=new Date().getTime();
>>
>>                     for(Long i=(long) 0;i<10000;i++)
>>                     {
>>                     IndexResponse response2 =
>>     this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();
>>                   MobileNumber++;
>>                     }
>>                     f=new Date().getTime();
>>     System.out.println("Indexing Speed = "+((240)/(f-k))+"  Time   "+
>>     (f-k));
>>
>>     Guys, required you help to increase the write performance.
>>
>>     Thanks,
>>     Regards,
>>     Ankit Jain
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group, send email to
> [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Not getting Good Write Performance

dadoonet
I add to Jorg answer that with bulk, Elasticsearch prepares documents for each shard. Then it has only to send the right docs to the right shard, index all, instead of doing this one by one.


--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 28 janv. 2013 à 21:32, Jörg Prante <[hidden email]> a écrit :

There is a difference:

- writing 100 docs per single doc request: 100 packets to the cluster, cluster must acknowledge the 100 packets on the wire = 100 roundtrips.

- writing 100 doc in a bulk request: 1 packet to the cluster and 1 back to the client (with reasonable doc size), 1 roundtrip.

So you save huge communication overhead.

Saving actionGet's only creates thread congestion, each will still end up in a single request and response on the wire.

Jörg

Am 28.01.13 21:23, schrieb ppearcy:

> I had heard at one point that using the Java API per document vs the bulk APIs there isn't much difference, perhaps that is (or never was the case).
>
> Anyways, for that specific code chunk, you're calling actionGet on every call, which makes the call synchronous. Instead you should save off the action futures and then call the actionGet after they have all been submitted.
>
> I'd be curious to know how that performs compared to the bulk API.
>
> Best Regards,
> Paul
>
> On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote:
>
>    Use the Bulk API:
>    https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113
>    <https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113>
>
>
>
>
>
>
>    Le 28 janv. 2013 à 17:54, Ankit Jain <[hidden email]
>    <javascript:>> a écrit :
>
>>    Hi All,
>>
>>    I am creating index using elasticsearch API but not getting good
>>    write speed.
>>
>>    Below is code that I used for creating client and record dump.
>>    //creates Client in our case
>>    if (localMode != null && localMode) {
>>    Settings settings =
>>    ImmutableSettings.settingsBuilder().put("node.local",true).build();
>>                 client = nodeBuilder().client(true).node().client();
>>            } else {
>>                 Settings settings = ImmutableSettings.settingsBuilder()
>>                         .build();
>>                 client = new
>>    TransportClient(settings).addTransportAddress(new
>>    InetSocketTransportAddress(
>>                         elasticSearchHost, elasticSearchPort));
>>            }
>>    //Record Dumping Part
>>    **Long MobileNumber= new Long(0);
>>                       System.out.println(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes().length);
>>                     Long k=null,f=null;
>>                     k=new Date().getTime();
>>
>>                    for(Long i=(long) 0;i<10000;i++)
>>                    {
>>                    IndexResponse response2 =
>>    this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();
>>                  MobileNumber++;
>>                    }
>>                    f=new Date().getTime();
>>    System.out.println("Indexing Speed = "+((240)/(f-k))+"  Time   "+
>>    (f-k));
>>
>>    Guys, required you help to increase the write performance.
>>
>>    Thanks,
>>    Regards,
>>    Ankit Jain
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group, send email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Not getting Good Write Performance

ppearcy
Yeah, that's valid point regarding the extra network I/O, it really comes down to how quickly you want data showing up in search and if it makes sense to batch items together depending on your use case. 

The thread congestion can be mitigated with a different threadpool type for indexing:
http://www.elasticsearch.org/guide/reference/modules/threadpool.html

Best Regards,
Paul

On Monday, January 28, 2013 1:37:37 PM UTC-7, David Pilato wrote:
I add to Jorg answer that with bulk, Elasticsearch prepares documents for each shard. Then it has only to send the right docs to the right shard, index all, instead of doing this one by one.


--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 28 janv. 2013 à 21:32, Jörg Prante <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="6-dkItgfzHYJ">joerg...@...> a écrit :

There is a difference:

- writing 100 docs per single doc request: 100 packets to the cluster, cluster must acknowledge the 100 packets on the wire = 100 roundtrips.

- writing 100 doc in a bulk request: 1 packet to the cluster and 1 back to the client (with reasonable doc size), 1 roundtrip.

So you save huge communication overhead.

Saving actionGet's only creates thread congestion, each will still end up in a single request and response on the wire.

Jörg

Am 28.01.13 21:23, schrieb ppearcy:

> I had heard at one point that using the Java API per document vs the bulk APIs there isn't much difference, perhaps that is (or never was the case).
>
> Anyways, for that specific code chunk, you're calling actionGet on every call, which makes the call synchronous. Instead you should save off the action futures and then call the actionGet after they have all been submitted.
>
> I'd be curious to know how that performs compared to the bulk API.
>
> Best Regards,
> Paul
>
> On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote:
>
>    Use the Bulk API:
>    https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113
>    <https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113>
>
>
>
>
>
>
>    Le 28 janv. 2013 à 17:54, Ankit Jain <[hidden email]
>    <javascript:>> a écrit :
>
>>    Hi All,
>>
>>    I am creating index using elasticsearch API but not getting good
>>    write speed.
>>
>>    Below is code that I used for creating client and record dump.
>>    //creates Client in our case
>>    if (localMode != null && localMode) {
>>    Settings settings =
>>    ImmutableSettings.settingsBuilder().put("node.local",true).build();
>>                 client = nodeBuilder().client(true).node().client();
>>            } else {
>>                 Settings settings = ImmutableSettings.settingsBuilder()
>>                         .build();
>>                 client = new
>>    TransportClient(settings).addTransportAddress(new
>>    InetSocketTransportAddress(
>>                         elasticSearchHost, elasticSearchPort));
>>            }
>>    //Record Dumping Part
>>    **Long MobileNumber= new Long(0);
>>                       System.out.println(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes().length);
>>                     Long k=null,f=null;
>>                     k=new Date().getTime();
>>
>>                    for(Long i=(long) 0;i<10000;i++)
>>                    {
>>                    IndexResponse response2 =
>>    this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();
>>                  MobileNumber++;
>>                    }
>>                    f=new Date().getTime();
>>    System.out.println("Indexing Speed = "+((240)/(f-k))+"  Time   "+
>>    (f-k));
>>
>>    Guys, required you help to increase the write performance.
>>
>>    Thanks,
>>    Regards,
>>    Ankit Jain
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="6-dkItgfzHYJ">elasticsearc...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="6-dkItgfzHYJ">elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.