Convert bulk request to json document and publish that document to ES as a seperate task

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Convert bulk request to json document and publish that document to ES as a seperate task

Manjula Piyumal
Hi,

I'm expecting to implement ES back statistic collecting application. For that I'm expecting to publish data using bulk API. I want to separate out request generating and request publishing tasks. First task is creating bulk request using JAVA API and write that request to a temp file as a JSON document and then request publisher publishes the requests which are in the files.
I have tried to write created bulk request using writeTo method as follows. It writes the request as a semi-json document containing some random weird bytes(resulting file is attached).
            
            FileOutputStream fileOutputStream = new FileOutputStream(new File("request.txt"));
            OutputStreamStreamOutput streamOutput = new OutputStreamStreamOutput(fileOutputStream);
            bulkRequestBuilder.request().writeTo(streamOutput);

I'm not sure whether I have missed something here. I want to understand what goes wrong while writing the request and is it possible to implement my application by separating request generating and publishing tasks as mentioned above. Any help would be appreciated.

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/690f5a0a-3c62-43f3-b662-259b4dce79e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

request.txt (419 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Convert bulk request to json document and publish that document to ES as a seperate task

joergprante@gmail.com
You are using the binary stream protocol of ES in the writeTo() method which is not appropriate for writing to files.

Once you added requests to a bulk request, you can not get your content back as JSON.

A better approach is to use an XContentBuilder with an OutputStream, and add the content to it, independent of BulkRequestBuilder.

Jörg

On Tue, Apr 28, 2015 at 8:21 PM, Manjula Piyumal <[hidden email]> wrote:
Hi,

I'm expecting to implement ES back statistic collecting application. For that I'm expecting to publish data using bulk API. I want to separate out request generating and request publishing tasks. First task is creating bulk request using JAVA API and write that request to a temp file as a JSON document and then request publisher publishes the requests which are in the files.
I have tried to write created bulk request using writeTo method as follows. It writes the request as a semi-json document containing some random weird bytes(resulting file is attached).
            
            FileOutputStream fileOutputStream = new FileOutputStream(new File("request.txt"));
            OutputStreamStreamOutput streamOutput = new OutputStreamStreamOutput(fileOutputStream);
            bulkRequestBuilder.request().writeTo(streamOutput);

I'm not sure whether I have missed something here. I want to understand what goes wrong while writing the request and is it possible to implement my application by separating request generating and publishing tasks as mentioned above. Any help would be appreciated.

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/690f5a0a-3c62-43f3-b662-259b4dce79e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHGAVSjOWX6%2BQ08pNUBWczf6ySYwGDm%2BXMsepMnsWysvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Convert bulk request to json document and publish that document to ES as a seperate task

Manjula Piyumal
Hi Jörg,

Thanks for the quick response. I'll try it in that way.

Thanks
Manjula

On Wed, Apr 29, 2015 at 12:32 AM, [hidden email] <[hidden email]> wrote:
You are using the binary stream protocol of ES in the writeTo() method which is not appropriate for writing to files.

Once you added requests to a bulk request, you can not get your content back as JSON.

A better approach is to use an XContentBuilder with an OutputStream, and add the content to it, independent of BulkRequestBuilder.

Jörg

On Tue, Apr 28, 2015 at 8:21 PM, Manjula Piyumal <[hidden email]> wrote:
Hi,

I'm expecting to implement ES back statistic collecting application. For that I'm expecting to publish data using bulk API. I want to separate out request generating and request publishing tasks. First task is creating bulk request using JAVA API and write that request to a temp file as a JSON document and then request publisher publishes the requests which are in the files.
I have tried to write created bulk request using writeTo method as follows. It writes the request as a semi-json document containing some random weird bytes(resulting file is attached).
            
            FileOutputStream fileOutputStream = new FileOutputStream(new File("request.txt"));
            OutputStreamStreamOutput streamOutput = new OutputStreamStreamOutput(fileOutputStream);
            bulkRequestBuilder.request().writeTo(streamOutput);

I'm not sure whether I have missed something here. I want to understand what goes wrong while writing the request and is it possible to implement my application by separating request generating and publishing tasks as mentioned above. Any help would be appreciated.

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/690f5a0a-3c62-43f3-b662-259b4dce79e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/RCz0JgIP5kY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHGAVSjOWX6%2BQ08pNUBWczf6ySYwGDm%2BXMsepMnsWysvA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFLAp0XqWcdKdnyrw3vg6%2Be4tCp%3DQY2MAYDrngKfEOMTkbpkUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.