Quantcast

unable to write data to elasticsearch using hadoop PIG

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

unable to write data to elasticsearch using hadoop PIG

siva mannem
I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic curl commands like XPUT and XGET to create some indices and retrieve the data in them.
i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
can anybody give me the syntax for giving JSON file as input for curl XPUT command?

my next issue is i copied  the following 4 elasticsearch-hadoop jar files
elasticsearch-hadoop-1.3.0.M2.jar          
elasticsearch-hadoop-1.3.0.M2-sources.jar
elasticsearch-hadoop-1.3.0.M2-javadoc.jar  
elasticsearch-hadoop-1.3.0.M2-yarn.jar

to  /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
and /usr/lib/gphd/pig/

i have the following json file j.json
++++++
{"k1":"v1" ,  "k2":"v2" , "k3":"v3"}
++++++++

in my_hdfs_path.

my pig script is write_data_to_es.pig
+++++++++++++
REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
++++++++++++++

when i run my pig script 
+++++++++
pig -x mapreduce  write_data_to_es.pig 
++++++++++++

i am getting following error
+++++++++
Input(s):
Failed to read data from "/my_hdfs_path/j.json"

Output(s):
Failed to produce result in "usa/ca"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1390436301987_0089


2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
Failed to read data from "/elastic_search/es_hadoop_test.json"

Output(s):
Failed to produce result in "mannem/siva"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1390436301987_0089

2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Encountered IOException. Out of nodes and retries; caught exception
Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
++++++++++++

i am using pivotal hadoop version (1.0.1)  which is basically apache hadoop (hadoop-2.0.2)
and pig version is 0.10.1
and elastic search version is 0.90.9

can anybody help me out here?
thank you so much in advance for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f4740e2a-868f-489c-8f9d-842c08ecddff%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unable to write data to elasticsearch using hadoop PIG

siva mannem


On Tuesday, March 4, 2014 9:32:55 PM UTC-8, siva mannem wrote:
I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic curl commands like XPUT and XGET to create some indices and retrieve the data in them.
i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
can anybody give me the syntax for giving JSON file as input for curl XPUT command?

my next issue is i copied  the following 4 elasticsearch-hadoop jar files
elasticsearch-hadoop-1.3.0.M2.jar          
elasticsearch-hadoop-1.3.0.M2-sources.jar
elasticsearch-hadoop-1.3.0.M2-javadoc.jar  
elasticsearch-hadoop-1.3.0.M2-yarn.jar

to  /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
and /usr/lib/gphd/pig/

i have the following json file j.json
++++++
{"k1":"v1" ,  "k2":"v2" , "k3":"v3"}
++++++++

in my_hdfs_path.

my pig script is write_data_to_es.pig
+++++++++++++
REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
++++++++++++++

when i run my pig script 
+++++++++
pig -x mapreduce  write_data_to_es.pig 
++++++++++++

i am getting following error
+++++++++
Input(s):
Failed to read data from "/my_hdfs_path/j.json"

Output(s):
Failed to produce result in "usa/ca"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1390436301987_0089


2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
Failed to read data from "/elastic_search/es_hadoop_test.json"

Output(s):
Failed to produce result in "usa/ca"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1390436301987_0089

2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Encountered IOException. Out of nodes and retries; caught exception
Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
++++++++++++

i am using pivotal hadoop version (1.0.1)  which is basically apache hadoop (hadoop-2.0.2)
and pig version is 0.10.1
and elastic search version is 0.90.9

can anybody help me out here?
thank you so much in advance for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7f32cae4-6ba4-4a57-889a-0cd826f69d09%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unable to write data to elasticsearch using hadoop PIG

Yann Barraud
In reply to this post by siva mannem
Hi,

Is your ES instance known by your Hadoop cluster (/etc/hosts) ?

It does not even seems to read in it.

Cheers,
Yann

Le mercredi 5 mars 2014 06:32:55 UTC+1, siva mannem a écrit :
I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic curl commands like XPUT and XGET to create some indices and retrieve the data in them.
i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
can anybody give me the syntax for giving JSON file as input for curl XPUT command?

my next issue is i copied  the following 4 elasticsearch-hadoop jar files
elasticsearch-hadoop-1.3.0.M2.jar          
elasticsearch-hadoop-1.3.0.M2-sources.jar
elasticsearch-hadoop-1.3.0.M2-javadoc.jar  
elasticsearch-hadoop-1.3.0.M2-yarn.jar

to  /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
and /usr/lib/gphd/pig/

i have the following json file j.json
++++++
{"k1":"v1" ,  "k2":"v2" , "k3":"v3"}
++++++++

in my_hdfs_path.

my pig script is write_data_to_es.pig
+++++++++++++
REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
++++++++++++++

when i run my pig script 
+++++++++
pig -x mapreduce  write_data_to_es.pig 
++++++++++++

i am getting following error
+++++++++
Input(s):
Failed to read data from "/my_hdfs_path/j.json"

Output(s):
Failed to produce result in "usa/ca"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1390436301987_0089


2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
Failed to read data from "/elastic_search/es_hadoop_test.json"

Output(s):
Failed to produce result in "mannem/siva"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1390436301987_0089

2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Encountered IOException. Out of nodes and retries; caught exception
Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
++++++++++++

i am using pivotal hadoop version (1.0.1)  which is basically apache hadoop (hadoop-2.0.2)
and pig version is 0.10.1
and elastic search version is 0.90.9

can anybody help me out here?
thank you so much in advance for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3cbde444-460e-4f14-bb16-f160feaf9a58%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unable to write data to elasticsearch using hadoop PIG

Costin Leau
In reply to this post by siva mannem
The error indicates that Pig cannot access Elasticsearch. Make sure that you specify the proper ES IP and port in your
configuration - the defaults localhost:9200 work only if you are running on a local node or if you have Elasticsearch
running on _each_ node of your Hadoop cluster.

Also when loading Json files, don't use JsonLoader but rather PigStorage - JsonLoader purpose is to transform JSON into
objects but that is not needed.
es-hadoop already does that and if you're data is in json, it will stream the data as is to ES.

Again, I recommend you go through the reference documentation:
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/pig.html
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapping.html


On 3/5/2014 7:32 AM, siva mannem wrote:

> I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic curl
> commands like XPUT and XGET to create some indices and retrieve the data in them.
> i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
> can anybody give me the syntax for giving JSON file as input for curl XPUT command?
>
> my next issue is i copied  the following 4 elasticsearch-hadoop jar files
> elasticsearch-hadoop-1.3.0.M2.jar
> elasticsearch-hadoop-1.3.0.M2-sources.jar
> elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> elasticsearch-hadoop-1.3.0.M2-yarn.jar
>
> to  /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
> and /usr/lib/gphd/pig/
>
> i have the following json file j.json
> ++++++
> {"k1":"v1" ,  "k2":"v2" , "k3":"v3"}
> ++++++++
>
> in my_hdfs_path.
>
> my pig script is write_data_to_es.pig
> +++++++++++++
> REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
> DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
> A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
> STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
> ++++++++++++++
>
> when i run my pig script
> +++++++++
> pig -x mapreduce  write_data_to_es.pig
> ++++++++++++
>
> i am getting following error
> +++++++++
> Input(s):
> Failed to read data from "/my_hdfs_path/j.json"
>
> Output(s):
> Failed to produce result in "usa/ca"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job_1390436301987_0089
>
>
> 2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
> Failed!
> 2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
> Failed to read data from "/elastic_search/es_hadoop_test.json"
>
> Output(s):
> Failed to produce result in "mannem/siva"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job_1390436301987_0089
>
> 2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
> Failed!
> 2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - *ERROR 2997: Encountered IOException. Out
> of nodes and retries; caught exception*
> Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
> ++++++++++++
>
> i am using pivotal hadoop version (1.0.1)  which is basically apache hadoop (hadoop-2.0.2)
> and pig version is 0.10.1
> and elastic search version is 0.90.9
>
> can anybody help me out here?
> thank you so much in advance for your help.
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> [hidden email] <mailto:[hidden email]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f4740e2a-868f-489c-8f9d-842c08ecddff%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/f4740e2a-868f-489c-8f9d-842c08ecddff%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53170168.5000909%40gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unable to write data to elasticsearch using hadoop PIG

siva mannem
In reply to this post by siva mannem
Yann and Costin,
thank you so much for your quick reply.
now i am able to write data to ES from PIG  and also read  data from ES using PIG.
i modified my DEFINE statement in PIG script as follows :
++++++++++
DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.nodes=gateway1,es.resource=usa/ca');
++++++++++++

On Tuesday, March 4, 2014 9:32:55 PM UTC-8, siva mannem wrote:
I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic curl commands like XPUT and XGET to create some indices and retrieve the data in them.
i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
can anybody give me the syntax for giving JSON file as input for curl XPUT command?

my next issue is i copied  the following 4 elasticsearch-hadoop jar files
elasticsearch-hadoop-1.3.0.M2.jar          
elasticsearch-hadoop-1.3.0.M2-sources.jar
elasticsearch-hadoop-1.3.0.M2-javadoc.jar  
elasticsearch-hadoop-1.3.0.M2-yarn.jar

to  /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
and /usr/lib/gphd/pig/

i have the following json file j.json
++++++
{"k1":"v1" ,  "k2":"v2" , "k3":"v3"}
++++++++

in my_hdfs_path.

my pig script is write_data_to_es.pig
+++++++++++++
REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
++++++++++++++

when i run my pig script 
+++++++++
pig -x mapreduce  write_data_to_es.pig 
++++++++++++

i am getting following error
+++++++++
Input(s):
Failed to read data from "/my_hdfs_path/j.json"

Output(s):
Failed to produce result in "usa/ca"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1390436301987_0089


2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
Failed to read data from "/elastic_search/es_hadoop_test.json"

Output(s):
Failed to produce result in "mannem/siva"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1390436301987_0089

2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Encountered IOException. Out of nodes and retries; caught exception
Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
++++++++++++

i am using pivotal hadoop version (1.0.1)  which is basically apache hadoop (hadoop-2.0.2)
and pig version is 0.10.1
and elastic search version is 0.90.9

can anybody help me out here?
thank you so much in advance for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/18960477-da43-4969-8571-02942ab77390%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unable to write data to elasticsearch using hadoop PIG

hanine
In reply to this post by Yann Barraud
I had get the same erreur but I don't know what I have to change in my "/etc/hosts"
thank you for your help

Le mercredi 5 mars 2014 09:39:46 UTC, Yann Barraud a écrit :
Hi,

Is your ES instance known by your Hadoop cluster (/etc/hosts) ?

It does not even seems to read in it.

Cheers,
Yann

Le mercredi 5 mars 2014 06:32:55 UTC+1, siva mannem a écrit :
I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic curl commands like XPUT and XGET to create some indices and retrieve the data in them.
i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
can anybody give me the syntax for giving JSON file as input for curl XPUT command?

my next issue is i copied  the following 4 elasticsearch-hadoop jar files
elasticsearch-hadoop-1.3.0.M2.jar          
elasticsearch-hadoop-1.3.0.M2-sources.jar
elasticsearch-hadoop-1.3.0.M2-javadoc.jar  
elasticsearch-hadoop-1.3.0.M2-yarn.jar

to  /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
and /usr/lib/gphd/pig/

i have the following json file j.json
++++++
{"k1":"v1" ,  "k2":"v2" , "k3":"v3"}
++++++++

in my_hdfs_path.

my pig script is write_data_to_es.pig
+++++++++++++
REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
++++++++++++++

when i run my pig script 
+++++++++
pig -x mapreduce  write_data_to_es.pig 
++++++++++++

i am getting following error
+++++++++
Input(s):
Failed to read data from "/my_hdfs_path/j.json"

Output(s):
Failed to produce result in "usa/ca"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1390436301987_0089


2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
Failed to read data from "/elastic_search/es_hadoop_test.json"

Output(s):
Failed to produce result in "mannem/siva"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1390436301987_0089

2014-03-05 00:26:50,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Encountered IOException. Out of nodes and retries; caught exception
Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
++++++++++++

i am using pivotal hadoop version (1.0.1)  which is basically apache hadoop (hadoop-2.0.2)
and pig version is 0.10.1
and elastic search version is 0.90.9

can anybody help me out here?
thank you so much in advance for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unable to write data to elasticsearch using hadoop PIG

Costin Leau
Check your network settings and make sure that the Hadoop nodes can communicate with the ES nodes.
If you install ES besides Hadoop itself, this shouldn't be a problem.
There are various way to check this - try ping, tracert, etc...

Please refer to your distro manual/documentation for more information about the configuration and setup.

Cheers,

On 5/12/14 3:42 PM, hanine haninne wrote:

> I had get the same erreur but I don't know what I have to change in my "/etc/hosts"
> thank you for your help
>
> Le mercredi 5 mars 2014 09:39:46 UTC, Yann Barraud a écrit :
>
>     Hi,
>
>     Is your ES instance known by your Hadoop cluster (/etc/hosts) ?
>
>     It does not even seems to read in it.
>
>     Cheers,
>     Yann
>
>     Le mercredi 5 mars 2014 06:32:55 UTC+1, siva mannem a écrit :
>
>         I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic
>         curl commands like XPUT and XGET to create some indices and retrieve the data in them.
>         i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
>         can anybody give me the syntax for giving JSON file as input for curl XPUT command?
>
>         my next issue is i copied  the following 4 elasticsearch-hadoop jar files
>         elasticsearch-hadoop-1.3.0.M2.jar
>         elasticsearch-hadoop-1.3.0.M2-sources.jar
>         elasticsearch-hadoop-1.3.0.M2-javadoc.jar
>         elasticsearch-hadoop-1.3.0.M2-yarn.jar
>
>         to  /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
>         and /usr/lib/gphd/pig/
>
>         i have the following json file j.json
>         ++++++
>         {"k1":"v1" ,  "k2":"v2" , "k3":"v3"}
>         ++++++++
>
>         in my_hdfs_path.
>
>         my pig script is write_data_to_es.pig
>         +++++++++++++
>         REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
>         DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
>         A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
>         STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
>         ++++++++++++++
>
>         when i run my pig script
>         +++++++++
>         pig -x mapreduce  write_data_to_es.pig
>         ++++++++++++
>
>         i am getting following error
>         +++++++++
>         Input(s):
>         Failed to read data from "/my_hdfs_path/j.json"
>
>         Output(s):
>         Failed to produce result in "usa/ca"
>
>         Counters:
>         Total records written : 0
>         Total bytes written : 0
>         Spillable Memory Manager spill count : 0
>         Total bags proactively spilled: 0
>         Total records proactively spilled: 0
>
>         Job DAG:
>         job_1390436301987_0089
>
>
>         2014-03-05 00:26:50,839 [main] INFO
>           org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
>         2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
>         Failed to read data from "/elastic_search/es_hadoop_test.json"
>
>         Output(s):
>         Failed to produce result in "mannem/siva"
>
>         Counters:
>         Total records written : 0
>         Total bytes written : 0
>         Spillable Memory Manager spill count : 0
>         Total bags proactively spilled: 0
>         Total records proactively spilled: 0
>
>         Job DAG:
>         job_1390436301987_0089
>
>         2014-03-05 00:26:50,839 [main] INFO
>           org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
>         2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - *ERROR 2997: Encountered
>         IOException. Out of nodes and retries; caught exception*
>         Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
>         ++++++++++++
>
>         i am using pivotal hadoop version (1.0.1)  which is basically apache hadoop (hadoop-2.0.2)
>         and pig version is 0.10.1
>         and elastic search version is 0.90.9
>
>         can anybody help me out here?
>         thank you so much in advance for your help.
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> [hidden email] <mailto:[hidden email]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5370E894.8070308%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unable to write data to elasticsearch using hadoop PIG

hanine
thank you so much for your quick reply,
Here is what I had done
1-instaled hadoop-1.2.1( pig-0.12.0 / hive-0.11.0 /...)
2-download Elasticsearch-1.0.1 and put it in the same file of hadoop
3-copied  the following 4 elasticsearch-hadoop jar files
elasticsearch-hadoop-1.3.0.M2.jar          
elasticsearch-hadoop-1.3.0.M2-sources.jar
elasticsearch-hadoop-1.3.0.M2-javadoc.jar  
elasticsearch-hadoop-1.3.0.M2-yarn.jar
to /pig and hadoop/lib
4- Add them in the PIG_CLASSPATH

knowing that when I take data from my Desktop and put it in elasticsearch using pig script it works very well, but when I try to get data from my HDFS it gives me that :

2014-05-12 23:16:31,765 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR: java.io.IOException: Out of nodes and retries; caught exception
2014-05-12 23:16:31,765 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2014-05-12 23:16:31,766 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion    PigVersion    UserId    StartedAt    FinishedAt    Features
1.2.1    0.12.0    hduser    2014-05-12 23:15:34    2014-05-12 23:16:31    GROUP_BY

Failed!

Failed Jobs:
JobId    Alias    Feature    Message    Outputs
job_201405122310_0001    weblog_count,weblog_group,weblogs    GROUP_BY,COMBINER    Message: Job failed! Error - # of failed Reduce Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201405122310_0001_r_000000    weblogs1/logs2,

Input(s):
Failed to read data from "/user/weblogs"

Output(s):
Failed to produce result in "weblogs1/logs2"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201405122310_0001


2014-05-12 23:16:31,766 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!


And here is the script :

weblogs = LOAD '/user/weblogs' USING PigStorage('\t')
AS (client_ip : chararray,
full_request_date : chararray,
day : int,
month : chararray,
month_num : int,
year : int,
hour : int,
minute : int,
second : int,
timezone : chararray,
http_verb : chararray,
uri : chararray,
http_status_code : chararray,
bytes_returned : chararray,
referrer : chararray,
user_agent : chararray
);
weblog_group = GROUP weblogs by (client_ip, year, month_num);
weblog_count = FOREACH weblog_group GENERATE group.client_ip, group.year, group.month_num, COUNT_STAR(weblogs) as pageviews;
STORE weblog_count INTO 'weblogs1/logs2' USING org.elasticsearch.hadoop.pig.EsStorage();


Le lundi 12 mai 2014 16:28:20 UTC+1, Costin Leau a écrit :
Check your network settings and make sure that the Hadoop nodes can communicate with the ES nodes.
If you install ES besides Hadoop itself, this shouldn't be a problem.
There are various way to check this - try ping, tracert, etc...

Please refer to your distro manual/documentation for more information about the configuration and setup.

Cheers,

On 5/12/14 3:42 PM, hanine haninne wrote:

> I had get the same erreur but I don't know what I have to change in my "/etc/hosts"
> thank you for your help
>
> Le mercredi 5 mars 2014 09:39:46 UTC, Yann Barraud a écrit :
>
>     Hi,
>
>     Is your ES instance known by your Hadoop cluster (/etc/hosts) ?
>
>     It does not even seems to read in it.
>
>     Cheers,
>     Yann
>
>     Le mercredi 5 mars 2014 06:32:55 UTC+1, siva mannem a écrit :
>
>         I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic
>         curl commands like XPUT and XGET to create some indices and retrieve the data in them.
>         i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
>         can anybody give me the syntax for giving JSON file as input for curl XPUT command?
>
>         my next issue is i copied  the following 4 elasticsearch-hadoop jar files
>         elasticsearch-hadoop-1.3.0.M2.jar
>         elasticsearch-hadoop-1.3.0.M2-sources.jar
>         elasticsearch-hadoop-1.3.0.M2-javadoc.jar
>         elasticsearch-hadoop-1.3.0.M2-yarn.jar
>
>         to  /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
>         and /usr/lib/gphd/pig/
>
>         i have the following json file j.json
>         ++++++
>         {"k1":"v1" ,  "k2":"v2" , "k3":"v3"}
>         ++++++++
>
>         in my_hdfs_path.
>
>         my pig script is write_data_to_es.pig
>         +++++++++++++
>         REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
>         DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
>         A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
>         STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
>         ++++++++++++++
>
>         when i run my pig script
>         +++++++++
>         pig -x mapreduce  write_data_to_es.pig
>         ++++++++++++
>
>         i am getting following error
>         +++++++++
>         Input(s):
>         Failed to read data from "/my_hdfs_path/j.json"
>
>         Output(s):
>         Failed to produce result in "usa/ca"
>
>         Counters:
>         Total records written : 0
>         Total bytes written : 0
>         Spillable Memory Manager spill count : 0
>         Total bags proactively spilled: 0
>         Total records proactively spilled: 0
>
>         Job DAG:
>         job_1390436301987_0089
>
>
>         2014-03-05 00:26:50,839 [main] INFO
>           org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
>         2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
>         Failed to read data from "/elastic_search/es_hadoop_test.json"
>
>         Output(s):
>         Failed to produce result in "mannem/siva"
>
>         Counters:
>         Total records written : 0
>         Total bytes written : 0
>         Spillable Memory Manager spill count : 0
>         Total bags proactively spilled: 0
>         Total records proactively spilled: 0
>
>         Job DAG:
>         job_1390436301987_0089
>
>         2014-03-05 00:26:50,839 [main] INFO
>           org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
>         2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - *ERROR 2997: Encountered
>         IOException. Out of nodes and retries; caught exception*
>         Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
>         ++++++++++++
>
>         i am using pivotal hadoop version (1.0.1)  which is basically apache hadoop (hadoop-2.0.2)
>         and pig version is 0.10.1
>         and elastic search version is 0.90.9
>
>         can anybody help me out here?
>         thank you so much in advance for your help.
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> <a href="javascript:" target="_blank" gdf-obfuscated-mailto="v0gpwpNQ50gJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com <mailto:<a href="javascript:" target="_blank" gdf-obfuscated-mailto="v0gpwpNQ50gJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearch+unsubscribe@...>.
> To view this discussion on the web visit
> <a href="https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com';return true;">https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com
> <<a href="https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cd9d3143-556a-43c8-9cfd-78b666db48b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unable to write data to elasticsearch using hadoop PIG

Costin Leau
I would recommend upgrading to the latest es-hadoop, 2.0 RC1.
Also considering reading [1]

Hope this helps,
[1] http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/troubleshooting.html

On 5/13/14 1:20 AM, hanine haninne wrote:

> thank you so much for your quick reply,
> Here is what I had done
> 1-instaled hadoop-1.2.1( pig-0.12.0 / hive-0.11.0 /...)
> 2-download Elasticsearch-1.0.1 and put it in the same file of hadoop
> 3-copied  the following 4 elasticsearch-hadoop jar files
> elasticsearch-hadoop-1.3.0.M2.jar
> elasticsearch-hadoop-1.3.0.M2-sources.jar
> elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> elasticsearch-hadoop-1.3.0.M2-yarn.jar
> to /pig and hadoop/lib
> 4- Add them in the PIG_CLASSPATH
>
> knowing that when I take data from my Desktop and put it in elasticsearch using pig script it works very well, but when
> I try to get data from my HDFS it gives me that :
>
> 2014-05-12 23:16:31,765 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR: java.io.IOException: Out of
> nodes and retries; caught exception
> 2014-05-12 23:16:31,765 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2014-05-12 23:16:31,766 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
>
> HadoopVersion    PigVersion    UserId    StartedAt    FinishedAt    Features
> 1.2.1    0.12.0    hduser    2014-05-12 23:15:34    2014-05-12 23:16:31    GROUP_BY
>
> Failed!
>
> Failed Jobs:
> JobId    Alias    Feature    Message    Outputs
> job_201405122310_0001    weblog_count,weblog_group,weblogs    GROUP_BY,COMBINER    Message: Job failed! Error - # of
> failed Reduce Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201405122310_0001_r_000000
> weblogs1/logs2,
>
> Input(s):
> Failed to read data from "/user/weblogs"
>
> Output(s):
> Failed to produce result in "weblogs1/logs2"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job_201405122310_0001
>
>
> 2014-05-12 23:16:31,766 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
> Failed!
>
>
> And here is the script :
>
> weblogs = LOAD '/user/weblogs' USING PigStorage('\t')
> AS (client_ip : chararray,
> full_request_date : chararray,
> day : int,
> month : chararray,
> month_num : int,
> year : int,
> hour : int,
> minute : int,
> second : int,
> timezone : chararray,
> http_verb : chararray,
> uri : chararray,
> http_status_code : chararray,
> bytes_returned : chararray,
> referrer : chararray,
> user_agent : chararray
> );
> weblog_group = GROUP weblogs by (client_ip, year, month_num);
> weblog_count = FOREACH weblog_group GENERATE group.client_ip, group.year, group.month_num, COUNT_STAR(weblogs) as pageviews;
> STORE weblog_count INTO 'weblogs1/logs2' USING org.elasticsearch.hadoop.pig.EsStorage();
>
>
> Le lundi 12 mai 2014 16:28:20 UTC+1, Costin Leau a écrit :
>
>     Check your network settings and make sure that the Hadoop nodes can communicate with the ES nodes.
>     If you install ES besides Hadoop itself, this shouldn't be a problem.
>     There are various way to check this - try ping, tracert, etc...
>
>     Please refer to your distro manual/documentation for more information about the configuration and setup.
>
>     Cheers,
>
>     On 5/12/14 3:42 PM, hanine haninne wrote:
>     > I had get the same erreur but I don't know what I have to change in my "/etc/hosts"
>     > thank you for your help
>     >
>     > Le mercredi 5 mars 2014 09:39:46 UTC, Yann Barraud a écrit :
>     >
>     >     Hi,
>     >
>     >     Is your ES instance known by your Hadoop cluster (/etc/hosts) ?
>     >
>     >     It does not even seems to read in it.
>     >
>     >     Cheers,
>     >     Yann
>     >
>     >     Le mercredi 5 mars 2014 06:32:55 UTC+1, siva mannem a écrit :
>     >
>     >         I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic
>     >         curl commands like XPUT and XGET to create some indices and retrieve the data in them.
>     >         i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
>     >         can anybody give me the syntax for giving JSON file as input for curl XPUT command?
>     >
>     >         my next issue is i copied  the following 4 elasticsearch-hadoop jar files
>     >         elasticsearch-hadoop-1.3.0.M2.jar
>     >         elasticsearch-hadoop-1.3.0.M2-sources.jar
>     >         elasticsearch-hadoop-1.3.0.M2-javadoc.jar
>     >         elasticsearch-hadoop-1.3.0.M2-yarn.jar
>     >
>     >         to  /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
>     >         and /usr/lib/gphd/pig/
>     >
>     >         i have the following json file j.json
>     >         ++++++
>     >         {"k1":"v1" ,  "k2":"v2" , "k3":"v3"}
>     >         ++++++++
>     >
>     >         in my_hdfs_path.
>     >
>     >         my pig script is write_data_to_es.pig
>     >         +++++++++++++
>     >         REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
>     >         DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
>     >         A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
>     >         STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
>     >         ++++++++++++++
>     >
>     >         when i run my pig script
>     >         +++++++++
>     >         pig -x mapreduce  write_data_to_es.pig
>     >         ++++++++++++
>     >
>     >         i am getting following error
>     >         +++++++++
>     >         Input(s):
>     >         Failed to read data from "/my_hdfs_path/j.json"
>     >
>     >         Output(s):
>     >         Failed to produce result in "usa/ca"
>     >
>     >         Counters:
>     >         Total records written : 0
>     >         Total bytes written : 0
>     >         Spillable Memory Manager spill count : 0
>     >         Total bags proactively spilled: 0
>     >         Total records proactively spilled: 0
>     >
>     >         Job DAG:
>     >         job_1390436301987_0089
>     >
>     >
>     >         2014-03-05 00:26:50,839 [main] INFO
>     >           org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
>     >         2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
>     >         Failed to read data from "/elastic_search/es_hadoop_test.json"
>     >
>     >         Output(s):
>     >         Failed to produce result in "mannem/siva"
>     >
>     >         Counters:
>     >         Total records written : 0
>     >         Total bytes written : 0
>     >         Spillable Memory Manager spill count : 0
>     >         Total bags proactively spilled: 0
>     >         Total records proactively spilled: 0
>     >
>     >         Job DAG:
>     >         job_1390436301987_0089
>     >
>     >         2014-03-05 00:26:50,839 [main] INFO
>     >           org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
>     >         2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - *ERROR 2997: Encountered
>     >         IOException. Out of nodes and retries; caught exception*
>     >         Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
>     >         ++++++++++++
>     >
>     >         i am using pivotal hadoop version (1.0.1)  which is basically apache hadoop (hadoop-2.0.2)
>     >         and pig version is 0.10.1
>     >         and elastic search version is 0.90.9
>     >
>     >         can anybody help me out here?
>     >         thank you so much in advance for your help.
>     >
>     > --
>     > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
>     > To unsubscribe from this group and stop receiving emails from it, send an email to
>     >[hidden email] <javascript:> <mailto:[hidden email] <javascript:>>.
>     > To view this discussion on the web visit
>     >https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com
>     <https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com>
>     > <https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>     > For more options, visithttps://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>     --
>     Costin
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> [hidden email] <mailto:[hidden email]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/cd9d3143-556a-43c8-9cfd-78b666db48b7%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/cd9d3143-556a-43c8-9cfd-78b666db48b7%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5371EBC9.40204%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unable to write data to elasticsearch using hadoop PIG

Sunitha
This post has NOT been accepted by the mailing list yet.
Hi Costin,

I have been trying to load the json data to elastic search from hdfs. I'm using pig to pump the data into the ES. But i couldnt load the same structure as it holds to the ES. Also is there a way to load the data to ES without specifying any field names. For example, my json dump has,

{A:"string",
B:"string",
C:"string"
}

Without specifying the fields A,B,C in the pig script during load statement, is it possible to store the exact json format into ES ?

Thanks,
Sunitha
Loading...