Elasticsearch hadoop - java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Elasticsearch hadoop - java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text

moon soo Lee
Hi, 

I'm using elasticsearch 1.0 GA and elasticsearch-hadoop-1.3.0.M2.jar with hive 0.11

I created external table like

CREATE EXTERNAL TABLE estable (
    data STRUCT
<
        intmap
:Map<STRING, INT>
   
>        
)


and when i try to insert data into this table, i get this exception

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text
        at org
.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:829)
        at org
.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
        at org
.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
        at org
.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
        at org
.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
        at org
.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
        at org
.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1066)
        at org
.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:890)
        at org
.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:758)
        at org
.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
        at org
.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
        at org
.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:253)
       
... 3 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text
        at org
.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.copyObject(WritableStringObjectInspector.java:35)
        at org
.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:249)
        at org
.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:229)
        at brickhouse
.udf.collect.CollectUDAF$MapCollectUDAFEvaluator.putIntoSet(CollectUDAF.java:252)
        at brickhouse
.udf.collect.CollectUDAF$MapCollectUDAFEvaluator.iterate(CollectUDAF.java:222)
        at org
.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:159)
        at org
.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:663)
        at org
.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:916)
        at org
.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:758)
        at org
.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
       
... 14 more


If i create a external table with Map<STRING, STRING>, like

CREATE EXTERNAL TABLE estable (
    data STRUCT
<
        intmap
:Map<STRING, STRING>
   
>        
)

everything works fine

Is it bug? or did i do something wrong? How can i fix it?

Thanks for helps.

--
Best
moon

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/835efed7-8f36-4025-b17e-bcab3a22047c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch hadoop - java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text

Costin Leau
You're Hive mapping needs to match the Elasticsearch one - there's no type conversion applied between the two.

The ClassCastException indicates that you are trying to write an INT (IntWritable) to a STRING (Text) - hence when you
change your mapping to STRING, STRING things are working.
I assume you already have some data in Elasticsearch and you are trying to read it in Hive through es-hadoop. However
your ES mapping points to a String not an Integer hence the type issue.

Either fix your mapping in ES or load the data as a String and then in Hive do the type conversion (casting) to int.

On 2/23/2014 2:27 AM, moon soo Lee wrote:

> Hi,
>
> I'm using elasticsearch 1.0 GA and elasticsearch-hadoop-1.3.0.M2.jar with hive 0.11
>
> I created external table like
>
> |
> CREATE EXTERNAL TABLE estable (
>      data STRUCT<
>          intmap:Map<STRING,INT>
>>
> )
> |
>
>
> and when i try to insert data into this table, i get this exception
>
> |
> Causedby:org.apache.hadoop.hive.ql.metadata.HiveException:java.lang.ClassCastException:org.apache.hadoop.io.IntWritablecannot
> be cast to org.apache.hadoop.io.Text
>          at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:829)
>          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
>          at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
>          at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
>          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
>          at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
>          at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1066)
>          at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:890)
>          at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:758)
>          at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
>          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
>          at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:253)
> ...3more
> Causedby:java.lang.ClassCastException:org.apache.hadoop.io.IntWritablecannot be cast to org.apache.hadoop.io.Text
>          at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.copyObject(WritableStringObjectInspector.java:35)
>          at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:249)
>          at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:229)
>          at brickhouse.udf.collect.CollectUDAF$MapCollectUDAFEvaluator.putIntoSet(CollectUDAF.java:252)
>          at brickhouse.udf.collect.CollectUDAF$MapCollectUDAFEvaluator.iterate(CollectUDAF.java:222)
>          at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:159)
>          at org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:663)
>          at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:916)
>          at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:758)
>          at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
> ...14more
> |
>
>
> If i create a external table with Map<STRING, STRING>, like
>
> |
> CREATE EXTERNAL TABLE estable (
>      data STRUCT<
>          intmap:Map<STRING,STRING>
>>
> )
> |
>
> everything works fine
>
> Is it bug? or did i do something wrong? How can i fix it?
>
> Thanks for helps.
>
> --
> Best
> moon
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> [hidden email].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/835efed7-8f36-4025-b17e-bcab3a22047c%40googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/530A0D96.10708%40gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.