Apache pig / Twitter elephant bird Json parser ClassCastException

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}

I'm trying to parse a rather simple json file using Pig and the Twitter's elephant-bird library, but it turns into a very painfull debugging process.

The json has the following structure:

oid_id: (oid:chararray), 

bookmarks: {(

  oid_id:(oid:chararray),

  id:chararray,

  creator: chararray,

  position:chararray,

  creationdate:($ate:chararray)

  )},

lastaction:(date:chararray),

settings:(preferredlanguage:chararray),

userid:chararray

An example of row:

{"oid_id":{"oid":"573239f905474a686e2333f0"},"bookmarks":[{"id":"LEGONINX106W0079264","creator":"player","position":96,"creationdate":{"date":"2016-12-26T09:37:36.916Z"},"oid_id":{"oid":"5860e4e0ca6baf9032edc0d0"}},{"id":"ONEPERCENTMW0128677","creator":"player","position":0.08,"creationdate":{"date":"2018-12-18T15:42:33.956Z"},"oid_id":{"oid":"5c191569faf8474953758930"}}],"lastaction":{"date":"2018-12-18T15:42:28.107Z"},"settings":{"preferredlanguage":"vf","preferredvideoquality":"hd"},"userid":"ocs_32a6ad6dd242d5e3842f9211fd236723_1461773211"}

Here is my code (inspired by this tutorial:
https://acadgild.com/blog/determining-popular-hashtags-in-twitter-using-pig)

register /path/to/json-simple-1.1.1.jar 

register /path/to/elephant-bird-core-4.17.jar

register /path/to/elephant-bird-pig-4.17.jar

register /path/to/elephant-bird-hadoop-compat-4.17.jar

define JsonLoaderEB com.twitter.elephantbird.pig.load.JsonLoader;



A = LOAD 'file.json' USING JsonLoaderEB('-nestedLoad=true') as myMap;

describe A;

input_table: {
myMap: bytearray }

B = foreach A generate flatten(myMap#'bookmarks') as (bookmark:map);

describe B;

B: {
bookmark: map }

When we dump the above relation, we can see that all the data has been loaded successfully.

([{"oid_id":{"oid":"5860e4e0ca6baf9032edc0d0"},"creator":"player","creationdate":{"date":"2016-12-26T09:37:36.916Z"},"id":"LEGONINX106W0079264","position":96},{"oid_id":{"oid":"5c191569faf8474953758930"},"creator":"player","creationdate":{"date":"2018-12-18T15:42:33.956Z"},"id":"ONEPERCENTMW0128677","position":0.08}])

Now we extract creationdate, creator, id and position from bookmark.

C = foreach B generate bookmark#'creationdate' as date_fact, bookmark#'creator' as creator, bookmark#'id' as id, bookmark#'position' as position;

C: {
date_fact: bytearray,
creator: bytearray,
id: bytearray,
position: bytearray
}

Dumping the table gives the following error:

Pig Stack Trace

ERROR 1066: Unable to open iterator for alias C. Backend error : Vertex failed, vertexName=scope-41, vertexId=vertex_1542613138136_6721
88_2_00, diagnostics=[Task failed, taskId=task_1542613138136_672188_2_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error
while running task ( failure ) : attempt_1542613138136_672188_2_00_000000_0:org.apache.pig.backend.executionengine.ExecException: ERROR
0: Exception while executing (Name: C: Store(hdfs://sandbox/tmp/temp-1543074195/tmp277240455:org.apache.pig.impl.io.InterStorage) - sc
ope-40 Operator Key: scope-40): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POMapLookUp (
Name: POMapLookUp[bytearray] - scope-28 Operator Key: scope-28) children: null at [null[4,31]]]: java.lang.ClassCastException: java.lan
g.String cannot be cast to java.util.Map
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:315)
at org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POStoreTez.getNextTuple(POStoreTez.java:123)
at org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.runPipeline(PigProcessor.java:376)
at org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.run(PigProcessor.java:241)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POMapLookUp (Name: POMapLookUp[byt
earray] - scope-28 Operator Key: scope-28) children: null at [null[4,31]]]: java.lang.ClassCastException: java.lang.String cannot be ca
st to java.util.Map
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:364)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:406)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:323)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:305)
1,9Top

edited Jan 8 at 16:34

asked Jan 3 at 15:11

dams

411310

add a comment |

I'm trying to parse a rather simple json file using Pig and the Twitter's elephant-bird library, but it turns into a very painfull debugging process.

The json has the following structure:

oid_id: (oid:chararray), 

bookmarks: {(

  oid_id:(oid:chararray),

  id:chararray,

  creator: chararray,

  position:chararray,

  creationdate:($ate:chararray)

  )},

lastaction:(date:chararray),

settings:(preferredlanguage:chararray),

userid:chararray

An example of row:

{"oid_id":{"oid":"573239f905474a686e2333f0"},"bookmarks":[{"id":"LEGONINX106W0079264","creator":"player","position":96,"creationdate":{"date":"2016-12-26T09:37:36.916Z"},"oid_id":{"oid":"5860e4e0ca6baf9032edc0d0"}},{"id":"ONEPERCENTMW0128677","creator":"player","position":0.08,"creationdate":{"date":"2018-12-18T15:42:33.956Z"},"oid_id":{"oid":"5c191569faf8474953758930"}}],"lastaction":{"date":"2018-12-18T15:42:28.107Z"},"settings":{"preferredlanguage":"vf","preferredvideoquality":"hd"},"userid":"ocs_32a6ad6dd242d5e3842f9211fd236723_1461773211"}

Here is my code (inspired by this tutorial:
https://acadgild.com/blog/determining-popular-hashtags-in-twitter-using-pig)

register /path/to/json-simple-1.1.1.jar 

register /path/to/elephant-bird-core-4.17.jar

register /path/to/elephant-bird-pig-4.17.jar

register /path/to/elephant-bird-hadoop-compat-4.17.jar

define JsonLoaderEB com.twitter.elephantbird.pig.load.JsonLoader;



A = LOAD 'file.json' USING JsonLoaderEB('-nestedLoad=true') as myMap;

describe A;

input_table: {
myMap: bytearray }

B = foreach A generate flatten(myMap#'bookmarks') as (bookmark:map);

describe B;

B: {
bookmark: map }

When we dump the above relation, we can see that all the data has been loaded successfully.

([{"oid_id":{"oid":"5860e4e0ca6baf9032edc0d0"},"creator":"player","creationdate":{"date":"2016-12-26T09:37:36.916Z"},"id":"LEGONINX106W0079264","position":96},{"oid_id":{"oid":"5c191569faf8474953758930"},"creator":"player","creationdate":{"date":"2018-12-18T15:42:33.956Z"},"id":"ONEPERCENTMW0128677","position":0.08}])

Now we extract creationdate, creator, id and position from bookmark.

C = foreach B generate bookmark#'creationdate' as date_fact, bookmark#'creator' as creator, bookmark#'id' as id, bookmark#'position' as position;

C: {
date_fact: bytearray,
creator: bytearray,
id: bytearray,
position: bytearray
}

Dumping the table gives the following error:

Pig Stack Trace

edited Jan 8 at 16:34

asked Jan 3 at 15:11

dams

411310

add a comment |

I'm trying to parse a rather simple json file using Pig and the Twitter's elephant-bird library, but it turns into a very painfull debugging process.

The json has the following structure:

oid_id: (oid:chararray), 

bookmarks: {(

  oid_id:(oid:chararray),

  id:chararray,

  creator: chararray,

  position:chararray,

  creationdate:($ate:chararray)

  )},

lastaction:(date:chararray),

settings:(preferredlanguage:chararray),

userid:chararray

An example of row:

{"oid_id":{"oid":"573239f905474a686e2333f0"},"bookmarks":[{"id":"LEGONINX106W0079264","creator":"player","position":96,"creationdate":{"date":"2016-12-26T09:37:36.916Z"},"oid_id":{"oid":"5860e4e0ca6baf9032edc0d0"}},{"id":"ONEPERCENTMW0128677","creator":"player","position":0.08,"creationdate":{"date":"2018-12-18T15:42:33.956Z"},"oid_id":{"oid":"5c191569faf8474953758930"}}],"lastaction":{"date":"2018-12-18T15:42:28.107Z"},"settings":{"preferredlanguage":"vf","preferredvideoquality":"hd"},"userid":"ocs_32a6ad6dd242d5e3842f9211fd236723_1461773211"}

Here is my code (inspired by this tutorial:
https://acadgild.com/blog/determining-popular-hashtags-in-twitter-using-pig)

register /path/to/json-simple-1.1.1.jar 

register /path/to/elephant-bird-core-4.17.jar

register /path/to/elephant-bird-pig-4.17.jar

register /path/to/elephant-bird-hadoop-compat-4.17.jar

define JsonLoaderEB com.twitter.elephantbird.pig.load.JsonLoader;



A = LOAD 'file.json' USING JsonLoaderEB('-nestedLoad=true') as myMap;

describe A;

input_table: {
myMap: bytearray }

B = foreach A generate flatten(myMap#'bookmarks') as (bookmark:map);

describe B;

B: {
bookmark: map }

When we dump the above relation, we can see that all the data has been loaded successfully.

([{"oid_id":{"oid":"5860e4e0ca6baf9032edc0d0"},"creator":"player","creationdate":{"date":"2016-12-26T09:37:36.916Z"},"id":"LEGONINX106W0079264","position":96},{"oid_id":{"oid":"5c191569faf8474953758930"},"creator":"player","creationdate":{"date":"2018-12-18T15:42:33.956Z"},"id":"ONEPERCENTMW0128677","position":0.08}])

Now we extract creationdate, creator, id and position from bookmark.

C = foreach B generate bookmark#'creationdate' as date_fact, bookmark#'creator' as creator, bookmark#'id' as id, bookmark#'position' as position;

C: {
date_fact: bytearray,
creator: bytearray,
id: bytearray,
position: bytearray
}

Dumping the table gives the following error:

Pig Stack Trace

edited Jan 8 at 16:34

asked Jan 3 at 15:11

dams

411310

I'm trying to parse a rather simple json file using Pig and the Twitter's elephant-bird library, but it turns into a very painfull debugging process.

The json has the following structure:

oid_id: (oid:chararray), 

bookmarks: {(

  oid_id:(oid:chararray),

  id:chararray,

  creator: chararray,

  position:chararray,

  creationdate:($ate:chararray)

  )},

lastaction:(date:chararray),

settings:(preferredlanguage:chararray),

userid:chararray

An example of row:

{"oid_id":{"oid":"573239f905474a686e2333f0"},"bookmarks":[{"id":"LEGONINX106W0079264","creator":"player","position":96,"creationdate":{"date":"2016-12-26T09:37:36.916Z"},"oid_id":{"oid":"5860e4e0ca6baf9032edc0d0"}},{"id":"ONEPERCENTMW0128677","creator":"player","position":0.08,"creationdate":{"date":"2018-12-18T15:42:33.956Z"},"oid_id":{"oid":"5c191569faf8474953758930"}}],"lastaction":{"date":"2018-12-18T15:42:28.107Z"},"settings":{"preferredlanguage":"vf","preferredvideoquality":"hd"},"userid":"ocs_32a6ad6dd242d5e3842f9211fd236723_1461773211"}

Here is my code (inspired by this tutorial:
https://acadgild.com/blog/determining-popular-hashtags-in-twitter-using-pig)

register /path/to/json-simple-1.1.1.jar 

register /path/to/elephant-bird-core-4.17.jar

register /path/to/elephant-bird-pig-4.17.jar

register /path/to/elephant-bird-hadoop-compat-4.17.jar

define JsonLoaderEB com.twitter.elephantbird.pig.load.JsonLoader;



A = LOAD 'file.json' USING JsonLoaderEB('-nestedLoad=true') as myMap;

describe A;

input_table: {
myMap: bytearray }

B = foreach A generate flatten(myMap#'bookmarks') as (bookmark:map);

describe B;

B: {
bookmark: map }

When we dump the above relation, we can see that all the data has been loaded successfully.

([{"oid_id":{"oid":"5860e4e0ca6baf9032edc0d0"},"creator":"player","creationdate":{"date":"2016-12-26T09:37:36.916Z"},"id":"LEGONINX106W0079264","position":96},{"oid_id":{"oid":"5c191569faf8474953758930"},"creator":"player","creationdate":{"date":"2018-12-18T15:42:33.956Z"},"id":"ONEPERCENTMW0128677","position":0.08}])

Now we extract creationdate, creator, id and position from bookmark.

C = foreach B generate bookmark#'creationdate' as date_fact, bookmark#'creator' as creator, bookmark#'id' as id, bookmark#'position' as position;

C: {
date_fact: bytearray,
creator: bytearray,
id: bytearray,
position: bytearray
}

Dumping the table gives the following error:

Pig Stack Trace

json hadoop apache-pig elephantbird

edited Jan 8 at 16:34

asked Jan 3 at 15:11

dams

411310

edited Jan 8 at 16:34

asked Jan 3 at 15:11

dams

411310

edited Jan 8 at 16:34

asked Jan 3 at 15:11

dams

411310

asked Jan 3 at 15:11

dams

411310

asked Jan 3 at 15:11

dams

411310

add a comment |

1 Answer
1

active

oldest

votes

Even if it gives a good result for table_extraction relation, it can be from the raw data.

Can u please remove or correct the following object, it looks invalid :

"oid":"5c191393faf8475cb76ee0d5"

answered Jan 8 at 15:04

54l3d

3,16321638

I've corrected/simplified the example file, but the same error remains...

– dams
Jan 8 at 16:35

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54024982%2fapache-pig-twitter-elephant-bird-json-parser-classcastexception%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

Even if it gives a good result for table_extraction relation, it can be from the raw data.

Can u please remove or correct the following object, it looks invalid :

"oid":"5c191393faf8475cb76ee0d5"

answered Jan 8 at 15:04

54l3d

3,16321638

I've corrected/simplified the example file, but the same error remains...

– dams
Jan 8 at 16:35

add a comment |

Even if it gives a good result for table_extraction relation, it can be from the raw data.

Can u please remove or correct the following object, it looks invalid :

"oid":"5c191393faf8475cb76ee0d5"

answered Jan 8 at 15:04

54l3d

3,16321638

I've corrected/simplified the example file, but the same error remains...

– dams
Jan 8 at 16:35

add a comment |

Even if it gives a good result for table_extraction relation, it can be from the raw data.

Can u please remove or correct the following object, it looks invalid :

"oid":"5c191393faf8475cb76ee0d5"

answered Jan 8 at 15:04

54l3d

3,16321638

Even if it gives a good result for table_extraction relation, it can be from the raw data.

Can u please remove or correct the following object, it looks invalid :

"oid":"5c191393faf8475cb76ee0d5"

answered Jan 8 at 15:04

54l3d

3,16321638

answered Jan 8 at 15:04

54l3d

3,16321638

answered Jan 8 at 15:04

54l3d

3,16321638

answered Jan 8 at 15:04

54l3d

3,16321638

I've corrected/simplified the example file, but the same error remains...

– dams
Jan 8 at 16:35

add a comment |

I've corrected/simplified the example file, but the same error remains...

– dams
Jan 8 at 16:35

I've corrected/simplified the example file, but the same error remains...

– dams
Jan 8 at 16:35

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu