Mixing
Spark left_outer join is returning items exist in both list
I'm running spark-shell to compare 2 csv files. Each file has the same number of columns and all have 600,000 rows. I'm expecting the 2 files have all the same rows. Here is my script.
val a =
spark
.read
.option("header", "true")
.option("delimiter", "|")
.csv("/tmp/1.csv")
.drop("unwanted_column").
.cache()
val b =
spark
.read
.option("header", "true")
.option("delimiter", "|")
.csv("/tmp/2.csv")
.drop("unwanted_column")
.cache()
val c = a.join(b, Seq("id", "year"), "left_outer").cache()
c.count() // this is returning 600,000
Now I'm trying to find out the difference by randomly picking a line with the same id and year in 2 datasets a and b.
val a1 = a.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()
val b1 = b.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()
Then I try to compare each column in a1 and b1.
(0 to (a1.length -1)).foreach { i =>
if (a1.getString(i) != null && !a1.getString(i).equals(b1.getString(i))) {
System.out.println(i + " = " + a1.getString(i) + " = " + b1.getString(i))
}
}
It didn't print anything. In other words, there is no difference.
I can't tell why c.count() is returning 600,000 like that.
scala apache-spark
edited Nov 22 '18 at 2:46
Luis Miguel Mejía Suárez
2,6361822
asked Nov 22 '18 at 0:10
TrongBangTrongBang
326313
add a comment |
I'm running spark-shell to compare 2 csv files. Each file has the same number of columns and all have 600,000 rows. I'm expecting the 2 files have all the same rows. Here is my script.
val a =
spark
.read
.option("header", "true")
.option("delimiter", "|")
.csv("/tmp/1.csv")
.drop("unwanted_column").
.cache()
val b =
spark
.read
.option("header", "true")
.option("delimiter", "|")
.csv("/tmp/2.csv")
.drop("unwanted_column")
.cache()
val c = a.join(b, Seq("id", "year"), "left_outer").cache()
c.count() // this is returning 600,000
Now I'm trying to find out the difference by randomly picking a line with the same id and year in 2 datasets a and b.
val a1 = a.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()
val b1 = b.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()
Then I try to compare each column in a1 and b1.
(0 to (a1.length -1)).foreach { i =>
if (a1.getString(i) != null && !a1.getString(i).equals(b1.getString(i))) {
System.out.println(i + " = " + a1.getString(i) + " = " + b1.getString(i))
}
}
It didn't print anything. In other words, there is no difference.
I can't tell why c.count() is returning 600,000 like that.
scala apache-spark
edited Nov 22 '18 at 2:46
Luis Miguel Mejía Suárez
2,6361822
asked Nov 22 '18 at 0:10
TrongBangTrongBang
326313
Hi @TrongBang, can you be more specific about why it seems like a problem for you? I mean, both lines are equals - why that should imply a problem with thejoin?
– Luis Miguel Mejía Suárez
Nov 22 '18 at 0:37
1
sorry man, I was confused. Bad morning. Need some more coffee. :)
– TrongBang
Nov 22 '18 at 0:58
add a comment |
I'm running spark-shell to compare 2 csv files. Each file has the same number of columns and all have 600,000 rows. I'm expecting the 2 files have all the same rows. Here is my script.
val a =
spark
.read
.option("header", "true")
.option("delimiter", "|")
.csv("/tmp/1.csv")
.drop("unwanted_column").
.cache()
val b =
spark
.read
.option("header", "true")
.option("delimiter", "|")
.csv("/tmp/2.csv")
.drop("unwanted_column")
.cache()
val c = a.join(b, Seq("id", "year"), "left_outer").cache()
c.count() // this is returning 600,000
Now I'm trying to find out the difference by randomly picking a line with the same id and year in 2 datasets a and b.
val a1 = a.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()
val b1 = b.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()
Then I try to compare each column in a1 and b1.
(0 to (a1.length -1)).foreach { i =>
if (a1.getString(i) != null && !a1.getString(i).equals(b1.getString(i))) {
System.out.println(i + " = " + a1.getString(i) + " = " + b1.getString(i))
}
}
It didn't print anything. In other words, there is no difference.
I can't tell why c.count() is returning 600,000 like that.
scala apache-spark
edited Nov 22 '18 at 2:46
Luis Miguel Mejía Suárez
2,6361822
asked Nov 22 '18 at 0:10
TrongBangTrongBang
326313
I'm running spark-shell to compare 2 csv files. Each file has the same number of columns and all have 600,000 rows. I'm expecting the 2 files have all the same rows. Here is my script.
val a =
spark
.read
.option("header", "true")
.option("delimiter", "|")
.csv("/tmp/1.csv")
.drop("unwanted_column").
.cache()
val b =
spark
.read
.option("header", "true")
.option("delimiter", "|")
.csv("/tmp/2.csv")
.drop("unwanted_column")
.cache()
val c = a.join(b, Seq("id", "year"), "left_outer").cache()
c.count() // this is returning 600,000
Now I'm trying to find out the difference by randomly picking a line with the same id and year in 2 datasets a and b.
val a1 = a.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()
val b1 = b.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()
Then I try to compare each column in a1 and b1.
(0 to (a1.length -1)).foreach { i =>
if (a1.getString(i) != null && !a1.getString(i).equals(b1.getString(i))) {
System.out.println(i + " = " + a1.getString(i) + " = " + b1.getString(i))
}
}
It didn't print anything. In other words, there is no difference.
I can't tell why c.count() is returning 600,000 like that.
scala apache-spark
scala apache-spark
edited Nov 22 '18 at 2:46
Luis Miguel Mejía Suárez
2,6361822
asked Nov 22 '18 at 0:10
TrongBangTrongBang
326313
edited Nov 22 '18 at 2:46
Luis Miguel Mejía Suárez
2,6361822
asked Nov 22 '18 at 0:10
TrongBangTrongBang
326313
edited Nov 22 '18 at 2:46
Luis Miguel Mejía Suárez
2,6361822
edited Nov 22 '18 at 2:46
Luis Miguel Mejía Suárez
2,6361822
edited Nov 22 '18 at 2:46
Luis Miguel Mejía Suárez
2,6361822
2,6361822
asked Nov 22 '18 at 0:10
TrongBangTrongBang
326313
asked Nov 22 '18 at 0:10
TrongBangTrongBang
326313
asked Nov 22 '18 at 0:10
TrongBangTrongBang
326313
326313
Hi @TrongBang, can you be more specific about why it seems like a problem for you? I mean, both lines are equals - why that should imply a problem with thejoin?
– Luis Miguel Mejía Suárez
Nov 22 '18 at 0:37
1
sorry man, I was confused. Bad morning. Need some more coffee. :)
– TrongBang
Nov 22 '18 at 0:58
add a comment |
Hi @TrongBang, can you be more specific about why it seems like a problem for you? I mean, both lines are equals - why that should imply a problem with thejoin?
– Luis Miguel Mejía Suárez
Nov 22 '18 at 0:37
1
sorry man, I was confused. Bad morning. Need some more coffee. :)
– TrongBang
Nov 22 '18 at 0:58
Hi @TrongBang, can you be more specific about why it seems like a problem for you? I mean, both lines are equals - why that should imply a problem with the
join ?– Luis Miguel Mejía Suárez
Nov 22 '18 at 0:37
Hi @TrongBang, can you be more specific about why it seems like a problem for you? I mean, both lines are equals - why that should imply a problem with the
join ?– Luis Miguel Mejía Suárez
Nov 22 '18 at 0:37
1
1
sorry man, I was confused. Bad morning. Need some more coffee. :)
– TrongBang
Nov 22 '18 at 0:58
sorry man, I was confused. Bad morning. Need some more coffee. :)
– TrongBang
Nov 22 '18 at 0:58
add a comment |
1 Answer
1
active
oldest
votes
Sorry guys, I guess it was my fault. Actually I was after a.subtract(b). My purpose is to find out the difference between a and b. I was confused about left_outer join.
answered Nov 22 '18 at 0:57
TrongBangTrongBang
326313
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53422181%2fspark-left-outer-join-is-returning-items-exist-in-both-list%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sorry guys, I guess it was my fault. Actually I was after a.subtract(b). My purpose is to find out the difference between a and b. I was confused about left_outer join.
answered Nov 22 '18 at 0:57
TrongBangTrongBang
326313
add a comment |
Sorry guys, I guess it was my fault. Actually I was after a.subtract(b). My purpose is to find out the difference between a and b. I was confused about left_outer join.
answered Nov 22 '18 at 0:57
TrongBangTrongBang
326313
add a comment |
Sorry guys, I guess it was my fault. Actually I was after a.subtract(b). My purpose is to find out the difference between a and b. I was confused about left_outer join.
answered Nov 22 '18 at 0:57
TrongBangTrongBang
326313
Sorry guys, I guess it was my fault. Actually I was after a.subtract(b). My purpose is to find out the difference between a and b. I was confused about left_outer join.
answered Nov 22 '18 at 0:57
TrongBangTrongBang
326313
answered Nov 22 '18 at 0:57
TrongBangTrongBang
326313
answered Nov 22 '18 at 0:57
TrongBangTrongBang
326313
answered Nov 22 '18 at 0:57
TrongBangTrongBang
326313
326313
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53422181%2fspark-left-outer-join-is-returning-items-exist-in-both-list%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Hi @TrongBang, can you be more specific about why it seems like a problem for you? I mean, both lines are equals - why that should imply a problem with the
join?– Luis Miguel Mejía Suárez
Nov 22 '18 at 0:37
1
sorry man, I was confused. Bad morning. Need some more coffee. :)
– TrongBang
Nov 22 '18 at 0:58