Mixing
Comparing 2 Scala 2D arrays: getting error: value sameElements is not a member of (String, String)
Hi Im trying the following operation in Scala:
I have 2 dataframes.
I want to compare their columns names and then column types.
I started by extracting the column names
Then I sorted the array
and finally printed it
val df1colArr = df1.dtypes
val df2colArr = df2.dtypes
Sorting.quickSort(df1colArr)
Sorting.quickSort(df2colArr)
println(df1colArr.deep.mkString("n"))
println(df2colArr.deep.mkString("n"))
The output looks like this:
(age,IntegerType)
(color,StringType)
(dealer_id,StringType)
(first_name,StringType)
(id,IntegerType)
(last_name,StringType)
(loyalty_score,StringType)
(model,StringType)
(purchase_date,TimestampType)
(purchase_price,StringType)
(rank_dr,IntegerType)
(service_date,TimestampType)
(vin_num,StringType)
(age,IntegerType)
(color,StringType)
(dealer_id,StringType)
(first_name,StringType)
(id,IntegerType)
(last_name,StringType)
(loyalty_score,IntegerType)
(model,StringType)
(purchase_date,TimestampType)
(purchase_price,StringType)
(rank_dr,IntegerType)
(repeat_likely,IntegerType)
(service_date,TimestampType)
(vin_num,StringType)
Next I have a simple utility to compare 2 arrays above based on their value at index 0:
val col_similar: ( Array[(String,String)], Array[(String, String)] )=> String
= (x,y) => {if (x(0).sameElements(y(0))) "similar" else "different"}
when I run the above code. I get the following error:
Error:(59, 105) value sameElements is not a member of (String, String)
val col_similar: ( Array[(String,String)], Array[(String, String)] ) => String
= (x,y) => {if (x(0).sameElements(y(0))) "similar" else "different"}
Please help me understand why this code wont work....
Thanks so much
scala apache-spark
asked Jan 1 at 17:57
banditKingbanditKing
94
add a comment |
Hi Im trying the following operation in Scala:
I have 2 dataframes.
I want to compare their columns names and then column types.
I started by extracting the column names
Then I sorted the array
and finally printed it
val df1colArr = df1.dtypes
val df2colArr = df2.dtypes
Sorting.quickSort(df1colArr)
Sorting.quickSort(df2colArr)
println(df1colArr.deep.mkString("n"))
println(df2colArr.deep.mkString("n"))
The output looks like this:
(age,IntegerType)
(color,StringType)
(dealer_id,StringType)
(first_name,StringType)
(id,IntegerType)
(last_name,StringType)
(loyalty_score,StringType)
(model,StringType)
(purchase_date,TimestampType)
(purchase_price,StringType)
(rank_dr,IntegerType)
(service_date,TimestampType)
(vin_num,StringType)
(age,IntegerType)
(color,StringType)
(dealer_id,StringType)
(first_name,StringType)
(id,IntegerType)
(last_name,StringType)
(loyalty_score,IntegerType)
(model,StringType)
(purchase_date,TimestampType)
(purchase_price,StringType)
(rank_dr,IntegerType)
(repeat_likely,IntegerType)
(service_date,TimestampType)
(vin_num,StringType)
Next I have a simple utility to compare 2 arrays above based on their value at index 0:
val col_similar: ( Array[(String,String)], Array[(String, String)] )=> String
= (x,y) => {if (x(0).sameElements(y(0))) "similar" else "different"}
when I run the above code. I get the following error:
Error:(59, 105) value sameElements is not a member of (String, String)
val col_similar: ( Array[(String,String)], Array[(String, String)] ) => String
= (x,y) => {if (x(0).sameElements(y(0))) "similar" else "different"}
Please help me understand why this code wont work....
Thanks so much
scala apache-spark
asked Jan 1 at 17:57
banditKingbanditKing
94
add a comment |
Hi Im trying the following operation in Scala:
I have 2 dataframes.
I want to compare their columns names and then column types.
I started by extracting the column names
Then I sorted the array
and finally printed it
val df1colArr = df1.dtypes
val df2colArr = df2.dtypes
Sorting.quickSort(df1colArr)
Sorting.quickSort(df2colArr)
println(df1colArr.deep.mkString("n"))
println(df2colArr.deep.mkString("n"))
The output looks like this:
(age,IntegerType)
(color,StringType)
(dealer_id,StringType)
(first_name,StringType)
(id,IntegerType)
(last_name,StringType)
(loyalty_score,StringType)
(model,StringType)
(purchase_date,TimestampType)
(purchase_price,StringType)
(rank_dr,IntegerType)
(service_date,TimestampType)
(vin_num,StringType)
(age,IntegerType)
(color,StringType)
(dealer_id,StringType)
(first_name,StringType)
(id,IntegerType)
(last_name,StringType)
(loyalty_score,IntegerType)
(model,StringType)
(purchase_date,TimestampType)
(purchase_price,StringType)
(rank_dr,IntegerType)
(repeat_likely,IntegerType)
(service_date,TimestampType)
(vin_num,StringType)
Next I have a simple utility to compare 2 arrays above based on their value at index 0:
val col_similar: ( Array[(String,String)], Array[(String, String)] )=> String
= (x,y) => {if (x(0).sameElements(y(0))) "similar" else "different"}
when I run the above code. I get the following error:
Error:(59, 105) value sameElements is not a member of (String, String)
val col_similar: ( Array[(String,String)], Array[(String, String)] ) => String
= (x,y) => {if (x(0).sameElements(y(0))) "similar" else "different"}
Please help me understand why this code wont work....
Thanks so much
scala apache-spark
asked Jan 1 at 17:57
banditKingbanditKing
94
Hi Im trying the following operation in Scala:
I have 2 dataframes.
I want to compare their columns names and then column types.
I started by extracting the column names
Then I sorted the array
and finally printed it
val df1colArr = df1.dtypes
val df2colArr = df2.dtypes
Sorting.quickSort(df1colArr)
Sorting.quickSort(df2colArr)
println(df1colArr.deep.mkString("n"))
println(df2colArr.deep.mkString("n"))
The output looks like this:
(age,IntegerType)
(color,StringType)
(dealer_id,StringType)
(first_name,StringType)
(id,IntegerType)
(last_name,StringType)
(loyalty_score,StringType)
(model,StringType)
(purchase_date,TimestampType)
(purchase_price,StringType)
(rank_dr,IntegerType)
(service_date,TimestampType)
(vin_num,StringType)
(age,IntegerType)
(color,StringType)
(dealer_id,StringType)
(first_name,StringType)
(id,IntegerType)
(last_name,StringType)
(loyalty_score,IntegerType)
(model,StringType)
(purchase_date,TimestampType)
(purchase_price,StringType)
(rank_dr,IntegerType)
(repeat_likely,IntegerType)
(service_date,TimestampType)
(vin_num,StringType)
Next I have a simple utility to compare 2 arrays above based on their value at index 0:
val col_similar: ( Array[(String,String)], Array[(String, String)] )=> String
= (x,y) => {if (x(0).sameElements(y(0))) "similar" else "different"}
when I run the above code. I get the following error:
Error:(59, 105) value sameElements is not a member of (String, String)
val col_similar: ( Array[(String,String)], Array[(String, String)] ) => String
= (x,y) => {if (x(0).sameElements(y(0))) "similar" else "different"}
Please help me understand why this code wont work....
Thanks so much
scala apache-spark
scala apache-spark
asked Jan 1 at 17:57
banditKingbanditKing
94
asked Jan 1 at 17:57
banditKingbanditKing
94
asked Jan 1 at 17:57
banditKingbanditKing
94
asked Jan 1 at 17:57
banditKingbanditKing
94
asked Jan 1 at 17:57
banditKingbanditKing
94
94
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
x(0) is a pair of strings. If you wanted to compare arrays of pairs x and y, then do it:
if (x sameElements y) ... else ...
By the way, I doubt that this approach will scale to actual datasets - collecting the entire dataframe to the master node is usually a bad idea. Maybe you can find some better ideas here.
answered Jan 1 at 20:41
Andrey TyukinAndrey Tyukin
29.3k42350
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53997687%2fcomparing-2-scala-2d-arrays-getting-error-value-sameelements-is-not-a-member-o%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
x(0) is a pair of strings. If you wanted to compare arrays of pairs x and y, then do it:
if (x sameElements y) ... else ...
By the way, I doubt that this approach will scale to actual datasets - collecting the entire dataframe to the master node is usually a bad idea. Maybe you can find some better ideas here.
answered Jan 1 at 20:41
Andrey TyukinAndrey Tyukin
29.3k42350
add a comment |
x(0) is a pair of strings. If you wanted to compare arrays of pairs x and y, then do it:
if (x sameElements y) ... else ...
By the way, I doubt that this approach will scale to actual datasets - collecting the entire dataframe to the master node is usually a bad idea. Maybe you can find some better ideas here.
answered Jan 1 at 20:41
Andrey TyukinAndrey Tyukin
29.3k42350
add a comment |
x(0) is a pair of strings. If you wanted to compare arrays of pairs x and y, then do it:
if (x sameElements y) ... else ...
By the way, I doubt that this approach will scale to actual datasets - collecting the entire dataframe to the master node is usually a bad idea. Maybe you can find some better ideas here.
answered Jan 1 at 20:41
Andrey TyukinAndrey Tyukin
29.3k42350
x(0) is a pair of strings. If you wanted to compare arrays of pairs x and y, then do it:
if (x sameElements y) ... else ...
By the way, I doubt that this approach will scale to actual datasets - collecting the entire dataframe to the master node is usually a bad idea. Maybe you can find some better ideas here.
answered Jan 1 at 20:41
Andrey TyukinAndrey Tyukin
29.3k42350
answered Jan 1 at 20:41
Andrey TyukinAndrey Tyukin
29.3k42350
answered Jan 1 at 20:41
Andrey TyukinAndrey Tyukin
29.3k42350
answered Jan 1 at 20:41
Andrey TyukinAndrey Tyukin
29.3k42350
29.3k42350
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53997687%2fcomparing-2-scala-2d-arrays-getting-error-value-sameelements-is-not-a-member-o%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
