Apache Spark: grouping different rows together based on conditionals
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I am attempting to simplify a dataframe in Apache Spark (Python).
I have a dataframe like this
person X N A B C D
NCC1701 1 16309 false true false false
NCC1864 1 16309 false false true false
...
I want to group of each row's X & N, like groupBy('X','N'), but I want to get a count of how often each column A-D shows up, like false = 0 and true = 1 so I get a result like this
X N A B C D
1 16309 0 1 1 0
In short, I am attempting to group together columns X and N, and get sums for each "true" and "false" for each pair of X and N. If 'true' and 'false' were exact numerics, I might know how to do this, but I don't know how to get 'true' as 1, and 'false' as 0, and then get sums.
How can I group the different cells together for each X and N?
thanks for your time
apache-spark pyspark
asked Nov 27 '18 at 2:04
concon
1,2701922
add a comment |
I am attempting to simplify a dataframe in Apache Spark (Python).
I have a dataframe like this
person X N A B C D
NCC1701 1 16309 false true false false
NCC1864 1 16309 false false true false
...
I want to group of each row's X & N, like groupBy('X','N'), but I want to get a count of how often each column A-D shows up, like false = 0 and true = 1 so I get a result like this
X N A B C D
1 16309 0 1 1 0
In short, I am attempting to group together columns X and N, and get sums for each "true" and "false" for each pair of X and N. If 'true' and 'false' were exact numerics, I might know how to do this, but I don't know how to get 'true' as 1, and 'false' as 0, and then get sums.
How can I group the different cells together for each X and N?
thanks for your time
apache-spark pyspark
asked Nov 27 '18 at 2:04
concon
1,2701922
add a comment |
I am attempting to simplify a dataframe in Apache Spark (Python).
I have a dataframe like this
person X N A B C D
NCC1701 1 16309 false true false false
NCC1864 1 16309 false false true false
...
I want to group of each row's X & N, like groupBy('X','N'), but I want to get a count of how often each column A-D shows up, like false = 0 and true = 1 so I get a result like this
X N A B C D
1 16309 0 1 1 0
In short, I am attempting to group together columns X and N, and get sums for each "true" and "false" for each pair of X and N. If 'true' and 'false' were exact numerics, I might know how to do this, but I don't know how to get 'true' as 1, and 'false' as 0, and then get sums.
How can I group the different cells together for each X and N?
thanks for your time
apache-spark pyspark
asked Nov 27 '18 at 2:04
concon
1,2701922
I am attempting to simplify a dataframe in Apache Spark (Python).
I have a dataframe like this
person X N A B C D
NCC1701 1 16309 false true false false
NCC1864 1 16309 false false true false
...
I want to group of each row's X & N, like groupBy('X','N'), but I want to get a count of how often each column A-D shows up, like false = 0 and true = 1 so I get a result like this
X N A B C D
1 16309 0 1 1 0
In short, I am attempting to group together columns X and N, and get sums for each "true" and "false" for each pair of X and N. If 'true' and 'false' were exact numerics, I might know how to do this, but I don't know how to get 'true' as 1, and 'false' as 0, and then get sums.
How can I group the different cells together for each X and N?
thanks for your time
apache-spark pyspark
apache-spark pyspark
asked Nov 27 '18 at 2:04
concon
1,2701922
asked Nov 27 '18 at 2:04
concon
1,2701922
asked Nov 27 '18 at 2:04
concon
1,2701922
asked Nov 27 '18 at 2:04
concon
1,2701922
asked Nov 27 '18 at 2:04
concon
1,2701922
1,2701922
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Use the cast method to convert the data type from boolean to integer, and then do the sum:
import pyspark.sql.functions as f
cols = ['A', 'B', 'C', 'D']
df.groupBy('X', 'N').agg(*(f.sum(f.col(x).cast('int')).alias(x) for x in cols)).show()
+---+-----+---+---+---+---+
| X| N| A| B| C| D|
+---+-----+---+---+---+---+
| 1|16309| 0| 1| 1| 0|
+---+-----+---+---+---+---+
answered Nov 27 '18 at 2:44
PsidomPsidom
128k1294142
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53491733%2fapache-spark-grouping-different-rows-together-based-on-conditionals%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Use the cast method to convert the data type from boolean to integer, and then do the sum:
import pyspark.sql.functions as f
cols = ['A', 'B', 'C', 'D']
df.groupBy('X', 'N').agg(*(f.sum(f.col(x).cast('int')).alias(x) for x in cols)).show()
+---+-----+---+---+---+---+
| X| N| A| B| C| D|
+---+-----+---+---+---+---+
| 1|16309| 0| 1| 1| 0|
+---+-----+---+---+---+---+
answered Nov 27 '18 at 2:44
PsidomPsidom
128k1294142
add a comment |
Use the cast method to convert the data type from boolean to integer, and then do the sum:
import pyspark.sql.functions as f
cols = ['A', 'B', 'C', 'D']
df.groupBy('X', 'N').agg(*(f.sum(f.col(x).cast('int')).alias(x) for x in cols)).show()
+---+-----+---+---+---+---+
| X| N| A| B| C| D|
+---+-----+---+---+---+---+
| 1|16309| 0| 1| 1| 0|
+---+-----+---+---+---+---+
answered Nov 27 '18 at 2:44
PsidomPsidom
128k1294142
add a comment |
Use the cast method to convert the data type from boolean to integer, and then do the sum:
import pyspark.sql.functions as f
cols = ['A', 'B', 'C', 'D']
df.groupBy('X', 'N').agg(*(f.sum(f.col(x).cast('int')).alias(x) for x in cols)).show()
+---+-----+---+---+---+---+
| X| N| A| B| C| D|
+---+-----+---+---+---+---+
| 1|16309| 0| 1| 1| 0|
+---+-----+---+---+---+---+
answered Nov 27 '18 at 2:44
PsidomPsidom
128k1294142
Use the cast method to convert the data type from boolean to integer, and then do the sum:
import pyspark.sql.functions as f
cols = ['A', 'B', 'C', 'D']
df.groupBy('X', 'N').agg(*(f.sum(f.col(x).cast('int')).alias(x) for x in cols)).show()
+---+-----+---+---+---+---+
| X| N| A| B| C| D|
+---+-----+---+---+---+---+
| 1|16309| 0| 1| 1| 0|
+---+-----+---+---+---+---+
answered Nov 27 '18 at 2:44
PsidomPsidom
128k1294142
answered Nov 27 '18 at 2:44
PsidomPsidom
128k1294142
answered Nov 27 '18 at 2:44
PsidomPsidom
128k1294142
answered Nov 27 '18 at 2:44
PsidomPsidom
128k1294142
128k1294142
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53491733%2fapache-spark-grouping-different-rows-together-based-on-conditionals%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
