Mixing
Pandas calculation works for only first quarter of all rows
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I've got a dataset with about 4mil observations. I'm doing a few simple transformations. However, the first couple works fine for all 4m obs, but the last few only works for the first 1.2m rows and then return NaN for all remaining rows. I can't see anything different about the data in those rows that this would be the case. I'm wondering if it's a memory issue or something, based on how I've written my code.
Anyway here's a short snippet. The first transformation below works fine for all 4m rows, the second one only works up to 1.2m rows then throws NaN. Any ideas?
Thanks!
#CREATE VAR FOR NUMBER OF PPL WHO'VE CLIMBED EACH ROUTE (SENDERS)
senders = routes.groupby(['route_id'])['user_id'].transform('nunique')
routes['senders'] = senders
routes = routes.reset_index()
routes = routes.drop(['index'], axis=1)
#CREATE VAR FOR WEIGHTED AVG RATING
avg_rating = routes.groupby(['route_id'])['rating'].mean().astype('float64')
routes['avg_rating'] = avg_rating.astype('float64')
routes['war'] = routes.sends * routes.avg_rating
routes = routes.reset_index()
routes = routes.drop(['index'], axis=1)
pandas-groupby nan mean
edited Jan 3 at 6:26
Zain Farooq
2,06121030
asked Jan 3 at 4:16
KeesKees
5718
add a comment |
I've got a dataset with about 4mil observations. I'm doing a few simple transformations. However, the first couple works fine for all 4m obs, but the last few only works for the first 1.2m rows and then return NaN for all remaining rows. I can't see anything different about the data in those rows that this would be the case. I'm wondering if it's a memory issue or something, based on how I've written my code.
Anyway here's a short snippet. The first transformation below works fine for all 4m rows, the second one only works up to 1.2m rows then throws NaN. Any ideas?
Thanks!
#CREATE VAR FOR NUMBER OF PPL WHO'VE CLIMBED EACH ROUTE (SENDERS)
senders = routes.groupby(['route_id'])['user_id'].transform('nunique')
routes['senders'] = senders
routes = routes.reset_index()
routes = routes.drop(['index'], axis=1)
#CREATE VAR FOR WEIGHTED AVG RATING
avg_rating = routes.groupby(['route_id'])['rating'].mean().astype('float64')
routes['avg_rating'] = avg_rating.astype('float64')
routes['war'] = routes.sends * routes.avg_rating
routes = routes.reset_index()
routes = routes.drop(['index'], axis=1)
pandas-groupby nan mean
edited Jan 3 at 6:26
Zain Farooq
2,06121030
asked Jan 3 at 4:16
KeesKees
5718
add a comment |
I've got a dataset with about 4mil observations. I'm doing a few simple transformations. However, the first couple works fine for all 4m obs, but the last few only works for the first 1.2m rows and then return NaN for all remaining rows. I can't see anything different about the data in those rows that this would be the case. I'm wondering if it's a memory issue or something, based on how I've written my code.
Anyway here's a short snippet. The first transformation below works fine for all 4m rows, the second one only works up to 1.2m rows then throws NaN. Any ideas?
Thanks!
#CREATE VAR FOR NUMBER OF PPL WHO'VE CLIMBED EACH ROUTE (SENDERS)
senders = routes.groupby(['route_id'])['user_id'].transform('nunique')
routes['senders'] = senders
routes = routes.reset_index()
routes = routes.drop(['index'], axis=1)
#CREATE VAR FOR WEIGHTED AVG RATING
avg_rating = routes.groupby(['route_id'])['rating'].mean().astype('float64')
routes['avg_rating'] = avg_rating.astype('float64')
routes['war'] = routes.sends * routes.avg_rating
routes = routes.reset_index()
routes = routes.drop(['index'], axis=1)
pandas-groupby nan mean
edited Jan 3 at 6:26
Zain Farooq
2,06121030
asked Jan 3 at 4:16
KeesKees
5718
I've got a dataset with about 4mil observations. I'm doing a few simple transformations. However, the first couple works fine for all 4m obs, but the last few only works for the first 1.2m rows and then return NaN for all remaining rows. I can't see anything different about the data in those rows that this would be the case. I'm wondering if it's a memory issue or something, based on how I've written my code.
Anyway here's a short snippet. The first transformation below works fine for all 4m rows, the second one only works up to 1.2m rows then throws NaN. Any ideas?
Thanks!
#CREATE VAR FOR NUMBER OF PPL WHO'VE CLIMBED EACH ROUTE (SENDERS)
senders = routes.groupby(['route_id'])['user_id'].transform('nunique')
routes['senders'] = senders
routes = routes.reset_index()
routes = routes.drop(['index'], axis=1)
#CREATE VAR FOR WEIGHTED AVG RATING
avg_rating = routes.groupby(['route_id'])['rating'].mean().astype('float64')
routes['avg_rating'] = avg_rating.astype('float64')
routes['war'] = routes.sends * routes.avg_rating
routes = routes.reset_index()
routes = routes.drop(['index'], axis=1)
pandas-groupby nan mean
pandas-groupby nan mean
edited Jan 3 at 6:26
Zain Farooq
2,06121030
asked Jan 3 at 4:16
KeesKees
5718
edited Jan 3 at 6:26
Zain Farooq
2,06121030
asked Jan 3 at 4:16
KeesKees
5718
edited Jan 3 at 6:26
Zain Farooq
2,06121030
edited Jan 3 at 6:26
Zain Farooq
2,06121030
edited Jan 3 at 6:26
Zain Farooq
2,06121030
2,06121030
asked Jan 3 at 4:16
KeesKees
5718
asked Jan 3 at 4:16
KeesKees
5718
asked Jan 3 at 4:16
KeesKees
5718
5718
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54016275%2fpandas-calculation-works-for-only-first-quarter-of-all-rows%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54016275%2fpandas-calculation-works-for-only-first-quarter-of-all-rows%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
