Mixing
Vectorize only some arguments in function when calculating on Pandas dataframe
I have written a function that is meant to calculate a new dataframe column based on two other columns, as a few points of data from another dataframe. I want to apply this function in a vectorized way to the main dataframe, such that the 2 column inputs are calculated in such a way. At the same time, I want the third argument to be a constant dataframe that is used for a separate interpolation calculation (i.e not vectorized). How can this be accomplished?
Main function (as an example):
def calc_fitted_values(L, option, df_ref):
'''
This calculates an outputval for each combination of L and option, based
on intermediate calculations involving fitted values from df_ref.
- L is some column in my main dataframe
- option is a second column in the main dataframe
- df_ref is a separate data frame used in the pre-calculations here
'''
df_ref_option = df_ref[df_ref['option']==option] # take slice of df_ref based on option
x = df_ref_option['x'].values # get data columns to be used for polyfit
y = df_ref_option['y'].values
C = np.polyfit(np.log(x), np.log(y), 1); # use polyfit to get log fit of the reference data
a = np.exp(C[1]);
b = C[0];
outputval = a*(L**b)
return outputval
Usage wanted from function:
df['outputval']] = calc_fitted_values(df['L'], df['option'], df_ref)
In this example, L and option will be array values obtained from my main data frame columns (df), but df_ref is unrelated in in terms of shape and size.
How can I best write a function for this type of situation?
Thanks.
EDIT: My current "solution" is to use lambda...
f = lambda L, option : calc_fitted_values(L, option, df_ref)
df['outputval'] = np.vectorize(f)(df['L'].values, df['option'].values)
But this appears to be very slow. Might be due to the calculation each time through with df_ref, so would it be better to have a function that returns a lambda-defined function? Not sure of the best approach to this.
python pandas numpy dataframe vectorization
asked Nov 19 '18 at 21:38
teepeeteepee
7021819
add a comment |
I have written a function that is meant to calculate a new dataframe column based on two other columns, as a few points of data from another dataframe. I want to apply this function in a vectorized way to the main dataframe, such that the 2 column inputs are calculated in such a way. At the same time, I want the third argument to be a constant dataframe that is used for a separate interpolation calculation (i.e not vectorized). How can this be accomplished?
Main function (as an example):
def calc_fitted_values(L, option, df_ref):
'''
This calculates an outputval for each combination of L and option, based
on intermediate calculations involving fitted values from df_ref.
- L is some column in my main dataframe
- option is a second column in the main dataframe
- df_ref is a separate data frame used in the pre-calculations here
'''
df_ref_option = df_ref[df_ref['option']==option] # take slice of df_ref based on option
x = df_ref_option['x'].values # get data columns to be used for polyfit
y = df_ref_option['y'].values
C = np.polyfit(np.log(x), np.log(y), 1); # use polyfit to get log fit of the reference data
a = np.exp(C[1]);
b = C[0];
outputval = a*(L**b)
return outputval
Usage wanted from function:
df['outputval']] = calc_fitted_values(df['L'], df['option'], df_ref)
In this example, L and option will be array values obtained from my main data frame columns (df), but df_ref is unrelated in in terms of shape and size.
How can I best write a function for this type of situation?
Thanks.
EDIT: My current "solution" is to use lambda...
f = lambda L, option : calc_fitted_values(L, option, df_ref)
df['outputval'] = np.vectorize(f)(df['L'].values, df['option'].values)
But this appears to be very slow. Might be due to the calculation each time through with df_ref, so would it be better to have a function that returns a lambda-defined function? Not sure of the best approach to this.
python pandas numpy dataframe vectorization
asked Nov 19 '18 at 21:38
teepeeteepee
7021819
one possible solution: torch.nn.CrossEntropyLoss discuss.pytorch.org/t/…
– teng
Nov 19 '18 at 21:57
If you usenp.vectorize(for convenience, not speed), try theexcludeparameter. And don't neglect theotypesparameter.
– hpaulj
Nov 19 '18 at 22:32
the parameteroptionis not used ...
– B. M.
Nov 20 '18 at 11:59
@B.M. sorry, fixed it
– teepee
Nov 20 '18 at 16:48
add a comment |
I have written a function that is meant to calculate a new dataframe column based on two other columns, as a few points of data from another dataframe. I want to apply this function in a vectorized way to the main dataframe, such that the 2 column inputs are calculated in such a way. At the same time, I want the third argument to be a constant dataframe that is used for a separate interpolation calculation (i.e not vectorized). How can this be accomplished?
Main function (as an example):
def calc_fitted_values(L, option, df_ref):
'''
This calculates an outputval for each combination of L and option, based
on intermediate calculations involving fitted values from df_ref.
- L is some column in my main dataframe
- option is a second column in the main dataframe
- df_ref is a separate data frame used in the pre-calculations here
'''
df_ref_option = df_ref[df_ref['option']==option] # take slice of df_ref based on option
x = df_ref_option['x'].values # get data columns to be used for polyfit
y = df_ref_option['y'].values
C = np.polyfit(np.log(x), np.log(y), 1); # use polyfit to get log fit of the reference data
a = np.exp(C[1]);
b = C[0];
outputval = a*(L**b)
return outputval
Usage wanted from function:
df['outputval']] = calc_fitted_values(df['L'], df['option'], df_ref)
In this example, L and option will be array values obtained from my main data frame columns (df), but df_ref is unrelated in in terms of shape and size.
How can I best write a function for this type of situation?
Thanks.
EDIT: My current "solution" is to use lambda...
f = lambda L, option : calc_fitted_values(L, option, df_ref)
df['outputval'] = np.vectorize(f)(df['L'].values, df['option'].values)
But this appears to be very slow. Might be due to the calculation each time through with df_ref, so would it be better to have a function that returns a lambda-defined function? Not sure of the best approach to this.
python pandas numpy dataframe vectorization
asked Nov 19 '18 at 21:38
teepeeteepee
7021819
I have written a function that is meant to calculate a new dataframe column based on two other columns, as a few points of data from another dataframe. I want to apply this function in a vectorized way to the main dataframe, such that the 2 column inputs are calculated in such a way. At the same time, I want the third argument to be a constant dataframe that is used for a separate interpolation calculation (i.e not vectorized). How can this be accomplished?
Main function (as an example):
def calc_fitted_values(L, option, df_ref):
'''
This calculates an outputval for each combination of L and option, based
on intermediate calculations involving fitted values from df_ref.
- L is some column in my main dataframe
- option is a second column in the main dataframe
- df_ref is a separate data frame used in the pre-calculations here
'''
df_ref_option = df_ref[df_ref['option']==option] # take slice of df_ref based on option
x = df_ref_option['x'].values # get data columns to be used for polyfit
y = df_ref_option['y'].values
C = np.polyfit(np.log(x), np.log(y), 1); # use polyfit to get log fit of the reference data
a = np.exp(C[1]);
b = C[0];
outputval = a*(L**b)
return outputval
Usage wanted from function:
df['outputval']] = calc_fitted_values(df['L'], df['option'], df_ref)
In this example, L and option will be array values obtained from my main data frame columns (df), but df_ref is unrelated in in terms of shape and size.
How can I best write a function for this type of situation?
Thanks.
EDIT: My current "solution" is to use lambda...
f = lambda L, option : calc_fitted_values(L, option, df_ref)
df['outputval'] = np.vectorize(f)(df['L'].values, df['option'].values)
But this appears to be very slow. Might be due to the calculation each time through with df_ref, so would it be better to have a function that returns a lambda-defined function? Not sure of the best approach to this.
python pandas numpy dataframe vectorization
python pandas numpy dataframe vectorization
asked Nov 19 '18 at 21:38
teepeeteepee
7021819
asked Nov 19 '18 at 21:38
teepeeteepee
7021819
edited Nov 20 '18 at 16:48
teepee
asked Nov 19 '18 at 21:38
teepeeteepee
7021819
asked Nov 19 '18 at 21:38
teepeeteepee
7021819
asked Nov 19 '18 at 21:38
teepeeteepee
7021819
7021819
one possible solution: torch.nn.CrossEntropyLoss discuss.pytorch.org/t/…
– teng
Nov 19 '18 at 21:57
If you usenp.vectorize(for convenience, not speed), try theexcludeparameter. And don't neglect theotypesparameter.
– hpaulj
Nov 19 '18 at 22:32
the parameteroptionis not used ...
– B. M.
Nov 20 '18 at 11:59
@B.M. sorry, fixed it
– teepee
Nov 20 '18 at 16:48
add a comment |
one possible solution: torch.nn.CrossEntropyLoss discuss.pytorch.org/t/…
– teng
Nov 19 '18 at 21:57
If you usenp.vectorize(for convenience, not speed), try theexcludeparameter. And don't neglect theotypesparameter.
– hpaulj
Nov 19 '18 at 22:32
the parameteroptionis not used ...
– B. M.
Nov 20 '18 at 11:59
@B.M. sorry, fixed it
– teepee
Nov 20 '18 at 16:48
one possible solution: torch.nn.CrossEntropyLoss discuss.pytorch.org/t/…
– teng
Nov 19 '18 at 21:57
one possible solution: torch.nn.CrossEntropyLoss discuss.pytorch.org/t/…
– teng
Nov 19 '18 at 21:57
If you use
np.vectorize (for convenience, not speed), try the exclude parameter. And don't neglect the otypes parameter.– hpaulj
Nov 19 '18 at 22:32
If you use
np.vectorize (for convenience, not speed), try the exclude parameter. And don't neglect the otypes parameter.– hpaulj
Nov 19 '18 at 22:32
the parameter
option is not used ...– B. M.
Nov 20 '18 at 11:59
the parameter
option is not used ...– B. M.
Nov 20 '18 at 11:59
@B.M. sorry, fixed it
– teepee
Nov 20 '18 at 16:48
@B.M. sorry, fixed it
– teepee
Nov 20 '18 at 16:48
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53383005%2fvectorize-only-some-arguments-in-function-when-calculating-on-pandas-dataframe%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53383005%2fvectorize-only-some-arguments-in-function-when-calculating-on-pandas-dataframe%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
one possible solution: torch.nn.CrossEntropyLoss discuss.pytorch.org/t/…
– teng
Nov 19 '18 at 21:57
If you use
np.vectorize(for convenience, not speed), try theexcludeparameter. And don't neglect theotypesparameter.– hpaulj
Nov 19 '18 at 22:32
the parameter
optionis not used ...– B. M.
Nov 20 '18 at 11:59
@B.M. sorry, fixed it
– teepee
Nov 20 '18 at 16:48