Mixing
Use sklearn to create custom metric in keras
Background
I create a simple model using keras (with tensorflow backend)
model = Sequential()
...
model.compile(loss='mse', optimizer=Adam(), metrics=[average_precision])
and then I want to base the early stopping on my custom metric:
model.fit(x=x_train, y=y_train,
...
callbacks=[EarlyStopping(monitor='average_precision', mode='max', patience=3)])
so far so good. But the problem is that the average_precision is implemented using average_precision_score from sklearn.metrics:
def average_precision(y_true, y_pred):
return average_precision_score(y_true, y_pred, average="micro")
It accepts numpy arrays only. But during training the function is fed with tensors.
Question
How can I use sklearn function in order to implement custom metric.
Remark:
I don't need to implement loss, so the function does not have to be differentiable.
What is not working
I tried juggling with some session/run/eval in order to get numpy arrays in the function but I was not successful.
Workaround that I am using but I am not happy with:
I wrote my own Early Stopping callback. I provide it with the validation data in a form of numpy arrays in its constructor. This solutions has multiple drawbacks and I look for sth more elegant.
What I do not want to do:
Rewriting by hand well tested functions from sklearn using keras backend.
python numpy tensorflow scikit-learn keras
asked Nov 21 '18 at 10:14
hanshans
16511
add a comment |
Background
I create a simple model using keras (with tensorflow backend)
model = Sequential()
...
model.compile(loss='mse', optimizer=Adam(), metrics=[average_precision])
and then I want to base the early stopping on my custom metric:
model.fit(x=x_train, y=y_train,
...
callbacks=[EarlyStopping(monitor='average_precision', mode='max', patience=3)])
so far so good. But the problem is that the average_precision is implemented using average_precision_score from sklearn.metrics:
def average_precision(y_true, y_pred):
return average_precision_score(y_true, y_pred, average="micro")
It accepts numpy arrays only. But during training the function is fed with tensors.
Question
How can I use sklearn function in order to implement custom metric.
Remark:
I don't need to implement loss, so the function does not have to be differentiable.
What is not working
I tried juggling with some session/run/eval in order to get numpy arrays in the function but I was not successful.
Workaround that I am using but I am not happy with:
I wrote my own Early Stopping callback. I provide it with the validation data in a form of numpy arrays in its constructor. This solutions has multiple drawbacks and I look for sth more elegant.
What I do not want to do:
Rewriting by hand well tested functions from sklearn using keras backend.
python numpy tensorflow scikit-learn keras
asked Nov 21 '18 at 10:14
hanshans
16511
add a comment |
Background
I create a simple model using keras (with tensorflow backend)
model = Sequential()
...
model.compile(loss='mse', optimizer=Adam(), metrics=[average_precision])
and then I want to base the early stopping on my custom metric:
model.fit(x=x_train, y=y_train,
...
callbacks=[EarlyStopping(monitor='average_precision', mode='max', patience=3)])
so far so good. But the problem is that the average_precision is implemented using average_precision_score from sklearn.metrics:
def average_precision(y_true, y_pred):
return average_precision_score(y_true, y_pred, average="micro")
It accepts numpy arrays only. But during training the function is fed with tensors.
Question
How can I use sklearn function in order to implement custom metric.
Remark:
I don't need to implement loss, so the function does not have to be differentiable.
What is not working
I tried juggling with some session/run/eval in order to get numpy arrays in the function but I was not successful.
Workaround that I am using but I am not happy with:
I wrote my own Early Stopping callback. I provide it with the validation data in a form of numpy arrays in its constructor. This solutions has multiple drawbacks and I look for sth more elegant.
What I do not want to do:
Rewriting by hand well tested functions from sklearn using keras backend.
python numpy tensorflow scikit-learn keras
asked Nov 21 '18 at 10:14
hanshans
16511
Background
I create a simple model using keras (with tensorflow backend)
model = Sequential()
...
model.compile(loss='mse', optimizer=Adam(), metrics=[average_precision])
and then I want to base the early stopping on my custom metric:
model.fit(x=x_train, y=y_train,
...
callbacks=[EarlyStopping(monitor='average_precision', mode='max', patience=3)])
so far so good. But the problem is that the average_precision is implemented using average_precision_score from sklearn.metrics:
def average_precision(y_true, y_pred):
return average_precision_score(y_true, y_pred, average="micro")
It accepts numpy arrays only. But during training the function is fed with tensors.
Question
How can I use sklearn function in order to implement custom metric.
Remark:
I don't need to implement loss, so the function does not have to be differentiable.
What is not working
I tried juggling with some session/run/eval in order to get numpy arrays in the function but I was not successful.
Workaround that I am using but I am not happy with:
I wrote my own Early Stopping callback. I provide it with the validation data in a form of numpy arrays in its constructor. This solutions has multiple drawbacks and I look for sth more elegant.
What I do not want to do:
Rewriting by hand well tested functions from sklearn using keras backend.
python numpy tensorflow scikit-learn keras
python numpy tensorflow scikit-learn keras
asked Nov 21 '18 at 10:14
hanshans
16511
asked Nov 21 '18 at 10:14
hanshans
16511
asked Nov 21 '18 at 10:14
hanshans
16511
asked Nov 21 '18 at 10:14
hanshans
16511
asked Nov 21 '18 at 10:14
hanshans
16511
16511
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53409752%2fuse-sklearn-to-create-custom-metric-in-keras%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53409752%2fuse-sklearn-to-create-custom-metric-in-keras%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
