Mixing
tensorflow - implementing experience replay memory with the estimator api
I try to implement an experience replay memory with the tf.estimator.Estimator API. However, i am not sure what is the best way to achieve a result which works atleast for all modes (TRAIN, EVALUATE, PREDICT). I tried the following:
- Implement the memory with a
tf.Variable, which causes issues with the batching and the input pipeline (i cannot input a custom experience in testing or prediction phase)
and currently try to:
- Implement the memory outside the
tf.Graph. Set the values after each run with atf.train.SessionRunHook. Load the experiences withtf.data.Dataset.from_generator()during training and testing. Manage the state on your own.
I am failing on several points and starting to believe that the tf.estimator.Estimator API does not provide me with the necessary interfaces to easily write this down.
Some code (first approach, which fails with the batch_size, since it is fixed for the slicing of the exp, i cannot use the model for prediction or evaluation):
def model_fn(self, features, labels, mode, params):
batch_size = features["matrix"].get_shape()[0].value
# get prev_exp
if mode == tf.estimator.ModeKeys.TRAIN:
erm = tf.get_variable("erm", shape=[30000, 10], initializer=tf.constant_initializer(self.erm.initial_train_erm()), trainable=False)
prev_exp = tf.slice(erm, [features["index"][0], 0], [batch_size, 10])
# model
pred = model(features["matrix"], prev_exp, params)
However: it would be better to have the erm inside the feature dict. But then i have to manage the erm outside the graph and also write back my newest experience with a SessionRunHook. Is there any better way or am i missing something?
python tensorflow deep-learning reinforcement-learning tensorflow-estimator
edited Nov 21 '18 at 5:26
Milo Lu
1,61511427
asked Nov 20 '18 at 23:47
ChocolateChocolate
1341211
add a comment |
I try to implement an experience replay memory with the tf.estimator.Estimator API. However, i am not sure what is the best way to achieve a result which works atleast for all modes (TRAIN, EVALUATE, PREDICT). I tried the following:
- Implement the memory with a
tf.Variable, which causes issues with the batching and the input pipeline (i cannot input a custom experience in testing or prediction phase)
and currently try to:
- Implement the memory outside the
tf.Graph. Set the values after each run with atf.train.SessionRunHook. Load the experiences withtf.data.Dataset.from_generator()during training and testing. Manage the state on your own.
I am failing on several points and starting to believe that the tf.estimator.Estimator API does not provide me with the necessary interfaces to easily write this down.
Some code (first approach, which fails with the batch_size, since it is fixed for the slicing of the exp, i cannot use the model for prediction or evaluation):
def model_fn(self, features, labels, mode, params):
batch_size = features["matrix"].get_shape()[0].value
# get prev_exp
if mode == tf.estimator.ModeKeys.TRAIN:
erm = tf.get_variable("erm", shape=[30000, 10], initializer=tf.constant_initializer(self.erm.initial_train_erm()), trainable=False)
prev_exp = tf.slice(erm, [features["index"][0], 0], [batch_size, 10])
# model
pred = model(features["matrix"], prev_exp, params)
However: it would be better to have the erm inside the feature dict. But then i have to manage the erm outside the graph and also write back my newest experience with a SessionRunHook. Is there any better way or am i missing something?
python tensorflow deep-learning reinforcement-learning tensorflow-estimator
edited Nov 21 '18 at 5:26
Milo Lu
1,61511427
asked Nov 20 '18 at 23:47
ChocolateChocolate
1341211
add a comment |
I try to implement an experience replay memory with the tf.estimator.Estimator API. However, i am not sure what is the best way to achieve a result which works atleast for all modes (TRAIN, EVALUATE, PREDICT). I tried the following:
- Implement the memory with a
tf.Variable, which causes issues with the batching and the input pipeline (i cannot input a custom experience in testing or prediction phase)
and currently try to:
- Implement the memory outside the
tf.Graph. Set the values after each run with atf.train.SessionRunHook. Load the experiences withtf.data.Dataset.from_generator()during training and testing. Manage the state on your own.
I am failing on several points and starting to believe that the tf.estimator.Estimator API does not provide me with the necessary interfaces to easily write this down.
Some code (first approach, which fails with the batch_size, since it is fixed for the slicing of the exp, i cannot use the model for prediction or evaluation):
def model_fn(self, features, labels, mode, params):
batch_size = features["matrix"].get_shape()[0].value
# get prev_exp
if mode == tf.estimator.ModeKeys.TRAIN:
erm = tf.get_variable("erm", shape=[30000, 10], initializer=tf.constant_initializer(self.erm.initial_train_erm()), trainable=False)
prev_exp = tf.slice(erm, [features["index"][0], 0], [batch_size, 10])
# model
pred = model(features["matrix"], prev_exp, params)
However: it would be better to have the erm inside the feature dict. But then i have to manage the erm outside the graph and also write back my newest experience with a SessionRunHook. Is there any better way or am i missing something?
python tensorflow deep-learning reinforcement-learning tensorflow-estimator
edited Nov 21 '18 at 5:26
Milo Lu
1,61511427
asked Nov 20 '18 at 23:47
ChocolateChocolate
1341211
I try to implement an experience replay memory with the tf.estimator.Estimator API. However, i am not sure what is the best way to achieve a result which works atleast for all modes (TRAIN, EVALUATE, PREDICT). I tried the following:
- Implement the memory with a
tf.Variable, which causes issues with the batching and the input pipeline (i cannot input a custom experience in testing or prediction phase)
and currently try to:
- Implement the memory outside the
tf.Graph. Set the values after each run with atf.train.SessionRunHook. Load the experiences withtf.data.Dataset.from_generator()during training and testing. Manage the state on your own.
I am failing on several points and starting to believe that the tf.estimator.Estimator API does not provide me with the necessary interfaces to easily write this down.
Some code (first approach, which fails with the batch_size, since it is fixed for the slicing of the exp, i cannot use the model for prediction or evaluation):
def model_fn(self, features, labels, mode, params):
batch_size = features["matrix"].get_shape()[0].value
# get prev_exp
if mode == tf.estimator.ModeKeys.TRAIN:
erm = tf.get_variable("erm", shape=[30000, 10], initializer=tf.constant_initializer(self.erm.initial_train_erm()), trainable=False)
prev_exp = tf.slice(erm, [features["index"][0], 0], [batch_size, 10])
# model
pred = model(features["matrix"], prev_exp, params)
However: it would be better to have the erm inside the feature dict. But then i have to manage the erm outside the graph and also write back my newest experience with a SessionRunHook. Is there any better way or am i missing something?
python tensorflow deep-learning reinforcement-learning tensorflow-estimator
python tensorflow deep-learning reinforcement-learning tensorflow-estimator
edited Nov 21 '18 at 5:26
Milo Lu
1,61511427
asked Nov 20 '18 at 23:47
ChocolateChocolate
1341211
edited Nov 21 '18 at 5:26
Milo Lu
1,61511427
asked Nov 20 '18 at 23:47
ChocolateChocolate
1341211
edited Nov 21 '18 at 5:26
Milo Lu
1,61511427
edited Nov 21 '18 at 5:26
Milo Lu
1,61511427
edited Nov 21 '18 at 5:26
Milo Lu
1,61511427
1,61511427
asked Nov 20 '18 at 23:47
ChocolateChocolate
1341211
asked Nov 20 '18 at 23:47
ChocolateChocolate
1341211
asked Nov 20 '18 at 23:47
ChocolateChocolate
1341211
1341211
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
I solved my issue by implementing the ERM outside the graph, feeding it back into the input pipeline with tf.data.Dataset.from_generator() and writing back by using SessionRunHooks. Yeah, pretty tedious but it is working.
answered Nov 22 '18 at 13:57
ChocolateChocolate
1341211
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53403311%2ftensorflow-implementing-experience-replay-memory-with-the-estimator-api%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I solved my issue by implementing the ERM outside the graph, feeding it back into the input pipeline with tf.data.Dataset.from_generator() and writing back by using SessionRunHooks. Yeah, pretty tedious but it is working.
answered Nov 22 '18 at 13:57
ChocolateChocolate
1341211
add a comment |
I solved my issue by implementing the ERM outside the graph, feeding it back into the input pipeline with tf.data.Dataset.from_generator() and writing back by using SessionRunHooks. Yeah, pretty tedious but it is working.
answered Nov 22 '18 at 13:57
ChocolateChocolate
1341211
add a comment |
I solved my issue by implementing the ERM outside the graph, feeding it back into the input pipeline with tf.data.Dataset.from_generator() and writing back by using SessionRunHooks. Yeah, pretty tedious but it is working.
answered Nov 22 '18 at 13:57
ChocolateChocolate
1341211
I solved my issue by implementing the ERM outside the graph, feeding it back into the input pipeline with tf.data.Dataset.from_generator() and writing back by using SessionRunHooks. Yeah, pretty tedious but it is working.
answered Nov 22 '18 at 13:57
ChocolateChocolate
1341211
answered Nov 22 '18 at 13:57
ChocolateChocolate
1341211
answered Nov 22 '18 at 13:57
ChocolateChocolate
1341211
answered Nov 22 '18 at 13:57
ChocolateChocolate
1341211
1341211
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53403311%2ftensorflow-implementing-experience-replay-memory-with-the-estimator-api%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
