Mixing
Automatically tuning the batch size by catching out of memory errors
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I would like my application to automatically find the maximum batch size possible so I set up a binary search that catches OOM exceptions like this:
min_batch_size = 3072
max_batch_size = 16384
min_range = 512 # Exit when the search range is smaller than this.
sample_iterations = 5
while max_batch_size - min_batch_size > min_range:
batch_size = (max_batch_size + min_batch_size) // 2
with tf.Graph().as_default() as graph:
train_op = build_graph()
try:
with tf.Session(graph=graph) as sess:
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())
for _ in range(sample_iterations):
sess.run(train_op)
except tf.errors.ResourceExhaustedError:
max_batch_size = batch_size
else:
min_batch_size = batch_size
return min_batch_size
Using the official TensorFlow 1.12 Docker image, it starts to converge to the actual batch size discarding sizes that are too large to fit on the current hardware. However, it fails on this internal error at the 6th iteration:
File "opennmt/runner.py", line 252, in _auto_tune_batch_size
with tf.Session(graph=graph, config=session_config) as sess:
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1551, in __init__
super(Session, self).__init__(target, graph, config=config)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 676, in __init__
self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: an illegal memory access was encountered
What is the cause of this error, given the process I described above? Are OOM errors generally not safe to recover from? How can I work around it to achieve my initial goal within the application and not by running scripts?
Thanks!
python tensorflow
asked Dec 17 '18 at 17:58
guillaumeklnguillaumekln
299315
add a comment |
I would like my application to automatically find the maximum batch size possible so I set up a binary search that catches OOM exceptions like this:
min_batch_size = 3072
max_batch_size = 16384
min_range = 512 # Exit when the search range is smaller than this.
sample_iterations = 5
while max_batch_size - min_batch_size > min_range:
batch_size = (max_batch_size + min_batch_size) // 2
with tf.Graph().as_default() as graph:
train_op = build_graph()
try:
with tf.Session(graph=graph) as sess:
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())
for _ in range(sample_iterations):
sess.run(train_op)
except tf.errors.ResourceExhaustedError:
max_batch_size = batch_size
else:
min_batch_size = batch_size
return min_batch_size
Using the official TensorFlow 1.12 Docker image, it starts to converge to the actual batch size discarding sizes that are too large to fit on the current hardware. However, it fails on this internal error at the 6th iteration:
File "opennmt/runner.py", line 252, in _auto_tune_batch_size
with tf.Session(graph=graph, config=session_config) as sess:
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1551, in __init__
super(Session, self).__init__(target, graph, config=config)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 676, in __init__
self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: an illegal memory access was encountered
What is the cause of this error, given the process I described above? Are OOM errors generally not safe to recover from? How can I work around it to achieve my initial goal within the application and not by running scripts?
Thanks!
python tensorflow
asked Dec 17 '18 at 17:58
guillaumeklnguillaumekln
299315
add a comment |
I would like my application to automatically find the maximum batch size possible so I set up a binary search that catches OOM exceptions like this:
min_batch_size = 3072
max_batch_size = 16384
min_range = 512 # Exit when the search range is smaller than this.
sample_iterations = 5
while max_batch_size - min_batch_size > min_range:
batch_size = (max_batch_size + min_batch_size) // 2
with tf.Graph().as_default() as graph:
train_op = build_graph()
try:
with tf.Session(graph=graph) as sess:
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())
for _ in range(sample_iterations):
sess.run(train_op)
except tf.errors.ResourceExhaustedError:
max_batch_size = batch_size
else:
min_batch_size = batch_size
return min_batch_size
Using the official TensorFlow 1.12 Docker image, it starts to converge to the actual batch size discarding sizes that are too large to fit on the current hardware. However, it fails on this internal error at the 6th iteration:
File "opennmt/runner.py", line 252, in _auto_tune_batch_size
with tf.Session(graph=graph, config=session_config) as sess:
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1551, in __init__
super(Session, self).__init__(target, graph, config=config)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 676, in __init__
self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: an illegal memory access was encountered
What is the cause of this error, given the process I described above? Are OOM errors generally not safe to recover from? How can I work around it to achieve my initial goal within the application and not by running scripts?
Thanks!
python tensorflow
asked Dec 17 '18 at 17:58
guillaumeklnguillaumekln
299315
I would like my application to automatically find the maximum batch size possible so I set up a binary search that catches OOM exceptions like this:
min_batch_size = 3072
max_batch_size = 16384
min_range = 512 # Exit when the search range is smaller than this.
sample_iterations = 5
while max_batch_size - min_batch_size > min_range:
batch_size = (max_batch_size + min_batch_size) // 2
with tf.Graph().as_default() as graph:
train_op = build_graph()
try:
with tf.Session(graph=graph) as sess:
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())
for _ in range(sample_iterations):
sess.run(train_op)
except tf.errors.ResourceExhaustedError:
max_batch_size = batch_size
else:
min_batch_size = batch_size
return min_batch_size
Using the official TensorFlow 1.12 Docker image, it starts to converge to the actual batch size discarding sizes that are too large to fit on the current hardware. However, it fails on this internal error at the 6th iteration:
File "opennmt/runner.py", line 252, in _auto_tune_batch_size
with tf.Session(graph=graph, config=session_config) as sess:
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1551, in __init__
super(Session, self).__init__(target, graph, config=config)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 676, in __init__
self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: an illegal memory access was encountered
What is the cause of this error, given the process I described above? Are OOM errors generally not safe to recover from? How can I work around it to achieve my initial goal within the application and not by running scripts?
Thanks!
python tensorflow
python tensorflow
asked Dec 17 '18 at 17:58
guillaumeklnguillaumekln
299315
asked Dec 17 '18 at 17:58
guillaumeklnguillaumekln
299315
edited Jan 3 at 15:03
guillaumekln
asked Dec 17 '18 at 17:58
guillaumeklnguillaumekln
299315
asked Dec 17 '18 at 17:58
guillaumeklnguillaumekln
299315
asked Dec 17 '18 at 17:58
guillaumeklnguillaumekln
299315
299315
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53820713%2fautomatically-tuning-the-batch-size-by-catching-out-of-memory-errors%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53820713%2fautomatically-tuning-the-batch-size-by-catching-out-of-memory-errors%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
