Mixing
Interlacing randomly a tf.Dataset with another tf.Dataset
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
1
I have two datasets:
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100)))
backgroud_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4])
I want a batch interleaving main_ds and backgroud_ds data randomly. For instance, a batch of size 10 should look like:
[3, 1017, 1039, 3, 2, 1024, 4, 1, 1053, 4]
I tried the following:
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
background_ds = background_ds.shuffle(10).repeat(-1)
background_it = background_ds.make_initializable_iterator()
background_next = background_it.get_next()
main_ds = main_ds.shuffle(10)
.repeat(-1)
.map(lambda x: interlace_background(x, background_next))
.batch(10)
main_it = main_ds.make_initializable_iterator()
main_next = main_it.get_next()
but I get a fixed background across all batches:
batch 0: [ 3 1006 3 1001 3 1005 1015 1000 3 3]
batch 1: [1007 3 1012 1018 1013 3 1008 1019 3 3]
batch 2: [1016 3 1025 3 3 3 1021 3 3 1035]
batch 3: [1038 3 3 1023 1020 3 3 1046 1034 1047]
batch 4: [ 3 3 1039 3 3 3 3 3 1053 3]
Why is the background fixed (cf. above where background is always 3) and how could I solve this?
Fully reproducible code below:
import tensorflow as tf
import numpy as np
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100)))
background_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4])
background_ds = background_ds.shuffle(10).repeat(-1)
background_it = background_ds.make_initializable_iterator()
background_next = background_it.get_next()
main_ds = main_ds.shuffle(10)
.repeat(-1)
.map(lambda x: interlace_background(x, background_next))
.batch(10)
main_it = main_ds.make_initializable_iterator()
main_next = main_it.get_next()
with tf.Session() as sess:
sess.run(background_it.initializer)
sess.run(main_it.initializer)
for i in range(5):
print('batch %i' % i, sess.run(main_next))
python tensorflow
asked Jan 3 at 15:16
BiBiBiBi
1,89911438
add a comment |
1
I have two datasets:
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100)))
backgroud_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4])
I want a batch interleaving main_ds and backgroud_ds data randomly. For instance, a batch of size 10 should look like:
[3, 1017, 1039, 3, 2, 1024, 4, 1, 1053, 4]
I tried the following:
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
background_ds = background_ds.shuffle(10).repeat(-1)
background_it = background_ds.make_initializable_iterator()
background_next = background_it.get_next()
main_ds = main_ds.shuffle(10)
.repeat(-1)
.map(lambda x: interlace_background(x, background_next))
.batch(10)
main_it = main_ds.make_initializable_iterator()
main_next = main_it.get_next()
but I get a fixed background across all batches:
batch 0: [ 3 1006 3 1001 3 1005 1015 1000 3 3]
batch 1: [1007 3 1012 1018 1013 3 1008 1019 3 3]
batch 2: [1016 3 1025 3 3 3 1021 3 3 1035]
batch 3: [1038 3 3 1023 1020 3 3 1046 1034 1047]
batch 4: [ 3 3 1039 3 3 3 3 3 1053 3]
Why is the background fixed (cf. above where background is always 3) and how could I solve this?
Fully reproducible code below:
import tensorflow as tf
import numpy as np
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100)))
background_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4])
background_ds = background_ds.shuffle(10).repeat(-1)
background_it = background_ds.make_initializable_iterator()
background_next = background_it.get_next()
main_ds = main_ds.shuffle(10)
.repeat(-1)
.map(lambda x: interlace_background(x, background_next))
.batch(10)
main_it = main_ds.make_initializable_iterator()
main_next = main_it.get_next()
with tf.Session() as sess:
sess.run(background_it.initializer)
sess.run(main_it.initializer)
for i in range(5):
print('batch %i' % i, sess.run(main_next))
python tensorflow
asked Jan 3 at 15:16
BiBiBiBi
1,89911438
add a comment |
1
1
1
I have two datasets:
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100)))
backgroud_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4])
I want a batch interleaving main_ds and backgroud_ds data randomly. For instance, a batch of size 10 should look like:
[3, 1017, 1039, 3, 2, 1024, 4, 1, 1053, 4]
I tried the following:
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
background_ds = background_ds.shuffle(10).repeat(-1)
background_it = background_ds.make_initializable_iterator()
background_next = background_it.get_next()
main_ds = main_ds.shuffle(10)
.repeat(-1)
.map(lambda x: interlace_background(x, background_next))
.batch(10)
main_it = main_ds.make_initializable_iterator()
main_next = main_it.get_next()
but I get a fixed background across all batches:
batch 0: [ 3 1006 3 1001 3 1005 1015 1000 3 3]
batch 1: [1007 3 1012 1018 1013 3 1008 1019 3 3]
batch 2: [1016 3 1025 3 3 3 1021 3 3 1035]
batch 3: [1038 3 3 1023 1020 3 3 1046 1034 1047]
batch 4: [ 3 3 1039 3 3 3 3 3 1053 3]
Why is the background fixed (cf. above where background is always 3) and how could I solve this?
Fully reproducible code below:
import tensorflow as tf
import numpy as np
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100)))
background_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4])
background_ds = background_ds.shuffle(10).repeat(-1)
background_it = background_ds.make_initializable_iterator()
background_next = background_it.get_next()
main_ds = main_ds.shuffle(10)
.repeat(-1)
.map(lambda x: interlace_background(x, background_next))
.batch(10)
main_it = main_ds.make_initializable_iterator()
main_next = main_it.get_next()
with tf.Session() as sess:
sess.run(background_it.initializer)
sess.run(main_it.initializer)
for i in range(5):
print('batch %i' % i, sess.run(main_next))
python tensorflow
asked Jan 3 at 15:16
BiBiBiBi
1,89911438
I have two datasets:
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100)))
backgroud_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4])
I want a batch interleaving main_ds and backgroud_ds data randomly. For instance, a batch of size 10 should look like:
[3, 1017, 1039, 3, 2, 1024, 4, 1, 1053, 4]
I tried the following:
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
background_ds = background_ds.shuffle(10).repeat(-1)
background_it = background_ds.make_initializable_iterator()
background_next = background_it.get_next()
main_ds = main_ds.shuffle(10)
.repeat(-1)
.map(lambda x: interlace_background(x, background_next))
.batch(10)
main_it = main_ds.make_initializable_iterator()
main_next = main_it.get_next()
but I get a fixed background across all batches:
batch 0: [ 3 1006 3 1001 3 1005 1015 1000 3 3]
batch 1: [1007 3 1012 1018 1013 3 1008 1019 3 3]
batch 2: [1016 3 1025 3 3 3 1021 3 3 1035]
batch 3: [1038 3 3 1023 1020 3 3 1046 1034 1047]
batch 4: [ 3 3 1039 3 3 3 3 3 1053 3]
Why is the background fixed (cf. above where background is always 3) and how could I solve this?
Fully reproducible code below:
import tensorflow as tf
import numpy as np
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100)))
background_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4])
background_ds = background_ds.shuffle(10).repeat(-1)
background_it = background_ds.make_initializable_iterator()
background_next = background_it.get_next()
main_ds = main_ds.shuffle(10)
.repeat(-1)
.map(lambda x: interlace_background(x, background_next))
.batch(10)
main_it = main_ds.make_initializable_iterator()
main_next = main_it.get_next()
with tf.Session() as sess:
sess.run(background_it.initializer)
sess.run(main_it.initializer)
for i in range(5):
print('batch %i' % i, sess.run(main_next))
python tensorflow
python tensorflow
asked Jan 3 at 15:16
BiBiBiBi
1,89911438
asked Jan 3 at 15:16
BiBiBiBi
1,89911438
asked Jan 3 at 15:16
BiBiBiBi
1,89911438
asked Jan 3 at 15:16
BiBiBiBi
1,89911438
asked Jan 3 at 15:16
BiBiBiBi
1,89911438
1,89911438
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
1
You can do the same thing with Dataset.zip() and Dataset.map().
Here is the code:
import tensorflow as tf
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100))).shuffle(100)
background_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4]).shuffle(4)
new_ds = tf.data.Dataset
.zip((main_ds, background_ds))
.repeat(-1)
.map(lambda x, y: interlace_background(x, y))
.batch(10)
iterator = new_ds.make_initializable_iterator()
next_item = iterator.get_next()
with tf.Session() as sess:
sess.run(iterator.initializer)
for i in range(5):
print('batch %i' % i, sess.run(next_item))
Output:
batch 0 [1065 2 4 1 2 4 1 1036 1072 1020]
batch 1 [ 4 3 2 1057 1 4 2 1077 3 1]
batch 2 [ 3 1044 1042 1049 1029 1 3 1069 1018 3]
batch 3 [ 2 4 1089 1094 2 1022 1041 1006 1 3]
batch 4 [1079 2 1 3 1023 1042 4 1018 1054 4]
answered Jan 3 at 15:38
AmirAmir
8,12774277
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54025069%2finterlacing-randomly-a-tf-dataset-with-another-tf-dataset%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
You can do the same thing with Dataset.zip() and Dataset.map().
Here is the code:
import tensorflow as tf
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100))).shuffle(100)
background_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4]).shuffle(4)
new_ds = tf.data.Dataset
.zip((main_ds, background_ds))
.repeat(-1)
.map(lambda x, y: interlace_background(x, y))
.batch(10)
iterator = new_ds.make_initializable_iterator()
next_item = iterator.get_next()
with tf.Session() as sess:
sess.run(iterator.initializer)
for i in range(5):
print('batch %i' % i, sess.run(next_item))
Output:
batch 0 [1065 2 4 1 2 4 1 1036 1072 1020]
batch 1 [ 4 3 2 1057 1 4 2 1077 3 1]
batch 2 [ 3 1044 1042 1049 1029 1 3 1069 1018 3]
batch 3 [ 2 4 1089 1094 2 1022 1041 1006 1 3]
batch 4 [1079 2 1 3 1023 1042 4 1018 1054 4]
answered Jan 3 at 15:38
AmirAmir
8,12774277
add a comment |
1
You can do the same thing with Dataset.zip() and Dataset.map().
Here is the code:
import tensorflow as tf
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100))).shuffle(100)
background_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4]).shuffle(4)
new_ds = tf.data.Dataset
.zip((main_ds, background_ds))
.repeat(-1)
.map(lambda x, y: interlace_background(x, y))
.batch(10)
iterator = new_ds.make_initializable_iterator()
next_item = iterator.get_next()
with tf.Session() as sess:
sess.run(iterator.initializer)
for i in range(5):
print('batch %i' % i, sess.run(next_item))
Output:
batch 0 [1065 2 4 1 2 4 1 1036 1072 1020]
batch 1 [ 4 3 2 1057 1 4 2 1077 3 1]
batch 2 [ 3 1044 1042 1049 1029 1 3 1069 1018 3]
batch 3 [ 2 4 1089 1094 2 1022 1041 1006 1 3]
batch 4 [1079 2 1 3 1023 1042 4 1018 1054 4]
answered Jan 3 at 15:38
AmirAmir
8,12774277
add a comment |
1
1
1
You can do the same thing with Dataset.zip() and Dataset.map().
Here is the code:
import tensorflow as tf
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100))).shuffle(100)
background_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4]).shuffle(4)
new_ds = tf.data.Dataset
.zip((main_ds, background_ds))
.repeat(-1)
.map(lambda x, y: interlace_background(x, y))
.batch(10)
iterator = new_ds.make_initializable_iterator()
next_item = iterator.get_next()
with tf.Session() as sess:
sess.run(iterator.initializer)
for i in range(5):
print('batch %i' % i, sess.run(next_item))
Output:
batch 0 [1065 2 4 1 2 4 1 1036 1072 1020]
batch 1 [ 4 3 2 1057 1 4 2 1077 3 1]
batch 2 [ 3 1044 1042 1049 1029 1 3 1069 1018 3]
batch 3 [ 2 4 1089 1094 2 1022 1041 1006 1 3]
batch 4 [1079 2 1 3 1023 1042 4 1018 1054 4]
answered Jan 3 at 15:38
AmirAmir
8,12774277
You can do the same thing with Dataset.zip() and Dataset.map().
Here is the code:
import tensorflow as tf
def interlace_background(image, background):
return tf.cond(tf.random_uniform() < .5, lambda: image, lambda: background)
main_ds = tf.data.Dataset.from_tensor_slices(list(range(1000, 1100))).shuffle(100)
background_ds = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4]).shuffle(4)
new_ds = tf.data.Dataset
.zip((main_ds, background_ds))
.repeat(-1)
.map(lambda x, y: interlace_background(x, y))
.batch(10)
iterator = new_ds.make_initializable_iterator()
next_item = iterator.get_next()
with tf.Session() as sess:
sess.run(iterator.initializer)
for i in range(5):
print('batch %i' % i, sess.run(next_item))
Output:
batch 0 [1065 2 4 1 2 4 1 1036 1072 1020]
batch 1 [ 4 3 2 1057 1 4 2 1077 3 1]
batch 2 [ 3 1044 1042 1049 1029 1 3 1069 1018 3]
batch 3 [ 2 4 1089 1094 2 1022 1041 1006 1 3]
batch 4 [1079 2 1 3 1023 1042 4 1018 1054 4]
answered Jan 3 at 15:38
AmirAmir
8,12774277
edited Jan 3 at 16:04
answered Jan 3 at 15:38
AmirAmir
8,12774277
answered Jan 3 at 15:38
AmirAmir
8,12774277
answered Jan 3 at 15:38
AmirAmir
8,12774277
8,12774277
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54025069%2finterlacing-randomly-a-tf-dataset-with-another-tf-dataset%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
