Mixing
Return distribution over set of action space from Neural Network
I am trying to build a neural network to output a probabilistic distribution over set of whole action space.
My action space is a vector of 3 individual actions : [a,b,c]
a can have 3 possible actions within itself a1,a2,a3 and similarly b has b1,b2,b3, c has c1,c2,c3. So In total i can have 27 different combinations of these actions 3^3 = 27. Ultimately the neural network should output 27 combinations of these actions (which is a matrix of 27 x 3) : [[a1,b1,c1],[a2,b2,c2],[a3,b3,c3],[a1,b1,c2],[a1,b1,c3],.....] and so on for all 27 combinations. Just to mention the input to my network is a state which is a vector of 5 elements.
I want a probability associated to each of these 27 combinations.
I know I can associate probability by using a softmax with 27 outputs but I don't understand how the network can output a matrix in this case where every row has a probability associated to it.
python tensorflow neural-network probability reinforcement-learning
asked Nov 19 '18 at 15:52
Siddhant Tandon
305
add a comment |
I am trying to build a neural network to output a probabilistic distribution over set of whole action space.
My action space is a vector of 3 individual actions : [a,b,c]
a can have 3 possible actions within itself a1,a2,a3 and similarly b has b1,b2,b3, c has c1,c2,c3. So In total i can have 27 different combinations of these actions 3^3 = 27. Ultimately the neural network should output 27 combinations of these actions (which is a matrix of 27 x 3) : [[a1,b1,c1],[a2,b2,c2],[a3,b3,c3],[a1,b1,c2],[a1,b1,c3],.....] and so on for all 27 combinations. Just to mention the input to my network is a state which is a vector of 5 elements.
I want a probability associated to each of these 27 combinations.
I know I can associate probability by using a softmax with 27 outputs but I don't understand how the network can output a matrix in this case where every row has a probability associated to it.
python tensorflow neural-network probability reinforcement-learning
asked Nov 19 '18 at 15:52
Siddhant Tandon
305
add a comment |
I am trying to build a neural network to output a probabilistic distribution over set of whole action space.
My action space is a vector of 3 individual actions : [a,b,c]
a can have 3 possible actions within itself a1,a2,a3 and similarly b has b1,b2,b3, c has c1,c2,c3. So In total i can have 27 different combinations of these actions 3^3 = 27. Ultimately the neural network should output 27 combinations of these actions (which is a matrix of 27 x 3) : [[a1,b1,c1],[a2,b2,c2],[a3,b3,c3],[a1,b1,c2],[a1,b1,c3],.....] and so on for all 27 combinations. Just to mention the input to my network is a state which is a vector of 5 elements.
I want a probability associated to each of these 27 combinations.
I know I can associate probability by using a softmax with 27 outputs but I don't understand how the network can output a matrix in this case where every row has a probability associated to it.
python tensorflow neural-network probability reinforcement-learning
asked Nov 19 '18 at 15:52
Siddhant Tandon
305
I am trying to build a neural network to output a probabilistic distribution over set of whole action space.
My action space is a vector of 3 individual actions : [a,b,c]
a can have 3 possible actions within itself a1,a2,a3 and similarly b has b1,b2,b3, c has c1,c2,c3. So In total i can have 27 different combinations of these actions 3^3 = 27. Ultimately the neural network should output 27 combinations of these actions (which is a matrix of 27 x 3) : [[a1,b1,c1],[a2,b2,c2],[a3,b3,c3],[a1,b1,c2],[a1,b1,c3],.....] and so on for all 27 combinations. Just to mention the input to my network is a state which is a vector of 5 elements.
I want a probability associated to each of these 27 combinations.
I know I can associate probability by using a softmax with 27 outputs but I don't understand how the network can output a matrix in this case where every row has a probability associated to it.
python tensorflow neural-network probability reinforcement-learning
python tensorflow neural-network probability reinforcement-learning
asked Nov 19 '18 at 15:52
Siddhant Tandon
305
asked Nov 19 '18 at 15:52
Siddhant Tandon
305
asked Nov 19 '18 at 15:52
Siddhant Tandon
305
asked Nov 19 '18 at 15:52
Siddhant Tandon
305
asked Nov 19 '18 at 15:52
Siddhant Tandon
305
305
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Is there any reason you want it to return a matrix of these actions? Why not just map each of the 27 combinations to integers 0-26? So your architecture could look like [Linear(5, n), ReLU, Linear(n, .) ... Softmax(Linear(., 27))]. Then when you need to evaluate, you can just map it back to the action sequence. This is similar to how in NLP tasks you map multidimensional word vectors to integers via stoi for training and bring them back via itos.
I should point out that if your training paradigm involves some further use of these discrete choices (say you use the argmax of this upstream of another net), then the nondifferentiability of argmax means that this architechture will not learn anything. I only mention this because you use the phrase "action space", which is typical in DRL. If this is the case, you may want to consider algorithms like REINFORCE where action sequences can be learned discretely and used upstream via policy gradient.
answered Nov 19 '18 at 16:04
rvd
43117
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more theargmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here
– Siddhant Tandon
Nov 19 '18 at 16:15
So you want the Q-values of each of the three values fora,b, andc? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.
– rvd
Nov 19 '18 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elementsa,b,cas output and repeat this 27 times ?
– Siddhant Tandon
Nov 19 '18 at 16:28
Not quite. I mean make the network emit a sequence like[a2, b1, c3]and strongly penalize making invalid sequences like 2as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.
– rvd
Nov 19 '18 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 '18 at 16:41
|
show 1 more comment
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53378284%2freturn-distribution-over-set-of-action-space-from-neural-network%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is there any reason you want it to return a matrix of these actions? Why not just map each of the 27 combinations to integers 0-26? So your architecture could look like [Linear(5, n), ReLU, Linear(n, .) ... Softmax(Linear(., 27))]. Then when you need to evaluate, you can just map it back to the action sequence. This is similar to how in NLP tasks you map multidimensional word vectors to integers via stoi for training and bring them back via itos.
I should point out that if your training paradigm involves some further use of these discrete choices (say you use the argmax of this upstream of another net), then the nondifferentiability of argmax means that this architechture will not learn anything. I only mention this because you use the phrase "action space", which is typical in DRL. If this is the case, you may want to consider algorithms like REINFORCE where action sequences can be learned discretely and used upstream via policy gradient.
answered Nov 19 '18 at 16:04
rvd
43117
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more theargmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here
– Siddhant Tandon
Nov 19 '18 at 16:15
So you want the Q-values of each of the three values fora,b, andc? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.
– rvd
Nov 19 '18 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elementsa,b,cas output and repeat this 27 times ?
– Siddhant Tandon
Nov 19 '18 at 16:28
Not quite. I mean make the network emit a sequence like[a2, b1, c3]and strongly penalize making invalid sequences like 2as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.
– rvd
Nov 19 '18 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 '18 at 16:41
|
show 1 more comment
Is there any reason you want it to return a matrix of these actions? Why not just map each of the 27 combinations to integers 0-26? So your architecture could look like [Linear(5, n), ReLU, Linear(n, .) ... Softmax(Linear(., 27))]. Then when you need to evaluate, you can just map it back to the action sequence. This is similar to how in NLP tasks you map multidimensional word vectors to integers via stoi for training and bring them back via itos.
I should point out that if your training paradigm involves some further use of these discrete choices (say you use the argmax of this upstream of another net), then the nondifferentiability of argmax means that this architechture will not learn anything. I only mention this because you use the phrase "action space", which is typical in DRL. If this is the case, you may want to consider algorithms like REINFORCE where action sequences can be learned discretely and used upstream via policy gradient.
answered Nov 19 '18 at 16:04
rvd
43117
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more theargmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here
– Siddhant Tandon
Nov 19 '18 at 16:15
So you want the Q-values of each of the three values fora,b, andc? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.
– rvd
Nov 19 '18 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elementsa,b,cas output and repeat this 27 times ?
– Siddhant Tandon
Nov 19 '18 at 16:28
Not quite. I mean make the network emit a sequence like[a2, b1, c3]and strongly penalize making invalid sequences like 2as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.
– rvd
Nov 19 '18 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 '18 at 16:41
|
show 1 more comment
Is there any reason you want it to return a matrix of these actions? Why not just map each of the 27 combinations to integers 0-26? So your architecture could look like [Linear(5, n), ReLU, Linear(n, .) ... Softmax(Linear(., 27))]. Then when you need to evaluate, you can just map it back to the action sequence. This is similar to how in NLP tasks you map multidimensional word vectors to integers via stoi for training and bring them back via itos.
I should point out that if your training paradigm involves some further use of these discrete choices (say you use the argmax of this upstream of another net), then the nondifferentiability of argmax means that this architechture will not learn anything. I only mention this because you use the phrase "action space", which is typical in DRL. If this is the case, you may want to consider algorithms like REINFORCE where action sequences can be learned discretely and used upstream via policy gradient.
answered Nov 19 '18 at 16:04
rvd
43117
Is there any reason you want it to return a matrix of these actions? Why not just map each of the 27 combinations to integers 0-26? So your architecture could look like [Linear(5, n), ReLU, Linear(n, .) ... Softmax(Linear(., 27))]. Then when you need to evaluate, you can just map it back to the action sequence. This is similar to how in NLP tasks you map multidimensional word vectors to integers via stoi for training and bring them back via itos.
I should point out that if your training paradigm involves some further use of these discrete choices (say you use the argmax of this upstream of another net), then the nondifferentiability of argmax means that this architechture will not learn anything. I only mention this because you use the phrase "action space", which is typical in DRL. If this is the case, you may want to consider algorithms like REINFORCE where action sequences can be learned discretely and used upstream via policy gradient.
answered Nov 19 '18 at 16:04
rvd
43117
answered Nov 19 '18 at 16:04
rvd
43117
answered Nov 19 '18 at 16:04
rvd
43117
answered Nov 19 '18 at 16:04
rvd
43117
43117
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more theargmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here
– Siddhant Tandon
Nov 19 '18 at 16:15
So you want the Q-values of each of the three values fora,b, andc? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.
– rvd
Nov 19 '18 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elementsa,b,cas output and repeat this 27 times ?
– Siddhant Tandon
Nov 19 '18 at 16:28
Not quite. I mean make the network emit a sequence like[a2, b1, c3]and strongly penalize making invalid sequences like 2as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.
– rvd
Nov 19 '18 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 '18 at 16:41
|
show 1 more comment
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more theargmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here
– Siddhant Tandon
Nov 19 '18 at 16:15
So you want the Q-values of each of the three values fora,b, andc? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.
– rvd
Nov 19 '18 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elementsa,b,cas output and repeat this 27 times ?
– Siddhant Tandon
Nov 19 '18 at 16:28
Not quite. I mean make the network emit a sequence like[a2, b1, c3]and strongly penalize making invalid sequences like 2as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.
– rvd
Nov 19 '18 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 '18 at 16:41
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more the
argmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here– Siddhant Tandon
Nov 19 '18 at 16:15
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more the
argmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here– Siddhant Tandon
Nov 19 '18 at 16:15
So you want the Q-values of each of the three values for
a, b, and c? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.– rvd
Nov 19 '18 at 16:19
So you want the Q-values of each of the three values for
a, b, and c? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.– rvd
Nov 19 '18 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elements
a,b,c as output and repeat this 27 times ?– Siddhant Tandon
Nov 19 '18 at 16:28
ah you mean something like feed the input in the network get a vector of 3 elements
a,b,c as output and repeat this 27 times ?– Siddhant Tandon
Nov 19 '18 at 16:28
Not quite. I mean make the network emit a sequence like
[a2, b1, c3] and strongly penalize making invalid sequences like 2 as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.– rvd
Nov 19 '18 at 16:31
Not quite. I mean make the network emit a sequence like
[a2, b1, c3] and strongly penalize making invalid sequences like 2 as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.– rvd
Nov 19 '18 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 '18 at 16:41
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 '18 at 16:41
|
show 1 more comment
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53378284%2freturn-distribution-over-set-of-action-space-from-neural-network%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
