Mixing
How to use log probabilities in PCA mixture EM algorithm
$begingroup$
I'm trying to implement PCA mixtures (Tipping & Bishop 2006 Appendix C) on the Tobomovirus. I'll summarize the mathematical background and algorithm here:
For a single PCA model, we assume a latent variable model
$$
mathbf{t} = mathbf{Wx} + boldsymbolmu + boldsymbolepsilon.
$$
This implies a probability distribution
$$
p(mathbf{t}|mathbf{x}) = (2pisigma^2)^{-d/2}mathrm{exp}{-frac{1}{2sigma^2}||mathbf{t} - mathbf{Wx}- boldsymbolmu||^2}
$$
since the noise term $boldsymbolepsilon$ is zero-mean Gaussian with variance $sigma^2$.
We define the Gaussian prior
$$
p(mathbf{x}) = (2pi)^{-q/2}mathrm{exp}{-frac{1}{2}mathbf{x}^Tmathbf{x}}
$$
which allows us to derive the marginal distribution
$$
p(mathbf{t}) = (2pi)^{-d/2}|mathbf{C}|^{-1/2}mathrm{exp}{-frac{1}{2}(mathbf{t} - boldsymbolmu)^Tmathbf{C}^-1(mathbf{t} - boldsymbolmu)}
$$
where $mathbf{C} = sigma^2mathbf{I} + mathbf{WW}^T$.
The PCA mixture model means that we use a mixture of PCA models like this:
$$
p(mathbf{t}) = sum_{i=1}^M pi_i p(mathbf{t}|i).
$$
The algorithm:
- Initialize $pi_i$, $boldsymbolmu_i$, $mathbf{W}_i$, $sigma_i^2$. These are the mixing coefficient, mean (of the marginal), weight matrix and variance of each model.
- Calculate the responsibilities,
$$
R_{ni} = frac{p(mathbf{t}_n|i)pi_i}{p(mathbf{t}_n)}
$$
where $p(mathbf{t}|i)$ is the probability for a single PCA model $i$.
- Calculate mixing coefficients
$$
tildepi_i = frac{1}{N}sum^N_{n=1} R_{ni}.
$$
- Calculate the means
$$
tilde{boldsymbolmu}_i = frac{sum_{n=1}^N R_{ni}mathbf{t}_n}{sum_{n=1}^NR_{ni}}
$$
- Calculate the new weight matrices
$$
mathbf{tilde W}_i = mathbf{S}_i mathbf{W}_i(sigma^2 mathbf{I}+mathbf{M}^{-1}mathbf{W}_i^Tmathbf{S}_imathbf{W}_i)^{-1}
$$
where $mathbf{S}_i = frac{1}{tildepi_i N}sum^N_{n=1}R_{ni}(mathbf{t}_n - tilde{boldsymbolmu})(mathbf{t}_n - tilde{boldsymbolmu})^T$.
- Repeat from step 2.
The tilde operator means that it is a new parameter being calculated.
Here's my problem:
When implementing this on the Tobomovirus dataset, the long 18x1 feature vectors and the resulting 18x18 $mathbf{C}$ matrix lead to the probabilities $p(mathbf{t})$ in step 2 becoming very very small, leading to underflow. I would like to work with the log probabilities instead, but I'm not sure how this will affect the algorithm. Could I just set
$$
R_{ni} = frac{ln(p(mathbf{t}_n|i))pi_i}{ln(p(mathbf{t}))}
$$
in step 2 and then leave the rest of the algorithm unchanged?
probability-theory machine-learning
asked Jan 9 at 16:55
SandiSandi
255112
$endgroup$
add a comment |
$begingroup$
I'm trying to implement PCA mixtures (Tipping & Bishop 2006 Appendix C) on the Tobomovirus. I'll summarize the mathematical background and algorithm here:
For a single PCA model, we assume a latent variable model
$$
mathbf{t} = mathbf{Wx} + boldsymbolmu + boldsymbolepsilon.
$$
This implies a probability distribution
$$
p(mathbf{t}|mathbf{x}) = (2pisigma^2)^{-d/2}mathrm{exp}{-frac{1}{2sigma^2}||mathbf{t} - mathbf{Wx}- boldsymbolmu||^2}
$$
since the noise term $boldsymbolepsilon$ is zero-mean Gaussian with variance $sigma^2$.
We define the Gaussian prior
$$
p(mathbf{x}) = (2pi)^{-q/2}mathrm{exp}{-frac{1}{2}mathbf{x}^Tmathbf{x}}
$$
which allows us to derive the marginal distribution
$$
p(mathbf{t}) = (2pi)^{-d/2}|mathbf{C}|^{-1/2}mathrm{exp}{-frac{1}{2}(mathbf{t} - boldsymbolmu)^Tmathbf{C}^-1(mathbf{t} - boldsymbolmu)}
$$
where $mathbf{C} = sigma^2mathbf{I} + mathbf{WW}^T$.
The PCA mixture model means that we use a mixture of PCA models like this:
$$
p(mathbf{t}) = sum_{i=1}^M pi_i p(mathbf{t}|i).
$$
The algorithm:
- Initialize $pi_i$, $boldsymbolmu_i$, $mathbf{W}_i$, $sigma_i^2$. These are the mixing coefficient, mean (of the marginal), weight matrix and variance of each model.
- Calculate the responsibilities,
$$
R_{ni} = frac{p(mathbf{t}_n|i)pi_i}{p(mathbf{t}_n)}
$$
where $p(mathbf{t}|i)$ is the probability for a single PCA model $i$.
- Calculate mixing coefficients
$$
tildepi_i = frac{1}{N}sum^N_{n=1} R_{ni}.
$$
- Calculate the means
$$
tilde{boldsymbolmu}_i = frac{sum_{n=1}^N R_{ni}mathbf{t}_n}{sum_{n=1}^NR_{ni}}
$$
- Calculate the new weight matrices
$$
mathbf{tilde W}_i = mathbf{S}_i mathbf{W}_i(sigma^2 mathbf{I}+mathbf{M}^{-1}mathbf{W}_i^Tmathbf{S}_imathbf{W}_i)^{-1}
$$
where $mathbf{S}_i = frac{1}{tildepi_i N}sum^N_{n=1}R_{ni}(mathbf{t}_n - tilde{boldsymbolmu})(mathbf{t}_n - tilde{boldsymbolmu})^T$.
- Repeat from step 2.
The tilde operator means that it is a new parameter being calculated.
Here's my problem:
When implementing this on the Tobomovirus dataset, the long 18x1 feature vectors and the resulting 18x18 $mathbf{C}$ matrix lead to the probabilities $p(mathbf{t})$ in step 2 becoming very very small, leading to underflow. I would like to work with the log probabilities instead, but I'm not sure how this will affect the algorithm. Could I just set
$$
R_{ni} = frac{ln(p(mathbf{t}_n|i))pi_i}{ln(p(mathbf{t}))}
$$
in step 2 and then leave the rest of the algorithm unchanged?
probability-theory machine-learning
asked Jan 9 at 16:55
SandiSandi
255112
$endgroup$
add a comment |
$begingroup$
I'm trying to implement PCA mixtures (Tipping & Bishop 2006 Appendix C) on the Tobomovirus. I'll summarize the mathematical background and algorithm here:
For a single PCA model, we assume a latent variable model
$$
mathbf{t} = mathbf{Wx} + boldsymbolmu + boldsymbolepsilon.
$$
This implies a probability distribution
$$
p(mathbf{t}|mathbf{x}) = (2pisigma^2)^{-d/2}mathrm{exp}{-frac{1}{2sigma^2}||mathbf{t} - mathbf{Wx}- boldsymbolmu||^2}
$$
since the noise term $boldsymbolepsilon$ is zero-mean Gaussian with variance $sigma^2$.
We define the Gaussian prior
$$
p(mathbf{x}) = (2pi)^{-q/2}mathrm{exp}{-frac{1}{2}mathbf{x}^Tmathbf{x}}
$$
which allows us to derive the marginal distribution
$$
p(mathbf{t}) = (2pi)^{-d/2}|mathbf{C}|^{-1/2}mathrm{exp}{-frac{1}{2}(mathbf{t} - boldsymbolmu)^Tmathbf{C}^-1(mathbf{t} - boldsymbolmu)}
$$
where $mathbf{C} = sigma^2mathbf{I} + mathbf{WW}^T$.
The PCA mixture model means that we use a mixture of PCA models like this:
$$
p(mathbf{t}) = sum_{i=1}^M pi_i p(mathbf{t}|i).
$$
The algorithm:
- Initialize $pi_i$, $boldsymbolmu_i$, $mathbf{W}_i$, $sigma_i^2$. These are the mixing coefficient, mean (of the marginal), weight matrix and variance of each model.
- Calculate the responsibilities,
$$
R_{ni} = frac{p(mathbf{t}_n|i)pi_i}{p(mathbf{t}_n)}
$$
where $p(mathbf{t}|i)$ is the probability for a single PCA model $i$.
- Calculate mixing coefficients
$$
tildepi_i = frac{1}{N}sum^N_{n=1} R_{ni}.
$$
- Calculate the means
$$
tilde{boldsymbolmu}_i = frac{sum_{n=1}^N R_{ni}mathbf{t}_n}{sum_{n=1}^NR_{ni}}
$$
- Calculate the new weight matrices
$$
mathbf{tilde W}_i = mathbf{S}_i mathbf{W}_i(sigma^2 mathbf{I}+mathbf{M}^{-1}mathbf{W}_i^Tmathbf{S}_imathbf{W}_i)^{-1}
$$
where $mathbf{S}_i = frac{1}{tildepi_i N}sum^N_{n=1}R_{ni}(mathbf{t}_n - tilde{boldsymbolmu})(mathbf{t}_n - tilde{boldsymbolmu})^T$.
- Repeat from step 2.
The tilde operator means that it is a new parameter being calculated.
Here's my problem:
When implementing this on the Tobomovirus dataset, the long 18x1 feature vectors and the resulting 18x18 $mathbf{C}$ matrix lead to the probabilities $p(mathbf{t})$ in step 2 becoming very very small, leading to underflow. I would like to work with the log probabilities instead, but I'm not sure how this will affect the algorithm. Could I just set
$$
R_{ni} = frac{ln(p(mathbf{t}_n|i))pi_i}{ln(p(mathbf{t}))}
$$
in step 2 and then leave the rest of the algorithm unchanged?
probability-theory machine-learning
asked Jan 9 at 16:55
SandiSandi
255112
$endgroup$
I'm trying to implement PCA mixtures (Tipping & Bishop 2006 Appendix C) on the Tobomovirus. I'll summarize the mathematical background and algorithm here:
For a single PCA model, we assume a latent variable model
$$
mathbf{t} = mathbf{Wx} + boldsymbolmu + boldsymbolepsilon.
$$
This implies a probability distribution
$$
p(mathbf{t}|mathbf{x}) = (2pisigma^2)^{-d/2}mathrm{exp}{-frac{1}{2sigma^2}||mathbf{t} - mathbf{Wx}- boldsymbolmu||^2}
$$
since the noise term $boldsymbolepsilon$ is zero-mean Gaussian with variance $sigma^2$.
We define the Gaussian prior
$$
p(mathbf{x}) = (2pi)^{-q/2}mathrm{exp}{-frac{1}{2}mathbf{x}^Tmathbf{x}}
$$
which allows us to derive the marginal distribution
$$
p(mathbf{t}) = (2pi)^{-d/2}|mathbf{C}|^{-1/2}mathrm{exp}{-frac{1}{2}(mathbf{t} - boldsymbolmu)^Tmathbf{C}^-1(mathbf{t} - boldsymbolmu)}
$$
where $mathbf{C} = sigma^2mathbf{I} + mathbf{WW}^T$.
The PCA mixture model means that we use a mixture of PCA models like this:
$$
p(mathbf{t}) = sum_{i=1}^M pi_i p(mathbf{t}|i).
$$
The algorithm:
- Initialize $pi_i$, $boldsymbolmu_i$, $mathbf{W}_i$, $sigma_i^2$. These are the mixing coefficient, mean (of the marginal), weight matrix and variance of each model.
- Calculate the responsibilities,
$$
R_{ni} = frac{p(mathbf{t}_n|i)pi_i}{p(mathbf{t}_n)}
$$
where $p(mathbf{t}|i)$ is the probability for a single PCA model $i$.
- Calculate mixing coefficients
$$
tildepi_i = frac{1}{N}sum^N_{n=1} R_{ni}.
$$
- Calculate the means
$$
tilde{boldsymbolmu}_i = frac{sum_{n=1}^N R_{ni}mathbf{t}_n}{sum_{n=1}^NR_{ni}}
$$
- Calculate the new weight matrices
$$
mathbf{tilde W}_i = mathbf{S}_i mathbf{W}_i(sigma^2 mathbf{I}+mathbf{M}^{-1}mathbf{W}_i^Tmathbf{S}_imathbf{W}_i)^{-1}
$$
where $mathbf{S}_i = frac{1}{tildepi_i N}sum^N_{n=1}R_{ni}(mathbf{t}_n - tilde{boldsymbolmu})(mathbf{t}_n - tilde{boldsymbolmu})^T$.
- Repeat from step 2.
The tilde operator means that it is a new parameter being calculated.
Here's my problem:
When implementing this on the Tobomovirus dataset, the long 18x1 feature vectors and the resulting 18x18 $mathbf{C}$ matrix lead to the probabilities $p(mathbf{t})$ in step 2 becoming very very small, leading to underflow. I would like to work with the log probabilities instead, but I'm not sure how this will affect the algorithm. Could I just set
$$
R_{ni} = frac{ln(p(mathbf{t}_n|i))pi_i}{ln(p(mathbf{t}))}
$$
in step 2 and then leave the rest of the algorithm unchanged?
probability-theory machine-learning
probability-theory machine-learning
asked Jan 9 at 16:55
SandiSandi
255112
asked Jan 9 at 16:55
SandiSandi
255112
edited Jan 10 at 0:38
Sandi
asked Jan 9 at 16:55
SandiSandi
255112
asked Jan 9 at 16:55
SandiSandi
255112
asked Jan 9 at 16:55
SandiSandi
255112
255112
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3067693%2fhow-to-use-log-probabilities-in-pca-mixture-em-algorithm%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3067693%2fhow-to-use-log-probabilities-in-pca-mixture-em-algorithm%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
