Mixing
How to find complete log likelihood for mixture of PPCA
$begingroup$
In Appendix C of a paper by Michael E. Tipping and Christopher M. Bishop about mixture models for probabilistic PCA, the probability of a single data vector $mathbf{t}$ is expressed as a mixture of PCA models (equation 69):
$$
p(mathbf{t}) = sum_{i=1}^Mpi_i p(mathbf{t}|i)
$$
where $pi$ is the mixing proportion and $p(mathbf{t}|i)$ is a single probabilistic PCA model.
The model underlying the probabilistic PCA method is (equation 2)
$$
mathbf{t} = mathbf{Wx} + boldsymbolmu + boldsymbolepsilon.
$$
Where $mathbf{x}$ is a latent variable. By introducing a new set of variables $z_{ni}$ "labelling which model is responsible for generating each data point $mathbf{t}_n$", Bishop formulates the complete log likelihood as (equation 70):
$$
mathcal{L}_C = sum_{n=1}^Nsum_{i=1}^Mz_{ni}ln{pi_ip(mathbf{t}_n, mathbf{x}_{ni})}.
$$
I would like to understand how he derives this expression as he doesn't provide a solution himself. How is this expression for the complete log likelihood found?
probability-theory machine-learning
asked Jan 4 at 11:07
SandiSandi
255112
$endgroup$
add a comment |
$begingroup$
In Appendix C of a paper by Michael E. Tipping and Christopher M. Bishop about mixture models for probabilistic PCA, the probability of a single data vector $mathbf{t}$ is expressed as a mixture of PCA models (equation 69):
$$
p(mathbf{t}) = sum_{i=1}^Mpi_i p(mathbf{t}|i)
$$
where $pi$ is the mixing proportion and $p(mathbf{t}|i)$ is a single probabilistic PCA model.
The model underlying the probabilistic PCA method is (equation 2)
$$
mathbf{t} = mathbf{Wx} + boldsymbolmu + boldsymbolepsilon.
$$
Where $mathbf{x}$ is a latent variable. By introducing a new set of variables $z_{ni}$ "labelling which model is responsible for generating each data point $mathbf{t}_n$", Bishop formulates the complete log likelihood as (equation 70):
$$
mathcal{L}_C = sum_{n=1}^Nsum_{i=1}^Mz_{ni}ln{pi_ip(mathbf{t}_n, mathbf{x}_{ni})}.
$$
I would like to understand how he derives this expression as he doesn't provide a solution himself. How is this expression for the complete log likelihood found?
probability-theory machine-learning
asked Jan 4 at 11:07
SandiSandi
255112
$endgroup$
add a comment |
$begingroup$
In Appendix C of a paper by Michael E. Tipping and Christopher M. Bishop about mixture models for probabilistic PCA, the probability of a single data vector $mathbf{t}$ is expressed as a mixture of PCA models (equation 69):
$$
p(mathbf{t}) = sum_{i=1}^Mpi_i p(mathbf{t}|i)
$$
where $pi$ is the mixing proportion and $p(mathbf{t}|i)$ is a single probabilistic PCA model.
The model underlying the probabilistic PCA method is (equation 2)
$$
mathbf{t} = mathbf{Wx} + boldsymbolmu + boldsymbolepsilon.
$$
Where $mathbf{x}$ is a latent variable. By introducing a new set of variables $z_{ni}$ "labelling which model is responsible for generating each data point $mathbf{t}_n$", Bishop formulates the complete log likelihood as (equation 70):
$$
mathcal{L}_C = sum_{n=1}^Nsum_{i=1}^Mz_{ni}ln{pi_ip(mathbf{t}_n, mathbf{x}_{ni})}.
$$
I would like to understand how he derives this expression as he doesn't provide a solution himself. How is this expression for the complete log likelihood found?
probability-theory machine-learning
asked Jan 4 at 11:07
SandiSandi
255112
$endgroup$
In Appendix C of a paper by Michael E. Tipping and Christopher M. Bishop about mixture models for probabilistic PCA, the probability of a single data vector $mathbf{t}$ is expressed as a mixture of PCA models (equation 69):
$$
p(mathbf{t}) = sum_{i=1}^Mpi_i p(mathbf{t}|i)
$$
where $pi$ is the mixing proportion and $p(mathbf{t}|i)$ is a single probabilistic PCA model.
The model underlying the probabilistic PCA method is (equation 2)
$$
mathbf{t} = mathbf{Wx} + boldsymbolmu + boldsymbolepsilon.
$$
Where $mathbf{x}$ is a latent variable. By introducing a new set of variables $z_{ni}$ "labelling which model is responsible for generating each data point $mathbf{t}_n$", Bishop formulates the complete log likelihood as (equation 70):
$$
mathcal{L}_C = sum_{n=1}^Nsum_{i=1}^Mz_{ni}ln{pi_ip(mathbf{t}_n, mathbf{x}_{ni})}.
$$
I would like to understand how he derives this expression as he doesn't provide a solution himself. How is this expression for the complete log likelihood found?
probability-theory machine-learning
probability-theory machine-learning
asked Jan 4 at 11:07
SandiSandi
255112
asked Jan 4 at 11:07
SandiSandi
255112
asked Jan 4 at 11:07
SandiSandi
255112
asked Jan 4 at 11:07
SandiSandi
255112
asked Jan 4 at 11:07
SandiSandi
255112
255112
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
Let's concentrate for the time being on the $n^{rm th}$ datapoint, $mathbf t_n$. Suppose this datapoint is generated from the $i_n^{rm th}$ model. Then $$ z_{ni} = begin{cases} 1 & {rm if } i = i_n \ 0 & {rm otherwise}end{cases}.$$Thus we have
$$sum_{i=1}^M z_{ni} ln left(pi_i p(mathbf t_n ,mathbf x_{ni}) right) = ln left( pi_{i_n} p(mathbf t_n , mathbf x_{n{i_n}})right).$$
The expression on the right-hand side is log-likelihood for the $n$th datapoint. To spell it out:
$pi_{i_n}$ is the probability that the $n^{rm th}$ datapoint is generated by the $i_n^{rm th}$ model.
$p(mathbf t_n, mathbf x_{ni_n})$ is the probability of encountering this particular latent vector $mathbf x_{ni_n}$ and this particular visible vector $mathbf t_n$ for the $n^{rm th}$ datapoint, given that this datapoint is generated from the the $i_n^{rm th}$ model. [In fact, $$p(mathbf t_n, mathbf x_{ni_n}) = mathcal N(mathbf x_{ni_n} | mathbf 0, mathbf I) times mathcal N(mathbf t_n - mathbf W mathbf x_{ni_n} - mathbf mu_n |mathbf 0, sigma_n^2 mathbf I),$$ assuming that $mathbf epsilon_{n} sim mathcal N(mathbf 0, sigma_n^2 mathbf I)$.]
Since the datapoints are generated independently, the log-likelihood for the entire dataset is a sum over the log-likelihoods for the individual datapoints, giving the desired result.
answered Jan 6 at 20:56
Kenny WongKenny Wong
18.5k21438
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3061531%2fhow-to-find-complete-log-likelihood-for-mixture-of-ppca%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Let's concentrate for the time being on the $n^{rm th}$ datapoint, $mathbf t_n$. Suppose this datapoint is generated from the $i_n^{rm th}$ model. Then $$ z_{ni} = begin{cases} 1 & {rm if } i = i_n \ 0 & {rm otherwise}end{cases}.$$Thus we have
$$sum_{i=1}^M z_{ni} ln left(pi_i p(mathbf t_n ,mathbf x_{ni}) right) = ln left( pi_{i_n} p(mathbf t_n , mathbf x_{n{i_n}})right).$$
The expression on the right-hand side is log-likelihood for the $n$th datapoint. To spell it out:
$pi_{i_n}$ is the probability that the $n^{rm th}$ datapoint is generated by the $i_n^{rm th}$ model.
$p(mathbf t_n, mathbf x_{ni_n})$ is the probability of encountering this particular latent vector $mathbf x_{ni_n}$ and this particular visible vector $mathbf t_n$ for the $n^{rm th}$ datapoint, given that this datapoint is generated from the the $i_n^{rm th}$ model. [In fact, $$p(mathbf t_n, mathbf x_{ni_n}) = mathcal N(mathbf x_{ni_n} | mathbf 0, mathbf I) times mathcal N(mathbf t_n - mathbf W mathbf x_{ni_n} - mathbf mu_n |mathbf 0, sigma_n^2 mathbf I),$$ assuming that $mathbf epsilon_{n} sim mathcal N(mathbf 0, sigma_n^2 mathbf I)$.]
Since the datapoints are generated independently, the log-likelihood for the entire dataset is a sum over the log-likelihoods for the individual datapoints, giving the desired result.
answered Jan 6 at 20:56
Kenny WongKenny Wong
18.5k21438
$endgroup$
add a comment |
$begingroup$
Let's concentrate for the time being on the $n^{rm th}$ datapoint, $mathbf t_n$. Suppose this datapoint is generated from the $i_n^{rm th}$ model. Then $$ z_{ni} = begin{cases} 1 & {rm if } i = i_n \ 0 & {rm otherwise}end{cases}.$$Thus we have
$$sum_{i=1}^M z_{ni} ln left(pi_i p(mathbf t_n ,mathbf x_{ni}) right) = ln left( pi_{i_n} p(mathbf t_n , mathbf x_{n{i_n}})right).$$
The expression on the right-hand side is log-likelihood for the $n$th datapoint. To spell it out:
$pi_{i_n}$ is the probability that the $n^{rm th}$ datapoint is generated by the $i_n^{rm th}$ model.
$p(mathbf t_n, mathbf x_{ni_n})$ is the probability of encountering this particular latent vector $mathbf x_{ni_n}$ and this particular visible vector $mathbf t_n$ for the $n^{rm th}$ datapoint, given that this datapoint is generated from the the $i_n^{rm th}$ model. [In fact, $$p(mathbf t_n, mathbf x_{ni_n}) = mathcal N(mathbf x_{ni_n} | mathbf 0, mathbf I) times mathcal N(mathbf t_n - mathbf W mathbf x_{ni_n} - mathbf mu_n |mathbf 0, sigma_n^2 mathbf I),$$ assuming that $mathbf epsilon_{n} sim mathcal N(mathbf 0, sigma_n^2 mathbf I)$.]
Since the datapoints are generated independently, the log-likelihood for the entire dataset is a sum over the log-likelihoods for the individual datapoints, giving the desired result.
answered Jan 6 at 20:56
Kenny WongKenny Wong
18.5k21438
$endgroup$
add a comment |
$begingroup$
Let's concentrate for the time being on the $n^{rm th}$ datapoint, $mathbf t_n$. Suppose this datapoint is generated from the $i_n^{rm th}$ model. Then $$ z_{ni} = begin{cases} 1 & {rm if } i = i_n \ 0 & {rm otherwise}end{cases}.$$Thus we have
$$sum_{i=1}^M z_{ni} ln left(pi_i p(mathbf t_n ,mathbf x_{ni}) right) = ln left( pi_{i_n} p(mathbf t_n , mathbf x_{n{i_n}})right).$$
The expression on the right-hand side is log-likelihood for the $n$th datapoint. To spell it out:
$pi_{i_n}$ is the probability that the $n^{rm th}$ datapoint is generated by the $i_n^{rm th}$ model.
$p(mathbf t_n, mathbf x_{ni_n})$ is the probability of encountering this particular latent vector $mathbf x_{ni_n}$ and this particular visible vector $mathbf t_n$ for the $n^{rm th}$ datapoint, given that this datapoint is generated from the the $i_n^{rm th}$ model. [In fact, $$p(mathbf t_n, mathbf x_{ni_n}) = mathcal N(mathbf x_{ni_n} | mathbf 0, mathbf I) times mathcal N(mathbf t_n - mathbf W mathbf x_{ni_n} - mathbf mu_n |mathbf 0, sigma_n^2 mathbf I),$$ assuming that $mathbf epsilon_{n} sim mathcal N(mathbf 0, sigma_n^2 mathbf I)$.]
Since the datapoints are generated independently, the log-likelihood for the entire dataset is a sum over the log-likelihoods for the individual datapoints, giving the desired result.
answered Jan 6 at 20:56
Kenny WongKenny Wong
18.5k21438
$endgroup$
Let's concentrate for the time being on the $n^{rm th}$ datapoint, $mathbf t_n$. Suppose this datapoint is generated from the $i_n^{rm th}$ model. Then $$ z_{ni} = begin{cases} 1 & {rm if } i = i_n \ 0 & {rm otherwise}end{cases}.$$Thus we have
$$sum_{i=1}^M z_{ni} ln left(pi_i p(mathbf t_n ,mathbf x_{ni}) right) = ln left( pi_{i_n} p(mathbf t_n , mathbf x_{n{i_n}})right).$$
The expression on the right-hand side is log-likelihood for the $n$th datapoint. To spell it out:
$pi_{i_n}$ is the probability that the $n^{rm th}$ datapoint is generated by the $i_n^{rm th}$ model.
$p(mathbf t_n, mathbf x_{ni_n})$ is the probability of encountering this particular latent vector $mathbf x_{ni_n}$ and this particular visible vector $mathbf t_n$ for the $n^{rm th}$ datapoint, given that this datapoint is generated from the the $i_n^{rm th}$ model. [In fact, $$p(mathbf t_n, mathbf x_{ni_n}) = mathcal N(mathbf x_{ni_n} | mathbf 0, mathbf I) times mathcal N(mathbf t_n - mathbf W mathbf x_{ni_n} - mathbf mu_n |mathbf 0, sigma_n^2 mathbf I),$$ assuming that $mathbf epsilon_{n} sim mathcal N(mathbf 0, sigma_n^2 mathbf I)$.]
Since the datapoints are generated independently, the log-likelihood for the entire dataset is a sum over the log-likelihoods for the individual datapoints, giving the desired result.
answered Jan 6 at 20:56
Kenny WongKenny Wong
18.5k21438
answered Jan 6 at 20:56
Kenny WongKenny Wong
18.5k21438
answered Jan 6 at 20:56
Kenny WongKenny Wong
18.5k21438
answered Jan 6 at 20:56
Kenny WongKenny Wong
18.5k21438
18.5k21438
add a comment |
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3061531%2fhow-to-find-complete-log-likelihood-for-mixture-of-ppca%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
