Mixing
find partial derivative of the function
$begingroup$
I have this equation which is basically a maximum likelihood equation for EM-algorithm.
$$L(theta) = sum_{i=1}^n{ln{(sum_{j=1}^kw_jp_j(x_i;theta_j))}}$$
I'm trying to derive a partial derivative by $theta_j$ from it.
What I'm getting is:
$$frac{partial L}{partial theta_j} = sum_{i=1}^n{w_jover p(x_i)}frac{partial p_j(x_i;theta_j)}{partial theta_j}$$
where $$p(x_i)=sum_{j=1}^kw_jp_j(x_i;theta_j)$$
This derivative should be equal to 0 and I can multiply it by $p_j(x_i;theta_j)$ and get
$$frac{partial L}{partial theta_j} = sum_{i=1}^n{w_jp_j(x_i;theta_j)over p(x_i)}frac{partial p_j(x_i;theta_j)}{partial theta_j}=frac{partial sum_{i=1}^n g_{ij}p_j(x_i;theta_j)}{partial theta_j}=0$$
Which is almost as it should be. But the problem is that I somewhere miss the logarithm as it should look like this:
$$frac{partial sum_{i=1}^n g_{ij}ln p_j(x_i;theta_j)}{partial theta_j}=0$$
Where am I wrong?
$g_{ij}={w_jp_j(x_i;theta_j)over p(x_i)}$ which is obtained in the first step of the EM-algorithm.
derivatives maximum-likelihood log-likelihood
edited Jan 10 at 18:45
Kristian S. Jensen
54
asked Jan 10 at 17:53
s0nicYouths0nicYouth
34
$endgroup$
add a comment |
$begingroup$
I have this equation which is basically a maximum likelihood equation for EM-algorithm.
$$L(theta) = sum_{i=1}^n{ln{(sum_{j=1}^kw_jp_j(x_i;theta_j))}}$$
I'm trying to derive a partial derivative by $theta_j$ from it.
What I'm getting is:
$$frac{partial L}{partial theta_j} = sum_{i=1}^n{w_jover p(x_i)}frac{partial p_j(x_i;theta_j)}{partial theta_j}$$
where $$p(x_i)=sum_{j=1}^kw_jp_j(x_i;theta_j)$$
This derivative should be equal to 0 and I can multiply it by $p_j(x_i;theta_j)$ and get
$$frac{partial L}{partial theta_j} = sum_{i=1}^n{w_jp_j(x_i;theta_j)over p(x_i)}frac{partial p_j(x_i;theta_j)}{partial theta_j}=frac{partial sum_{i=1}^n g_{ij}p_j(x_i;theta_j)}{partial theta_j}=0$$
Which is almost as it should be. But the problem is that I somewhere miss the logarithm as it should look like this:
$$frac{partial sum_{i=1}^n g_{ij}ln p_j(x_i;theta_j)}{partial theta_j}=0$$
Where am I wrong?
$g_{ij}={w_jp_j(x_i;theta_j)over p(x_i)}$ which is obtained in the first step of the EM-algorithm.
derivatives maximum-likelihood log-likelihood
edited Jan 10 at 18:45
Kristian S. Jensen
54
asked Jan 10 at 17:53
s0nicYouths0nicYouth
34
$endgroup$
$begingroup$
What are the $g_{i j}$ in the "answer"?
$endgroup$
– coffeemath
Jan 10 at 18:04
$begingroup$
@coffeemath added clarification into the question.
$endgroup$
– s0nicYouth
Jan 10 at 18:09
1
$begingroup$
You multiply and divide by $p_j(x_i;theta_j)$.
$endgroup$
– Math Lover
Jan 10 at 18:13
$begingroup$
@MathLover no. Only multiply. Added clarification that this derivative should be equal to 0.
$endgroup$
– s0nicYouth
Jan 10 at 18:21
add a comment |
$begingroup$
I have this equation which is basically a maximum likelihood equation for EM-algorithm.
$$L(theta) = sum_{i=1}^n{ln{(sum_{j=1}^kw_jp_j(x_i;theta_j))}}$$
I'm trying to derive a partial derivative by $theta_j$ from it.
What I'm getting is:
$$frac{partial L}{partial theta_j} = sum_{i=1}^n{w_jover p(x_i)}frac{partial p_j(x_i;theta_j)}{partial theta_j}$$
where $$p(x_i)=sum_{j=1}^kw_jp_j(x_i;theta_j)$$
This derivative should be equal to 0 and I can multiply it by $p_j(x_i;theta_j)$ and get
$$frac{partial L}{partial theta_j} = sum_{i=1}^n{w_jp_j(x_i;theta_j)over p(x_i)}frac{partial p_j(x_i;theta_j)}{partial theta_j}=frac{partial sum_{i=1}^n g_{ij}p_j(x_i;theta_j)}{partial theta_j}=0$$
Which is almost as it should be. But the problem is that I somewhere miss the logarithm as it should look like this:
$$frac{partial sum_{i=1}^n g_{ij}ln p_j(x_i;theta_j)}{partial theta_j}=0$$
Where am I wrong?
$g_{ij}={w_jp_j(x_i;theta_j)over p(x_i)}$ which is obtained in the first step of the EM-algorithm.
derivatives maximum-likelihood log-likelihood
edited Jan 10 at 18:45
Kristian S. Jensen
54
asked Jan 10 at 17:53
s0nicYouths0nicYouth
34
$endgroup$
I have this equation which is basically a maximum likelihood equation for EM-algorithm.
$$L(theta) = sum_{i=1}^n{ln{(sum_{j=1}^kw_jp_j(x_i;theta_j))}}$$
I'm trying to derive a partial derivative by $theta_j$ from it.
What I'm getting is:
$$frac{partial L}{partial theta_j} = sum_{i=1}^n{w_jover p(x_i)}frac{partial p_j(x_i;theta_j)}{partial theta_j}$$
where $$p(x_i)=sum_{j=1}^kw_jp_j(x_i;theta_j)$$
This derivative should be equal to 0 and I can multiply it by $p_j(x_i;theta_j)$ and get
$$frac{partial L}{partial theta_j} = sum_{i=1}^n{w_jp_j(x_i;theta_j)over p(x_i)}frac{partial p_j(x_i;theta_j)}{partial theta_j}=frac{partial sum_{i=1}^n g_{ij}p_j(x_i;theta_j)}{partial theta_j}=0$$
Which is almost as it should be. But the problem is that I somewhere miss the logarithm as it should look like this:
$$frac{partial sum_{i=1}^n g_{ij}ln p_j(x_i;theta_j)}{partial theta_j}=0$$
Where am I wrong?
$g_{ij}={w_jp_j(x_i;theta_j)over p(x_i)}$ which is obtained in the first step of the EM-algorithm.
derivatives maximum-likelihood log-likelihood
derivatives maximum-likelihood log-likelihood
edited Jan 10 at 18:45
Kristian S. Jensen
54
asked Jan 10 at 17:53
s0nicYouths0nicYouth
34
edited Jan 10 at 18:45
Kristian S. Jensen
54
asked Jan 10 at 17:53
s0nicYouths0nicYouth
34
edited Jan 10 at 18:45
Kristian S. Jensen
54
edited Jan 10 at 18:45
Kristian S. Jensen
54
edited Jan 10 at 18:45
Kristian S. Jensen
54
54
asked Jan 10 at 17:53
s0nicYouths0nicYouth
34
asked Jan 10 at 17:53
s0nicYouths0nicYouth
34
asked Jan 10 at 17:53
s0nicYouths0nicYouth
34
34
$begingroup$
What are the $g_{i j}$ in the "answer"?
$endgroup$
– coffeemath
Jan 10 at 18:04
$begingroup$
@coffeemath added clarification into the question.
$endgroup$
– s0nicYouth
Jan 10 at 18:09
1
$begingroup$
You multiply and divide by $p_j(x_i;theta_j)$.
$endgroup$
– Math Lover
Jan 10 at 18:13
$begingroup$
@MathLover no. Only multiply. Added clarification that this derivative should be equal to 0.
$endgroup$
– s0nicYouth
Jan 10 at 18:21
add a comment |
$begingroup$
What are the $g_{i j}$ in the "answer"?
$endgroup$
– coffeemath
Jan 10 at 18:04
$begingroup$
@coffeemath added clarification into the question.
$endgroup$
– s0nicYouth
Jan 10 at 18:09
1
$begingroup$
You multiply and divide by $p_j(x_i;theta_j)$.
$endgroup$
– Math Lover
Jan 10 at 18:13
$begingroup$
@MathLover no. Only multiply. Added clarification that this derivative should be equal to 0.
$endgroup$
– s0nicYouth
Jan 10 at 18:21
$begingroup$
What are the $g_{i j}$ in the "answer"?
$endgroup$
– coffeemath
Jan 10 at 18:04
$begingroup$
What are the $g_{i j}$ in the "answer"?
$endgroup$
– coffeemath
Jan 10 at 18:04
$begingroup$
@coffeemath added clarification into the question.
$endgroup$
– s0nicYouth
Jan 10 at 18:09
$begingroup$
@coffeemath added clarification into the question.
$endgroup$
– s0nicYouth
Jan 10 at 18:09
1
1
$begingroup$
You multiply and divide by $p_j(x_i;theta_j)$.
$endgroup$
– Math Lover
Jan 10 at 18:13
$begingroup$
You multiply and divide by $p_j(x_i;theta_j)$.
$endgroup$
– Math Lover
Jan 10 at 18:13
$begingroup$
@MathLover no. Only multiply. Added clarification that this derivative should be equal to 0.
$endgroup$
– s0nicYouth
Jan 10 at 18:21
$begingroup$
@MathLover no. Only multiply. Added clarification that this derivative should be equal to 0.
$endgroup$
– s0nicYouth
Jan 10 at 18:21
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
It is not because your derivative is zero that you can multiply all the terms in your summation by different weights $p_j(x_i;theta_j)$ (they are different because they change with $x_i$). You should as mentioned by Math Lover, multiply and divide by $p_j(x_i;theta_j)$ and get
$$frac{partial L}{partial theta_j}(theta) = sum_{i=1}^n{w_jp_j(x_i;theta_j)over p(x_i)}frac{1}{p_j(x_i;theta_j)} frac{partial p_j(x_i;theta_j)}{partial theta_j}=frac{partial sum_{i=1}^n g_{ij}log(p_j(x_i;theta_j))}{partial theta_j}=0.$$
answered Jan 10 at 20:27
BertrandBertrand
16615
$endgroup$
$begingroup$
Thanks, I got it. Stupid mistake.
$endgroup$
– s0nicYouth
Jan 10 at 21:38
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3068966%2ffind-partial-derivative-of-the-function%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
It is not because your derivative is zero that you can multiply all the terms in your summation by different weights $p_j(x_i;theta_j)$ (they are different because they change with $x_i$). You should as mentioned by Math Lover, multiply and divide by $p_j(x_i;theta_j)$ and get
$$frac{partial L}{partial theta_j}(theta) = sum_{i=1}^n{w_jp_j(x_i;theta_j)over p(x_i)}frac{1}{p_j(x_i;theta_j)} frac{partial p_j(x_i;theta_j)}{partial theta_j}=frac{partial sum_{i=1}^n g_{ij}log(p_j(x_i;theta_j))}{partial theta_j}=0.$$
answered Jan 10 at 20:27
BertrandBertrand
16615
$endgroup$
$begingroup$
Thanks, I got it. Stupid mistake.
$endgroup$
– s0nicYouth
Jan 10 at 21:38
add a comment |
$begingroup$
It is not because your derivative is zero that you can multiply all the terms in your summation by different weights $p_j(x_i;theta_j)$ (they are different because they change with $x_i$). You should as mentioned by Math Lover, multiply and divide by $p_j(x_i;theta_j)$ and get
$$frac{partial L}{partial theta_j}(theta) = sum_{i=1}^n{w_jp_j(x_i;theta_j)over p(x_i)}frac{1}{p_j(x_i;theta_j)} frac{partial p_j(x_i;theta_j)}{partial theta_j}=frac{partial sum_{i=1}^n g_{ij}log(p_j(x_i;theta_j))}{partial theta_j}=0.$$
answered Jan 10 at 20:27
BertrandBertrand
16615
$endgroup$
$begingroup$
Thanks, I got it. Stupid mistake.
$endgroup$
– s0nicYouth
Jan 10 at 21:38
add a comment |
$begingroup$
It is not because your derivative is zero that you can multiply all the terms in your summation by different weights $p_j(x_i;theta_j)$ (they are different because they change with $x_i$). You should as mentioned by Math Lover, multiply and divide by $p_j(x_i;theta_j)$ and get
$$frac{partial L}{partial theta_j}(theta) = sum_{i=1}^n{w_jp_j(x_i;theta_j)over p(x_i)}frac{1}{p_j(x_i;theta_j)} frac{partial p_j(x_i;theta_j)}{partial theta_j}=frac{partial sum_{i=1}^n g_{ij}log(p_j(x_i;theta_j))}{partial theta_j}=0.$$
answered Jan 10 at 20:27
BertrandBertrand
16615
$endgroup$
It is not because your derivative is zero that you can multiply all the terms in your summation by different weights $p_j(x_i;theta_j)$ (they are different because they change with $x_i$). You should as mentioned by Math Lover, multiply and divide by $p_j(x_i;theta_j)$ and get
$$frac{partial L}{partial theta_j}(theta) = sum_{i=1}^n{w_jp_j(x_i;theta_j)over p(x_i)}frac{1}{p_j(x_i;theta_j)} frac{partial p_j(x_i;theta_j)}{partial theta_j}=frac{partial sum_{i=1}^n g_{ij}log(p_j(x_i;theta_j))}{partial theta_j}=0.$$
answered Jan 10 at 20:27
BertrandBertrand
16615
answered Jan 10 at 20:27
BertrandBertrand
16615
answered Jan 10 at 20:27
BertrandBertrand
16615
answered Jan 10 at 20:27
BertrandBertrand
16615
16615
$begingroup$
Thanks, I got it. Stupid mistake.
$endgroup$
– s0nicYouth
Jan 10 at 21:38
add a comment |
$begingroup$
Thanks, I got it. Stupid mistake.
$endgroup$
– s0nicYouth
Jan 10 at 21:38
$begingroup$
Thanks, I got it. Stupid mistake.
$endgroup$
– s0nicYouth
Jan 10 at 21:38
$begingroup$
Thanks, I got it. Stupid mistake.
$endgroup$
– s0nicYouth
Jan 10 at 21:38
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3068966%2ffind-partial-derivative-of-the-function%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
What are the $g_{i j}$ in the "answer"?
$endgroup$
– coffeemath
Jan 10 at 18:04
$begingroup$
@coffeemath added clarification into the question.
$endgroup$
– s0nicYouth
Jan 10 at 18:09
1
$begingroup$
You multiply and divide by $p_j(x_i;theta_j)$.
$endgroup$
– Math Lover
Jan 10 at 18:13
$begingroup$
@MathLover no. Only multiply. Added clarification that this derivative should be equal to 0.
$endgroup$
– s0nicYouth
Jan 10 at 18:21