Mixing
Optimization problem / Derivative of matrix Confusion
I have attached a small snippet from my lecture notes where we are dealing with constrained optimization. $E,C$ are matrices.
I am confused with Eq. (4.12) as to how they have differentiated with respect to $E$ when $E^T$ is present. Then, I do not know how they get from Eq. (4.12) to Eq. (4.13).
multivariable-calculus optimization lagrange-multiplier
asked Dec 31 '18 at 18:19
Thomas MooreThomas Moore
419410
|
show 3 more comments
I have attached a small snippet from my lecture notes where we are dealing with constrained optimization. $E,C$ are matrices.
I am confused with Eq. (4.12) as to how they have differentiated with respect to $E$ when $E^T$ is present. Then, I do not know how they get from Eq. (4.12) to Eq. (4.13).
multivariable-calculus optimization lagrange-multiplier
asked Dec 31 '18 at 18:19
Thomas MooreThomas Moore
419410
1
Are $C, E$ symmetric matrices?
– Cesareo
Dec 31 '18 at 18:27
have you computed the derivative of (4.10) with respect to $E_{ij}$?
– LinAlg
Dec 31 '18 at 18:27
@Cesareo E is a rotation matrix, so it is not symmetric in general. However, C is symmetric.
– Thomas Moore
Dec 31 '18 at 18:29
@LinAlg I tried this, but I do not know how to re-write the first term so that it is a scalar.
– Thomas Moore
Dec 31 '18 at 18:31
You should update the question to add relevant information on $C$, $E$, etc.
– copper.hat
Dec 31 '18 at 18:32
|
show 3 more comments
I have attached a small snippet from my lecture notes where we are dealing with constrained optimization. $E,C$ are matrices.
I am confused with Eq. (4.12) as to how they have differentiated with respect to $E$ when $E^T$ is present. Then, I do not know how they get from Eq. (4.12) to Eq. (4.13).
multivariable-calculus optimization lagrange-multiplier
asked Dec 31 '18 at 18:19
Thomas MooreThomas Moore
419410
I have attached a small snippet from my lecture notes where we are dealing with constrained optimization. $E,C$ are matrices.
I am confused with Eq. (4.12) as to how they have differentiated with respect to $E$ when $E^T$ is present. Then, I do not know how they get from Eq. (4.12) to Eq. (4.13).
multivariable-calculus optimization lagrange-multiplier
multivariable-calculus optimization lagrange-multiplier
asked Dec 31 '18 at 18:19
Thomas MooreThomas Moore
419410
asked Dec 31 '18 at 18:19
Thomas MooreThomas Moore
419410
asked Dec 31 '18 at 18:19
Thomas MooreThomas Moore
419410
asked Dec 31 '18 at 18:19
Thomas MooreThomas Moore
419410
asked Dec 31 '18 at 18:19
Thomas MooreThomas Moore
419410
419410
1
Are $C, E$ symmetric matrices?
– Cesareo
Dec 31 '18 at 18:27
have you computed the derivative of (4.10) with respect to $E_{ij}$?
– LinAlg
Dec 31 '18 at 18:27
@Cesareo E is a rotation matrix, so it is not symmetric in general. However, C is symmetric.
– Thomas Moore
Dec 31 '18 at 18:29
@LinAlg I tried this, but I do not know how to re-write the first term so that it is a scalar.
– Thomas Moore
Dec 31 '18 at 18:31
You should update the question to add relevant information on $C$, $E$, etc.
– copper.hat
Dec 31 '18 at 18:32
|
show 3 more comments
1
Are $C, E$ symmetric matrices?
– Cesareo
Dec 31 '18 at 18:27
have you computed the derivative of (4.10) with respect to $E_{ij}$?
– LinAlg
Dec 31 '18 at 18:27
@Cesareo E is a rotation matrix, so it is not symmetric in general. However, C is symmetric.
– Thomas Moore
Dec 31 '18 at 18:29
@LinAlg I tried this, but I do not know how to re-write the first term so that it is a scalar.
– Thomas Moore
Dec 31 '18 at 18:31
You should update the question to add relevant information on $C$, $E$, etc.
– copper.hat
Dec 31 '18 at 18:32
1
1
Are $C, E$ symmetric matrices?
– Cesareo
Dec 31 '18 at 18:27
Are $C, E$ symmetric matrices?
– Cesareo
Dec 31 '18 at 18:27
have you computed the derivative of (4.10) with respect to $E_{ij}$?
– LinAlg
Dec 31 '18 at 18:27
have you computed the derivative of (4.10) with respect to $E_{ij}$?
– LinAlg
Dec 31 '18 at 18:27
@Cesareo E is a rotation matrix, so it is not symmetric in general. However, C is symmetric.
– Thomas Moore
Dec 31 '18 at 18:29
@Cesareo E is a rotation matrix, so it is not symmetric in general. However, C is symmetric.
– Thomas Moore
Dec 31 '18 at 18:29
@LinAlg I tried this, but I do not know how to re-write the first term so that it is a scalar.
– Thomas Moore
Dec 31 '18 at 18:31
@LinAlg I tried this, but I do not know how to re-write the first term so that it is a scalar.
– Thomas Moore
Dec 31 '18 at 18:31
You should update the question to add relevant information on $C$, $E$, etc.
– copper.hat
Dec 31 '18 at 18:32
You should update the question to add relevant information on $C$, $E$, etc.
– copper.hat
Dec 31 '18 at 18:32
|
show 3 more comments
1 Answer
1
active
oldest
votes
I am assuming that $C$ is symmetric.
Let $phi(E) = E^T CE -lambda E^T E$, then
$phi(E+H) = phi(E)+H^TCE+ E^TCH -lambda H^T E-lambda E^T H +r(H)$, where
$|r(H)| le K |H|^2$.
Then $D phi(E)H = H^T(CE - lambda E) + ( CE - lambda E)^T H$.
Suppose $D phi(E)(H) = 0$ for all $H$, then choose $H=CE - lambda E$ to get
$2 (CE - lambda E)^T (CE - lambda E) = 0$ from which we get
$CE - lambda E = 0$ (since $A=0$ iff $A^TA = 0$).
I believe the derivation given in the question is incorrect, or at least misleading.
answered Dec 31 '18 at 19:00
copper.hatcopper.hat
126k559160
For a bonus point: have you found a minimum or a maximum?
– LinAlg
Dec 31 '18 at 19:19
@LinAlg: I would need to know what the optimisation problem is first... Most Lagrangians I have encountered are scalar valued :-).
– copper.hat
Dec 31 '18 at 19:23
A Google search on "the variance (4.7) can be maximized" reveals a background of principal component analysis. $C$ is a covariance matrix, and the constraint is $E^TE = 1$. I agree that minimizing a matrix valued function is odd at least.
– LinAlg
Dec 31 '18 at 19:29
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3057928%2foptimization-problem-derivative-of-matrix-confusion%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I am assuming that $C$ is symmetric.
Let $phi(E) = E^T CE -lambda E^T E$, then
$phi(E+H) = phi(E)+H^TCE+ E^TCH -lambda H^T E-lambda E^T H +r(H)$, where
$|r(H)| le K |H|^2$.
Then $D phi(E)H = H^T(CE - lambda E) + ( CE - lambda E)^T H$.
Suppose $D phi(E)(H) = 0$ for all $H$, then choose $H=CE - lambda E$ to get
$2 (CE - lambda E)^T (CE - lambda E) = 0$ from which we get
$CE - lambda E = 0$ (since $A=0$ iff $A^TA = 0$).
I believe the derivation given in the question is incorrect, or at least misleading.
answered Dec 31 '18 at 19:00
copper.hatcopper.hat
126k559160
For a bonus point: have you found a minimum or a maximum?
– LinAlg
Dec 31 '18 at 19:19
@LinAlg: I would need to know what the optimisation problem is first... Most Lagrangians I have encountered are scalar valued :-).
– copper.hat
Dec 31 '18 at 19:23
A Google search on "the variance (4.7) can be maximized" reveals a background of principal component analysis. $C$ is a covariance matrix, and the constraint is $E^TE = 1$. I agree that minimizing a matrix valued function is odd at least.
– LinAlg
Dec 31 '18 at 19:29
add a comment |
I am assuming that $C$ is symmetric.
Let $phi(E) = E^T CE -lambda E^T E$, then
$phi(E+H) = phi(E)+H^TCE+ E^TCH -lambda H^T E-lambda E^T H +r(H)$, where
$|r(H)| le K |H|^2$.
Then $D phi(E)H = H^T(CE - lambda E) + ( CE - lambda E)^T H$.
Suppose $D phi(E)(H) = 0$ for all $H$, then choose $H=CE - lambda E$ to get
$2 (CE - lambda E)^T (CE - lambda E) = 0$ from which we get
$CE - lambda E = 0$ (since $A=0$ iff $A^TA = 0$).
I believe the derivation given in the question is incorrect, or at least misleading.
answered Dec 31 '18 at 19:00
copper.hatcopper.hat
126k559160
For a bonus point: have you found a minimum or a maximum?
– LinAlg
Dec 31 '18 at 19:19
@LinAlg: I would need to know what the optimisation problem is first... Most Lagrangians I have encountered are scalar valued :-).
– copper.hat
Dec 31 '18 at 19:23
A Google search on "the variance (4.7) can be maximized" reveals a background of principal component analysis. $C$ is a covariance matrix, and the constraint is $E^TE = 1$. I agree that minimizing a matrix valued function is odd at least.
– LinAlg
Dec 31 '18 at 19:29
add a comment |
I am assuming that $C$ is symmetric.
Let $phi(E) = E^T CE -lambda E^T E$, then
$phi(E+H) = phi(E)+H^TCE+ E^TCH -lambda H^T E-lambda E^T H +r(H)$, where
$|r(H)| le K |H|^2$.
Then $D phi(E)H = H^T(CE - lambda E) + ( CE - lambda E)^T H$.
Suppose $D phi(E)(H) = 0$ for all $H$, then choose $H=CE - lambda E$ to get
$2 (CE - lambda E)^T (CE - lambda E) = 0$ from which we get
$CE - lambda E = 0$ (since $A=0$ iff $A^TA = 0$).
I believe the derivation given in the question is incorrect, or at least misleading.
answered Dec 31 '18 at 19:00
copper.hatcopper.hat
126k559160
I am assuming that $C$ is symmetric.
Let $phi(E) = E^T CE -lambda E^T E$, then
$phi(E+H) = phi(E)+H^TCE+ E^TCH -lambda H^T E-lambda E^T H +r(H)$, where
$|r(H)| le K |H|^2$.
Then $D phi(E)H = H^T(CE - lambda E) + ( CE - lambda E)^T H$.
Suppose $D phi(E)(H) = 0$ for all $H$, then choose $H=CE - lambda E$ to get
$2 (CE - lambda E)^T (CE - lambda E) = 0$ from which we get
$CE - lambda E = 0$ (since $A=0$ iff $A^TA = 0$).
I believe the derivation given in the question is incorrect, or at least misleading.
answered Dec 31 '18 at 19:00
copper.hatcopper.hat
126k559160
answered Dec 31 '18 at 19:00
copper.hatcopper.hat
126k559160
answered Dec 31 '18 at 19:00
copper.hatcopper.hat
126k559160
answered Dec 31 '18 at 19:00
copper.hatcopper.hat
126k559160
126k559160
For a bonus point: have you found a minimum or a maximum?
– LinAlg
Dec 31 '18 at 19:19
@LinAlg: I would need to know what the optimisation problem is first... Most Lagrangians I have encountered are scalar valued :-).
– copper.hat
Dec 31 '18 at 19:23
A Google search on "the variance (4.7) can be maximized" reveals a background of principal component analysis. $C$ is a covariance matrix, and the constraint is $E^TE = 1$. I agree that minimizing a matrix valued function is odd at least.
– LinAlg
Dec 31 '18 at 19:29
add a comment |
For a bonus point: have you found a minimum or a maximum?
– LinAlg
Dec 31 '18 at 19:19
@LinAlg: I would need to know what the optimisation problem is first... Most Lagrangians I have encountered are scalar valued :-).
– copper.hat
Dec 31 '18 at 19:23
A Google search on "the variance (4.7) can be maximized" reveals a background of principal component analysis. $C$ is a covariance matrix, and the constraint is $E^TE = 1$. I agree that minimizing a matrix valued function is odd at least.
– LinAlg
Dec 31 '18 at 19:29
For a bonus point: have you found a minimum or a maximum?
– LinAlg
Dec 31 '18 at 19:19
For a bonus point: have you found a minimum or a maximum?
– LinAlg
Dec 31 '18 at 19:19
@LinAlg: I would need to know what the optimisation problem is first... Most Lagrangians I have encountered are scalar valued :-).
– copper.hat
Dec 31 '18 at 19:23
@LinAlg: I would need to know what the optimisation problem is first... Most Lagrangians I have encountered are scalar valued :-).
– copper.hat
Dec 31 '18 at 19:23
A Google search on "the variance (4.7) can be maximized" reveals a background of principal component analysis. $C$ is a covariance matrix, and the constraint is $E^TE = 1$. I agree that minimizing a matrix valued function is odd at least.
– LinAlg
Dec 31 '18 at 19:29
A Google search on "the variance (4.7) can be maximized" reveals a background of principal component analysis. $C$ is a covariance matrix, and the constraint is $E^TE = 1$. I agree that minimizing a matrix valued function is odd at least.
– LinAlg
Dec 31 '18 at 19:29
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3057928%2foptimization-problem-derivative-of-matrix-confusion%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Are $C, E$ symmetric matrices?
– Cesareo
Dec 31 '18 at 18:27
have you computed the derivative of (4.10) with respect to $E_{ij}$?
– LinAlg
Dec 31 '18 at 18:27
@Cesareo E is a rotation matrix, so it is not symmetric in general. However, C is symmetric.
– Thomas Moore
Dec 31 '18 at 18:29
@LinAlg I tried this, but I do not know how to re-write the first term so that it is a scalar.
– Thomas Moore
Dec 31 '18 at 18:31
You should update the question to add relevant information on $C$, $E$, etc.
– copper.hat
Dec 31 '18 at 18:32