Mixing
Characterize the critical points of $Vert Ax - b Vert^{2}$
$begingroup$
Let $A$ a $m times n$ matrix, $b$ a $m times 1$ matrix and $x$ a $n times 1$ matrix. Consider $f: mathbb{R}^n to mathbb{R}$ defined by
$$f(x) = Vert Ax - b Vert^2.$$
Determine a condition that characterizes the critical points of $f$.
My attempt. Note that
$$Vert Ax - b Vert^{2} = (Ax - b)^{T}(Ax - b) = (x^{T}A^{T} - b^{T})(Ax - b) = x^{T}A^{T}Ax - x^{T}A^{T}b - b^{T}Ax + b^{T}b.$$
Thus,
$$frac{partial}{partial x_{i}}f(x) = 2A^{T}Ax_{i} - A^{T}b - color{red}{b^{T}A}.$$
But I know that $nabla f(x) = 2A^{T}Ax - 2A^{T}b$, so "$b^{T}A$ should be $A^{T}b$". How do I correct this?
Assuming that $nabla f(x) = 2A^{T}Ax - 2A^{T}b = 2A^{T}(Ax - b)$, the critical points of $f$ are the solutions of $Ax - b$. I dont understand what it means to characterize the critical points, but I suppose it's about maximum and minimum.
Let $H: mathbb{R}^n to mathbb{R}$ a quadratic form given by $Hv^{2} = sum h_{ij}alpha_{i}alpha_{j}$. The quadric form of $H$ is positive when $Hv^{2} > 0$ for all $v neq 0$ in $mathbb{R}^n$ and is negative when $Hv^{2} < 0$ for all $v neq 0$ in $mathbb{R}^n$, otherwise we say that $Hv^{2}$ is undefined.
Theorem. Let $f: U to mathbb{R}$ a $C^{2}$ function, $p in U$ a critical point of $f$ and $H$ the Hessian quadratic form of $f$ in $p$. Then
(i) If $H$ is positive, $p$ is a local minimum point (non-degenerated).
(ii) If $H$ is negative, $p$ is a maximum local point (non-degenerated).
(iii) If $H$ is undefined, $p$ is not a local maximum or minimum point.
The Hessian of $f$ is $nabla^{2} f(x) = 2A^{T}A = 2Vert A Vert^2$. But, the Hessian is independent of the point and $2Vert A Vert^2$ is non-negative. So the quadratic form is always non-negative? This is the first one I try about Hessian and critical points so, I dont have much practice.
I appreciate any help!
real-analysis derivatives hessian-matrix
asked Jan 20 at 16:49
Lucas CorrêaLucas Corrêa
1,6181321
$endgroup$
add a comment |
$begingroup$
Let $A$ a $m times n$ matrix, $b$ a $m times 1$ matrix and $x$ a $n times 1$ matrix. Consider $f: mathbb{R}^n to mathbb{R}$ defined by
$$f(x) = Vert Ax - b Vert^2.$$
Determine a condition that characterizes the critical points of $f$.
My attempt. Note that
$$Vert Ax - b Vert^{2} = (Ax - b)^{T}(Ax - b) = (x^{T}A^{T} - b^{T})(Ax - b) = x^{T}A^{T}Ax - x^{T}A^{T}b - b^{T}Ax + b^{T}b.$$
Thus,
$$frac{partial}{partial x_{i}}f(x) = 2A^{T}Ax_{i} - A^{T}b - color{red}{b^{T}A}.$$
But I know that $nabla f(x) = 2A^{T}Ax - 2A^{T}b$, so "$b^{T}A$ should be $A^{T}b$". How do I correct this?
Assuming that $nabla f(x) = 2A^{T}Ax - 2A^{T}b = 2A^{T}(Ax - b)$, the critical points of $f$ are the solutions of $Ax - b$. I dont understand what it means to characterize the critical points, but I suppose it's about maximum and minimum.
Let $H: mathbb{R}^n to mathbb{R}$ a quadratic form given by $Hv^{2} = sum h_{ij}alpha_{i}alpha_{j}$. The quadric form of $H$ is positive when $Hv^{2} > 0$ for all $v neq 0$ in $mathbb{R}^n$ and is negative when $Hv^{2} < 0$ for all $v neq 0$ in $mathbb{R}^n$, otherwise we say that $Hv^{2}$ is undefined.
Theorem. Let $f: U to mathbb{R}$ a $C^{2}$ function, $p in U$ a critical point of $f$ and $H$ the Hessian quadratic form of $f$ in $p$. Then
(i) If $H$ is positive, $p$ is a local minimum point (non-degenerated).
(ii) If $H$ is negative, $p$ is a maximum local point (non-degenerated).
(iii) If $H$ is undefined, $p$ is not a local maximum or minimum point.
The Hessian of $f$ is $nabla^{2} f(x) = 2A^{T}A = 2Vert A Vert^2$. But, the Hessian is independent of the point and $2Vert A Vert^2$ is non-negative. So the quadratic form is always non-negative? This is the first one I try about Hessian and critical points so, I dont have much practice.
I appreciate any help!
real-analysis derivatives hessian-matrix
asked Jan 20 at 16:49
Lucas CorrêaLucas Corrêa
1,6181321
$endgroup$
add a comment |
$begingroup$
Let $A$ a $m times n$ matrix, $b$ a $m times 1$ matrix and $x$ a $n times 1$ matrix. Consider $f: mathbb{R}^n to mathbb{R}$ defined by
$$f(x) = Vert Ax - b Vert^2.$$
Determine a condition that characterizes the critical points of $f$.
My attempt. Note that
$$Vert Ax - b Vert^{2} = (Ax - b)^{T}(Ax - b) = (x^{T}A^{T} - b^{T})(Ax - b) = x^{T}A^{T}Ax - x^{T}A^{T}b - b^{T}Ax + b^{T}b.$$
Thus,
$$frac{partial}{partial x_{i}}f(x) = 2A^{T}Ax_{i} - A^{T}b - color{red}{b^{T}A}.$$
But I know that $nabla f(x) = 2A^{T}Ax - 2A^{T}b$, so "$b^{T}A$ should be $A^{T}b$". How do I correct this?
Assuming that $nabla f(x) = 2A^{T}Ax - 2A^{T}b = 2A^{T}(Ax - b)$, the critical points of $f$ are the solutions of $Ax - b$. I dont understand what it means to characterize the critical points, but I suppose it's about maximum and minimum.
Let $H: mathbb{R}^n to mathbb{R}$ a quadratic form given by $Hv^{2} = sum h_{ij}alpha_{i}alpha_{j}$. The quadric form of $H$ is positive when $Hv^{2} > 0$ for all $v neq 0$ in $mathbb{R}^n$ and is negative when $Hv^{2} < 0$ for all $v neq 0$ in $mathbb{R}^n$, otherwise we say that $Hv^{2}$ is undefined.
Theorem. Let $f: U to mathbb{R}$ a $C^{2}$ function, $p in U$ a critical point of $f$ and $H$ the Hessian quadratic form of $f$ in $p$. Then
(i) If $H$ is positive, $p$ is a local minimum point (non-degenerated).
(ii) If $H$ is negative, $p$ is a maximum local point (non-degenerated).
(iii) If $H$ is undefined, $p$ is not a local maximum or minimum point.
The Hessian of $f$ is $nabla^{2} f(x) = 2A^{T}A = 2Vert A Vert^2$. But, the Hessian is independent of the point and $2Vert A Vert^2$ is non-negative. So the quadratic form is always non-negative? This is the first one I try about Hessian and critical points so, I dont have much practice.
I appreciate any help!
real-analysis derivatives hessian-matrix
asked Jan 20 at 16:49
Lucas CorrêaLucas Corrêa
1,6181321
$endgroup$
Let $A$ a $m times n$ matrix, $b$ a $m times 1$ matrix and $x$ a $n times 1$ matrix. Consider $f: mathbb{R}^n to mathbb{R}$ defined by
$$f(x) = Vert Ax - b Vert^2.$$
Determine a condition that characterizes the critical points of $f$.
My attempt. Note that
$$Vert Ax - b Vert^{2} = (Ax - b)^{T}(Ax - b) = (x^{T}A^{T} - b^{T})(Ax - b) = x^{T}A^{T}Ax - x^{T}A^{T}b - b^{T}Ax + b^{T}b.$$
Thus,
$$frac{partial}{partial x_{i}}f(x) = 2A^{T}Ax_{i} - A^{T}b - color{red}{b^{T}A}.$$
But I know that $nabla f(x) = 2A^{T}Ax - 2A^{T}b$, so "$b^{T}A$ should be $A^{T}b$". How do I correct this?
Assuming that $nabla f(x) = 2A^{T}Ax - 2A^{T}b = 2A^{T}(Ax - b)$, the critical points of $f$ are the solutions of $Ax - b$. I dont understand what it means to characterize the critical points, but I suppose it's about maximum and minimum.
Let $H: mathbb{R}^n to mathbb{R}$ a quadratic form given by $Hv^{2} = sum h_{ij}alpha_{i}alpha_{j}$. The quadric form of $H$ is positive when $Hv^{2} > 0$ for all $v neq 0$ in $mathbb{R}^n$ and is negative when $Hv^{2} < 0$ for all $v neq 0$ in $mathbb{R}^n$, otherwise we say that $Hv^{2}$ is undefined.
Theorem. Let $f: U to mathbb{R}$ a $C^{2}$ function, $p in U$ a critical point of $f$ and $H$ the Hessian quadratic form of $f$ in $p$. Then
(i) If $H$ is positive, $p$ is a local minimum point (non-degenerated).
(ii) If $H$ is negative, $p$ is a maximum local point (non-degenerated).
(iii) If $H$ is undefined, $p$ is not a local maximum or minimum point.
The Hessian of $f$ is $nabla^{2} f(x) = 2A^{T}A = 2Vert A Vert^2$. But, the Hessian is independent of the point and $2Vert A Vert^2$ is non-negative. So the quadratic form is always non-negative? This is the first one I try about Hessian and critical points so, I dont have much practice.
I appreciate any help!
real-analysis derivatives hessian-matrix
real-analysis derivatives hessian-matrix
asked Jan 20 at 16:49
Lucas CorrêaLucas Corrêa
1,6181321
asked Jan 20 at 16:49
Lucas CorrêaLucas Corrêa
1,6181321
asked Jan 20 at 16:49
Lucas CorrêaLucas Corrêa
1,6181321
asked Jan 20 at 16:49
Lucas CorrêaLucas Corrêa
1,6181321
asked Jan 20 at 16:49
Lucas CorrêaLucas Corrêa
1,6181321
1,6181321
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
The partial derivative should be
$$frac{partial}{partial x_i} f(x) = 2 A^top A x_i - (A^top b)_i - (A^top b)_i$$
where $(A^top b)_i$ denotes the $i$th component of the vector $A^top b$.
"Determine a condition that characterizes the critical points" is simply asking for the condition $A^top (Ax - b) = 0$. That's it! :) [That is, if someone asked you "how can you tell me if this $x$ is a critical point?" you can check whether $A^top (Ax - b) = 0$ holds or not.]
More importantly, the above condition is not equivalent to $Ax-b=0$. If $Ax-b=0$ it is then true that $A^top (Ax-b) = 0$, but the converse may not hold, in particular if $A^top$ has a nontrivial nullspace.
I would not write $A^top A = |A|^2$ since $A$ is a matrix, not a vector.
But indeed the Hessian is $H = 2 A^top A$ which is a positive semi-definite matrix, i.e. it satisfies $v^top H v ge 0$ for any vector $v$. Thus any critical point is a local minimum.
answered Jan 20 at 17:09
angryavianangryavian
41.9k23381
$endgroup$
$begingroup$
Its a nice answer! Can you explain a little more the partial derivative, precisely the last term?
$endgroup$
– Lucas Corrêa
Jan 20 at 17:25
1
$begingroup$
@LucasCorrêa It may be clearer for you to simply write $x^top A^top b = b^top A x = sum_j sum_k A_{jk} b_j x_k$ and take the partial derivative.
$endgroup$
– angryavian
Jan 20 at 17:51
$begingroup$
I got it! Thank you!
$endgroup$
– Lucas Corrêa
Jan 20 at 18:36
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3080810%2fcharacterize-the-critical-points-of-vert-ax-b-vert2%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The partial derivative should be
$$frac{partial}{partial x_i} f(x) = 2 A^top A x_i - (A^top b)_i - (A^top b)_i$$
where $(A^top b)_i$ denotes the $i$th component of the vector $A^top b$.
"Determine a condition that characterizes the critical points" is simply asking for the condition $A^top (Ax - b) = 0$. That's it! :) [That is, if someone asked you "how can you tell me if this $x$ is a critical point?" you can check whether $A^top (Ax - b) = 0$ holds or not.]
More importantly, the above condition is not equivalent to $Ax-b=0$. If $Ax-b=0$ it is then true that $A^top (Ax-b) = 0$, but the converse may not hold, in particular if $A^top$ has a nontrivial nullspace.
I would not write $A^top A = |A|^2$ since $A$ is a matrix, not a vector.
But indeed the Hessian is $H = 2 A^top A$ which is a positive semi-definite matrix, i.e. it satisfies $v^top H v ge 0$ for any vector $v$. Thus any critical point is a local minimum.
answered Jan 20 at 17:09
angryavianangryavian
41.9k23381
$endgroup$
$begingroup$
Its a nice answer! Can you explain a little more the partial derivative, precisely the last term?
$endgroup$
– Lucas Corrêa
Jan 20 at 17:25
1
$begingroup$
@LucasCorrêa It may be clearer for you to simply write $x^top A^top b = b^top A x = sum_j sum_k A_{jk} b_j x_k$ and take the partial derivative.
$endgroup$
– angryavian
Jan 20 at 17:51
$begingroup$
I got it! Thank you!
$endgroup$
– Lucas Corrêa
Jan 20 at 18:36
add a comment |
$begingroup$
The partial derivative should be
$$frac{partial}{partial x_i} f(x) = 2 A^top A x_i - (A^top b)_i - (A^top b)_i$$
where $(A^top b)_i$ denotes the $i$th component of the vector $A^top b$.
"Determine a condition that characterizes the critical points" is simply asking for the condition $A^top (Ax - b) = 0$. That's it! :) [That is, if someone asked you "how can you tell me if this $x$ is a critical point?" you can check whether $A^top (Ax - b) = 0$ holds or not.]
More importantly, the above condition is not equivalent to $Ax-b=0$. If $Ax-b=0$ it is then true that $A^top (Ax-b) = 0$, but the converse may not hold, in particular if $A^top$ has a nontrivial nullspace.
I would not write $A^top A = |A|^2$ since $A$ is a matrix, not a vector.
But indeed the Hessian is $H = 2 A^top A$ which is a positive semi-definite matrix, i.e. it satisfies $v^top H v ge 0$ for any vector $v$. Thus any critical point is a local minimum.
answered Jan 20 at 17:09
angryavianangryavian
41.9k23381
$endgroup$
$begingroup$
Its a nice answer! Can you explain a little more the partial derivative, precisely the last term?
$endgroup$
– Lucas Corrêa
Jan 20 at 17:25
1
$begingroup$
@LucasCorrêa It may be clearer for you to simply write $x^top A^top b = b^top A x = sum_j sum_k A_{jk} b_j x_k$ and take the partial derivative.
$endgroup$
– angryavian
Jan 20 at 17:51
$begingroup$
I got it! Thank you!
$endgroup$
– Lucas Corrêa
Jan 20 at 18:36
add a comment |
$begingroup$
The partial derivative should be
$$frac{partial}{partial x_i} f(x) = 2 A^top A x_i - (A^top b)_i - (A^top b)_i$$
where $(A^top b)_i$ denotes the $i$th component of the vector $A^top b$.
"Determine a condition that characterizes the critical points" is simply asking for the condition $A^top (Ax - b) = 0$. That's it! :) [That is, if someone asked you "how can you tell me if this $x$ is a critical point?" you can check whether $A^top (Ax - b) = 0$ holds or not.]
More importantly, the above condition is not equivalent to $Ax-b=0$. If $Ax-b=0$ it is then true that $A^top (Ax-b) = 0$, but the converse may not hold, in particular if $A^top$ has a nontrivial nullspace.
I would not write $A^top A = |A|^2$ since $A$ is a matrix, not a vector.
But indeed the Hessian is $H = 2 A^top A$ which is a positive semi-definite matrix, i.e. it satisfies $v^top H v ge 0$ for any vector $v$. Thus any critical point is a local minimum.
answered Jan 20 at 17:09
angryavianangryavian
41.9k23381
$endgroup$
The partial derivative should be
$$frac{partial}{partial x_i} f(x) = 2 A^top A x_i - (A^top b)_i - (A^top b)_i$$
where $(A^top b)_i$ denotes the $i$th component of the vector $A^top b$.
"Determine a condition that characterizes the critical points" is simply asking for the condition $A^top (Ax - b) = 0$. That's it! :) [That is, if someone asked you "how can you tell me if this $x$ is a critical point?" you can check whether $A^top (Ax - b) = 0$ holds or not.]
More importantly, the above condition is not equivalent to $Ax-b=0$. If $Ax-b=0$ it is then true that $A^top (Ax-b) = 0$, but the converse may not hold, in particular if $A^top$ has a nontrivial nullspace.
I would not write $A^top A = |A|^2$ since $A$ is a matrix, not a vector.
But indeed the Hessian is $H = 2 A^top A$ which is a positive semi-definite matrix, i.e. it satisfies $v^top H v ge 0$ for any vector $v$. Thus any critical point is a local minimum.
answered Jan 20 at 17:09
angryavianangryavian
41.9k23381
answered Jan 20 at 17:09
angryavianangryavian
41.9k23381
answered Jan 20 at 17:09
angryavianangryavian
41.9k23381
answered Jan 20 at 17:09
angryavianangryavian
41.9k23381
41.9k23381
$begingroup$
Its a nice answer! Can you explain a little more the partial derivative, precisely the last term?
$endgroup$
– Lucas Corrêa
Jan 20 at 17:25
1
$begingroup$
@LucasCorrêa It may be clearer for you to simply write $x^top A^top b = b^top A x = sum_j sum_k A_{jk} b_j x_k$ and take the partial derivative.
$endgroup$
– angryavian
Jan 20 at 17:51
$begingroup$
I got it! Thank you!
$endgroup$
– Lucas Corrêa
Jan 20 at 18:36
add a comment |
$begingroup$
Its a nice answer! Can you explain a little more the partial derivative, precisely the last term?
$endgroup$
– Lucas Corrêa
Jan 20 at 17:25
1
$begingroup$
@LucasCorrêa It may be clearer for you to simply write $x^top A^top b = b^top A x = sum_j sum_k A_{jk} b_j x_k$ and take the partial derivative.
$endgroup$
– angryavian
Jan 20 at 17:51
$begingroup$
I got it! Thank you!
$endgroup$
– Lucas Corrêa
Jan 20 at 18:36
$begingroup$
Its a nice answer! Can you explain a little more the partial derivative, precisely the last term?
$endgroup$
– Lucas Corrêa
Jan 20 at 17:25
$begingroup$
Its a nice answer! Can you explain a little more the partial derivative, precisely the last term?
$endgroup$
– Lucas Corrêa
Jan 20 at 17:25
1
1
$begingroup$
@LucasCorrêa It may be clearer for you to simply write $x^top A^top b = b^top A x = sum_j sum_k A_{jk} b_j x_k$ and take the partial derivative.
$endgroup$
– angryavian
Jan 20 at 17:51
$begingroup$
@LucasCorrêa It may be clearer for you to simply write $x^top A^top b = b^top A x = sum_j sum_k A_{jk} b_j x_k$ and take the partial derivative.
$endgroup$
– angryavian
Jan 20 at 17:51
$begingroup$
I got it! Thank you!
$endgroup$
– Lucas Corrêa
Jan 20 at 18:36
$begingroup$
I got it! Thank you!
$endgroup$
– Lucas Corrêa
Jan 20 at 18:36
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3080810%2fcharacterize-the-critical-points-of-vert-ax-b-vert2%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
