Mixing
Matrix differentials
$begingroup$
My goal is to minimise:
$$min_wleft( w'(a cdotmathrm{diag}(1/|w_1|,...,1/|w_n|+ X)w+b'w+x'wright)$$
where $Xinmathbb{R}^{ntimes n}$ semi-positive definite, $b,w,xinmathbb{R}^n$ and $a>0$. Also, $w_1,...,w_nneq 0$. I am having trouble with the following expression:
$$frac{partial}{partial w}left(w'(acdot mathrm{diag}(1/|w_1|,...,1/|w_n|+ X)wright)$$
I am aware that $w'Aw=2A$, if $A=A'$. In the above case $A$ is a function of $w$, so I am not sure how to go about this. Could anyone explain to me please, how to deal with this scenario?
linear-algebra derivatives optimization
asked Jan 6 at 3:10
WaqasWaqas
15913
$endgroup$
add a comment |
$begingroup$
My goal is to minimise:
$$min_wleft( w'(a cdotmathrm{diag}(1/|w_1|,...,1/|w_n|+ X)w+b'w+x'wright)$$
where $Xinmathbb{R}^{ntimes n}$ semi-positive definite, $b,w,xinmathbb{R}^n$ and $a>0$. Also, $w_1,...,w_nneq 0$. I am having trouble with the following expression:
$$frac{partial}{partial w}left(w'(acdot mathrm{diag}(1/|w_1|,...,1/|w_n|+ X)wright)$$
I am aware that $w'Aw=2A$, if $A=A'$. In the above case $A$ is a function of $w$, so I am not sure how to go about this. Could anyone explain to me please, how to deal with this scenario?
linear-algebra derivatives optimization
asked Jan 6 at 3:10
WaqasWaqas
15913
$endgroup$
add a comment |
$begingroup$
My goal is to minimise:
$$min_wleft( w'(a cdotmathrm{diag}(1/|w_1|,...,1/|w_n|+ X)w+b'w+x'wright)$$
where $Xinmathbb{R}^{ntimes n}$ semi-positive definite, $b,w,xinmathbb{R}^n$ and $a>0$. Also, $w_1,...,w_nneq 0$. I am having trouble with the following expression:
$$frac{partial}{partial w}left(w'(acdot mathrm{diag}(1/|w_1|,...,1/|w_n|+ X)wright)$$
I am aware that $w'Aw=2A$, if $A=A'$. In the above case $A$ is a function of $w$, so I am not sure how to go about this. Could anyone explain to me please, how to deal with this scenario?
linear-algebra derivatives optimization
asked Jan 6 at 3:10
WaqasWaqas
15913
$endgroup$
My goal is to minimise:
$$min_wleft( w'(a cdotmathrm{diag}(1/|w_1|,...,1/|w_n|+ X)w+b'w+x'wright)$$
where $Xinmathbb{R}^{ntimes n}$ semi-positive definite, $b,w,xinmathbb{R}^n$ and $a>0$. Also, $w_1,...,w_nneq 0$. I am having trouble with the following expression:
$$frac{partial}{partial w}left(w'(acdot mathrm{diag}(1/|w_1|,...,1/|w_n|+ X)wright)$$
I am aware that $w'Aw=2A$, if $A=A'$. In the above case $A$ is a function of $w$, so I am not sure how to go about this. Could anyone explain to me please, how to deal with this scenario?
linear-algebra derivatives optimization
linear-algebra derivatives optimization
asked Jan 6 at 3:10
WaqasWaqas
15913
asked Jan 6 at 3:10
WaqasWaqas
15913
asked Jan 6 at 3:10
WaqasWaqas
15913
asked Jan 6 at 3:10
WaqasWaqas
15913
asked Jan 6 at 3:10
WaqasWaqas
15913
15913
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
Let's use a colon to denote the trace/Frobenius product, i.e.
$$A:B = {rm Tr}(A^TB)$$
and define the vectors
$$eqalign{
c &= b + x,quad
s &= {rm sign}(w) cr
}$$
and the matrices
$$eqalign{
S &= {rm Diag}(s),quad
W &= {rm Diag}(w) cr
}$$
as well as the scalar $alpha = a$
The following relationships will prove useful.
$$eqalign{
SW &= WS = {rm abs}(W) cr
W1 &= w implies W^{-1}w=1 cr
S1 &= s implies S^{-1}s=1 cr
A!:!B &= B!:!A = A^T!:!B^T cr
A!:!BC &= AC^T!:!B = B^TA!:!C cr
cr
}$$
Write the cost function in terms of these new variables.
$$eqalign{
phi
&= alpha SW^{-1}:ww^T + X:ww^T + c:w cr
&= alpha SW^{-1}w:w + X:ww^T + c:w cr
&= alpha S1:w + X:ww^T + c:w cr
&= (alpha s+c):w + X:ww^T cr
}$$
Then find its differential and gradient.
$$eqalign{
dphi
&= (alpha s+c):dw + X:(dw,w^T+w,dw^T) cr
&= (alpha s + c + Xw + X^Tw):dw cr
frac{partialphi}{partial w}
&= alpha s + c + Xw + X^Tw cr
}$$
answered Jan 6 at 6:01
greggreg
7,7951821
$endgroup$
$begingroup$
Thanks. I thought it would be much simpler than this. So, can I say:$$0=as+c+(X+X')wimplies w=-(as+c)(X+X')^{-1}$$ assuming that $X$ is invertible.
$endgroup$
– Waqas
Jan 6 at 6:16
$begingroup$
Since the vector $s$ is a function of $w$, i.e. $,,s={rm sign}(w)$ you cannot obtain a closed-form solution. But you can setup an iterative solution $$w_{k+1} = (X^T+X)^{-1}(alpha,{rm sign}(w_k) + c)$$
$endgroup$
– greg
Jan 7 at 22:41
$begingroup$
To initialize the iteration, try $,w_0 = c,$ or perhaps $,w_0 = 1.$
$endgroup$
– greg
Jan 7 at 22:52
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3063431%2fmatrix-differentials%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Let's use a colon to denote the trace/Frobenius product, i.e.
$$A:B = {rm Tr}(A^TB)$$
and define the vectors
$$eqalign{
c &= b + x,quad
s &= {rm sign}(w) cr
}$$
and the matrices
$$eqalign{
S &= {rm Diag}(s),quad
W &= {rm Diag}(w) cr
}$$
as well as the scalar $alpha = a$
The following relationships will prove useful.
$$eqalign{
SW &= WS = {rm abs}(W) cr
W1 &= w implies W^{-1}w=1 cr
S1 &= s implies S^{-1}s=1 cr
A!:!B &= B!:!A = A^T!:!B^T cr
A!:!BC &= AC^T!:!B = B^TA!:!C cr
cr
}$$
Write the cost function in terms of these new variables.
$$eqalign{
phi
&= alpha SW^{-1}:ww^T + X:ww^T + c:w cr
&= alpha SW^{-1}w:w + X:ww^T + c:w cr
&= alpha S1:w + X:ww^T + c:w cr
&= (alpha s+c):w + X:ww^T cr
}$$
Then find its differential and gradient.
$$eqalign{
dphi
&= (alpha s+c):dw + X:(dw,w^T+w,dw^T) cr
&= (alpha s + c + Xw + X^Tw):dw cr
frac{partialphi}{partial w}
&= alpha s + c + Xw + X^Tw cr
}$$
answered Jan 6 at 6:01
greggreg
7,7951821
$endgroup$
$begingroup$
Thanks. I thought it would be much simpler than this. So, can I say:$$0=as+c+(X+X')wimplies w=-(as+c)(X+X')^{-1}$$ assuming that $X$ is invertible.
$endgroup$
– Waqas
Jan 6 at 6:16
$begingroup$
Since the vector $s$ is a function of $w$, i.e. $,,s={rm sign}(w)$ you cannot obtain a closed-form solution. But you can setup an iterative solution $$w_{k+1} = (X^T+X)^{-1}(alpha,{rm sign}(w_k) + c)$$
$endgroup$
– greg
Jan 7 at 22:41
$begingroup$
To initialize the iteration, try $,w_0 = c,$ or perhaps $,w_0 = 1.$
$endgroup$
– greg
Jan 7 at 22:52
add a comment |
$begingroup$
Let's use a colon to denote the trace/Frobenius product, i.e.
$$A:B = {rm Tr}(A^TB)$$
and define the vectors
$$eqalign{
c &= b + x,quad
s &= {rm sign}(w) cr
}$$
and the matrices
$$eqalign{
S &= {rm Diag}(s),quad
W &= {rm Diag}(w) cr
}$$
as well as the scalar $alpha = a$
The following relationships will prove useful.
$$eqalign{
SW &= WS = {rm abs}(W) cr
W1 &= w implies W^{-1}w=1 cr
S1 &= s implies S^{-1}s=1 cr
A!:!B &= B!:!A = A^T!:!B^T cr
A!:!BC &= AC^T!:!B = B^TA!:!C cr
cr
}$$
Write the cost function in terms of these new variables.
$$eqalign{
phi
&= alpha SW^{-1}:ww^T + X:ww^T + c:w cr
&= alpha SW^{-1}w:w + X:ww^T + c:w cr
&= alpha S1:w + X:ww^T + c:w cr
&= (alpha s+c):w + X:ww^T cr
}$$
Then find its differential and gradient.
$$eqalign{
dphi
&= (alpha s+c):dw + X:(dw,w^T+w,dw^T) cr
&= (alpha s + c + Xw + X^Tw):dw cr
frac{partialphi}{partial w}
&= alpha s + c + Xw + X^Tw cr
}$$
answered Jan 6 at 6:01
greggreg
7,7951821
$endgroup$
$begingroup$
Thanks. I thought it would be much simpler than this. So, can I say:$$0=as+c+(X+X')wimplies w=-(as+c)(X+X')^{-1}$$ assuming that $X$ is invertible.
$endgroup$
– Waqas
Jan 6 at 6:16
$begingroup$
Since the vector $s$ is a function of $w$, i.e. $,,s={rm sign}(w)$ you cannot obtain a closed-form solution. But you can setup an iterative solution $$w_{k+1} = (X^T+X)^{-1}(alpha,{rm sign}(w_k) + c)$$
$endgroup$
– greg
Jan 7 at 22:41
$begingroup$
To initialize the iteration, try $,w_0 = c,$ or perhaps $,w_0 = 1.$
$endgroup$
– greg
Jan 7 at 22:52
add a comment |
$begingroup$
Let's use a colon to denote the trace/Frobenius product, i.e.
$$A:B = {rm Tr}(A^TB)$$
and define the vectors
$$eqalign{
c &= b + x,quad
s &= {rm sign}(w) cr
}$$
and the matrices
$$eqalign{
S &= {rm Diag}(s),quad
W &= {rm Diag}(w) cr
}$$
as well as the scalar $alpha = a$
The following relationships will prove useful.
$$eqalign{
SW &= WS = {rm abs}(W) cr
W1 &= w implies W^{-1}w=1 cr
S1 &= s implies S^{-1}s=1 cr
A!:!B &= B!:!A = A^T!:!B^T cr
A!:!BC &= AC^T!:!B = B^TA!:!C cr
cr
}$$
Write the cost function in terms of these new variables.
$$eqalign{
phi
&= alpha SW^{-1}:ww^T + X:ww^T + c:w cr
&= alpha SW^{-1}w:w + X:ww^T + c:w cr
&= alpha S1:w + X:ww^T + c:w cr
&= (alpha s+c):w + X:ww^T cr
}$$
Then find its differential and gradient.
$$eqalign{
dphi
&= (alpha s+c):dw + X:(dw,w^T+w,dw^T) cr
&= (alpha s + c + Xw + X^Tw):dw cr
frac{partialphi}{partial w}
&= alpha s + c + Xw + X^Tw cr
}$$
answered Jan 6 at 6:01
greggreg
7,7951821
$endgroup$
Let's use a colon to denote the trace/Frobenius product, i.e.
$$A:B = {rm Tr}(A^TB)$$
and define the vectors
$$eqalign{
c &= b + x,quad
s &= {rm sign}(w) cr
}$$
and the matrices
$$eqalign{
S &= {rm Diag}(s),quad
W &= {rm Diag}(w) cr
}$$
as well as the scalar $alpha = a$
The following relationships will prove useful.
$$eqalign{
SW &= WS = {rm abs}(W) cr
W1 &= w implies W^{-1}w=1 cr
S1 &= s implies S^{-1}s=1 cr
A!:!B &= B!:!A = A^T!:!B^T cr
A!:!BC &= AC^T!:!B = B^TA!:!C cr
cr
}$$
Write the cost function in terms of these new variables.
$$eqalign{
phi
&= alpha SW^{-1}:ww^T + X:ww^T + c:w cr
&= alpha SW^{-1}w:w + X:ww^T + c:w cr
&= alpha S1:w + X:ww^T + c:w cr
&= (alpha s+c):w + X:ww^T cr
}$$
Then find its differential and gradient.
$$eqalign{
dphi
&= (alpha s+c):dw + X:(dw,w^T+w,dw^T) cr
&= (alpha s + c + Xw + X^Tw):dw cr
frac{partialphi}{partial w}
&= alpha s + c + Xw + X^Tw cr
}$$
answered Jan 6 at 6:01
greggreg
7,7951821
edited Jan 6 at 6:06
answered Jan 6 at 6:01
greggreg
7,7951821
answered Jan 6 at 6:01
greggreg
7,7951821
answered Jan 6 at 6:01
greggreg
7,7951821
7,7951821
$begingroup$
Thanks. I thought it would be much simpler than this. So, can I say:$$0=as+c+(X+X')wimplies w=-(as+c)(X+X')^{-1}$$ assuming that $X$ is invertible.
$endgroup$
– Waqas
Jan 6 at 6:16
$begingroup$
Since the vector $s$ is a function of $w$, i.e. $,,s={rm sign}(w)$ you cannot obtain a closed-form solution. But you can setup an iterative solution $$w_{k+1} = (X^T+X)^{-1}(alpha,{rm sign}(w_k) + c)$$
$endgroup$
– greg
Jan 7 at 22:41
$begingroup$
To initialize the iteration, try $,w_0 = c,$ or perhaps $,w_0 = 1.$
$endgroup$
– greg
Jan 7 at 22:52
add a comment |
$begingroup$
Thanks. I thought it would be much simpler than this. So, can I say:$$0=as+c+(X+X')wimplies w=-(as+c)(X+X')^{-1}$$ assuming that $X$ is invertible.
$endgroup$
– Waqas
Jan 6 at 6:16
$begingroup$
Since the vector $s$ is a function of $w$, i.e. $,,s={rm sign}(w)$ you cannot obtain a closed-form solution. But you can setup an iterative solution $$w_{k+1} = (X^T+X)^{-1}(alpha,{rm sign}(w_k) + c)$$
$endgroup$
– greg
Jan 7 at 22:41
$begingroup$
To initialize the iteration, try $,w_0 = c,$ or perhaps $,w_0 = 1.$
$endgroup$
– greg
Jan 7 at 22:52
$begingroup$
Thanks. I thought it would be much simpler than this. So, can I say:$$0=as+c+(X+X')wimplies w=-(as+c)(X+X')^{-1}$$ assuming that $X$ is invertible.
$endgroup$
– Waqas
Jan 6 at 6:16
$begingroup$
Thanks. I thought it would be much simpler than this. So, can I say:$$0=as+c+(X+X')wimplies w=-(as+c)(X+X')^{-1}$$ assuming that $X$ is invertible.
$endgroup$
– Waqas
Jan 6 at 6:16
$begingroup$
Since the vector $s$ is a function of $w$, i.e. $,,s={rm sign}(w)$ you cannot obtain a closed-form solution. But you can setup an iterative solution $$w_{k+1} = (X^T+X)^{-1}(alpha,{rm sign}(w_k) + c)$$
$endgroup$
– greg
Jan 7 at 22:41
$begingroup$
Since the vector $s$ is a function of $w$, i.e. $,,s={rm sign}(w)$ you cannot obtain a closed-form solution. But you can setup an iterative solution $$w_{k+1} = (X^T+X)^{-1}(alpha,{rm sign}(w_k) + c)$$
$endgroup$
– greg
Jan 7 at 22:41
$begingroup$
To initialize the iteration, try $,w_0 = c,$ or perhaps $,w_0 = 1.$
$endgroup$
– greg
Jan 7 at 22:52
$begingroup$
To initialize the iteration, try $,w_0 = c,$ or perhaps $,w_0 = 1.$
$endgroup$
– greg
Jan 7 at 22:52
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3063431%2fmatrix-differentials%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
