Mixing
What is the principle components matrix in PCA with SVD.
$begingroup$
Doing PCA on a matrix using SVD yields a result of three matrices, expressed as:
$$
M = U Sigma V^T
$$
where $M$ is our initial data with zero mean.
If we want to make a plot of the two principle components we project the data onto principal component space.
$$
Z = M * V
$$
and then use the two first columns of Z for our plot. Maybe I have already answered my own question, but I am struggling to understand if $Z$ is what would be called the Principle Component matrix, and if not, how do we find that?
Also, I am not sure what the operation $M*V$ does to the data. As I understand it, $V$ is an expression of the general trends of each of the attributes in the data set. By calculating the dot product between our data $M$ and the trends $V$ of the data, we end up with a matrix (PC matrix?) that captures the original data in a structured manner which allows for dimensionality reduction.
Are my assumptions correct, or have I misread the theory?
linear-algebra
asked Sep 13 '12 at 19:57
Paul HunterPaul Hunter
1113
$endgroup$
add a comment |
$begingroup$
Doing PCA on a matrix using SVD yields a result of three matrices, expressed as:
$$
M = U Sigma V^T
$$
where $M$ is our initial data with zero mean.
If we want to make a plot of the two principle components we project the data onto principal component space.
$$
Z = M * V
$$
and then use the two first columns of Z for our plot. Maybe I have already answered my own question, but I am struggling to understand if $Z$ is what would be called the Principle Component matrix, and if not, how do we find that?
Also, I am not sure what the operation $M*V$ does to the data. As I understand it, $V$ is an expression of the general trends of each of the attributes in the data set. By calculating the dot product between our data $M$ and the trends $V$ of the data, we end up with a matrix (PC matrix?) that captures the original data in a structured manner which allows for dimensionality reduction.
Are my assumptions correct, or have I misread the theory?
linear-algebra
asked Sep 13 '12 at 19:57
Paul HunterPaul Hunter
1113
$endgroup$
2
$begingroup$
question is not very clear. What do you wanna learn exactly?
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 20:04
$begingroup$
I apologize for the vagueness of my question, which probably reflects my vague understanding of the subject matter. I specifically wish to learn if what I calculate to be matrix Z contains the principal components of my PCA analysis. I.e. would column one of Z be the first principal component of my data matrix M?
$endgroup$
– Paul Hunter
Sep 13 '12 at 20:10
1
$begingroup$
You are right. Basically you obtain your principle components by multiplying it with the eigen matrices that you obtained after $SVD$ The most significant components after multiplication are called principle components.
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 21:37
add a comment |
$begingroup$
Doing PCA on a matrix using SVD yields a result of three matrices, expressed as:
$$
M = U Sigma V^T
$$
where $M$ is our initial data with zero mean.
If we want to make a plot of the two principle components we project the data onto principal component space.
$$
Z = M * V
$$
and then use the two first columns of Z for our plot. Maybe I have already answered my own question, but I am struggling to understand if $Z$ is what would be called the Principle Component matrix, and if not, how do we find that?
Also, I am not sure what the operation $M*V$ does to the data. As I understand it, $V$ is an expression of the general trends of each of the attributes in the data set. By calculating the dot product between our data $M$ and the trends $V$ of the data, we end up with a matrix (PC matrix?) that captures the original data in a structured manner which allows for dimensionality reduction.
Are my assumptions correct, or have I misread the theory?
linear-algebra
asked Sep 13 '12 at 19:57
Paul HunterPaul Hunter
1113
$endgroup$
Doing PCA on a matrix using SVD yields a result of three matrices, expressed as:
$$
M = U Sigma V^T
$$
where $M$ is our initial data with zero mean.
If we want to make a plot of the two principle components we project the data onto principal component space.
$$
Z = M * V
$$
and then use the two first columns of Z for our plot. Maybe I have already answered my own question, but I am struggling to understand if $Z$ is what would be called the Principle Component matrix, and if not, how do we find that?
Also, I am not sure what the operation $M*V$ does to the data. As I understand it, $V$ is an expression of the general trends of each of the attributes in the data set. By calculating the dot product between our data $M$ and the trends $V$ of the data, we end up with a matrix (PC matrix?) that captures the original data in a structured manner which allows for dimensionality reduction.
Are my assumptions correct, or have I misread the theory?
linear-algebra
linear-algebra
asked Sep 13 '12 at 19:57
Paul HunterPaul Hunter
1113
asked Sep 13 '12 at 19:57
Paul HunterPaul Hunter
1113
edited Sep 13 '12 at 20:03
Paul Hunter
asked Sep 13 '12 at 19:57
Paul HunterPaul Hunter
1113
asked Sep 13 '12 at 19:57
Paul HunterPaul Hunter
1113
asked Sep 13 '12 at 19:57
Paul HunterPaul Hunter
1113
1113
2
$begingroup$
question is not very clear. What do you wanna learn exactly?
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 20:04
$begingroup$
I apologize for the vagueness of my question, which probably reflects my vague understanding of the subject matter. I specifically wish to learn if what I calculate to be matrix Z contains the principal components of my PCA analysis. I.e. would column one of Z be the first principal component of my data matrix M?
$endgroup$
– Paul Hunter
Sep 13 '12 at 20:10
1
$begingroup$
You are right. Basically you obtain your principle components by multiplying it with the eigen matrices that you obtained after $SVD$ The most significant components after multiplication are called principle components.
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 21:37
add a comment |
2
$begingroup$
question is not very clear. What do you wanna learn exactly?
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 20:04
$begingroup$
I apologize for the vagueness of my question, which probably reflects my vague understanding of the subject matter. I specifically wish to learn if what I calculate to be matrix Z contains the principal components of my PCA analysis. I.e. would column one of Z be the first principal component of my data matrix M?
$endgroup$
– Paul Hunter
Sep 13 '12 at 20:10
1
$begingroup$
You are right. Basically you obtain your principle components by multiplying it with the eigen matrices that you obtained after $SVD$ The most significant components after multiplication are called principle components.
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 21:37
2
2
$begingroup$
question is not very clear. What do you wanna learn exactly?
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 20:04
$begingroup$
question is not very clear. What do you wanna learn exactly?
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 20:04
$begingroup$
I apologize for the vagueness of my question, which probably reflects my vague understanding of the subject matter. I specifically wish to learn if what I calculate to be matrix Z contains the principal components of my PCA analysis. I.e. would column one of Z be the first principal component of my data matrix M?
$endgroup$
– Paul Hunter
Sep 13 '12 at 20:10
$begingroup$
I apologize for the vagueness of my question, which probably reflects my vague understanding of the subject matter. I specifically wish to learn if what I calculate to be matrix Z contains the principal components of my PCA analysis. I.e. would column one of Z be the first principal component of my data matrix M?
$endgroup$
– Paul Hunter
Sep 13 '12 at 20:10
1
1
$begingroup$
You are right. Basically you obtain your principle components by multiplying it with the eigen matrices that you obtained after $SVD$ The most significant components after multiplication are called principle components.
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 21:37
$begingroup$
You are right. Basically you obtain your principle components by multiplying it with the eigen matrices that you obtained after $SVD$ The most significant components after multiplication are called principle components.
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 21:37
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
Ok - Ill give it a shot. Let me start from the top and try to recap PCA, and then show the connection to SVD.
Recall: For PCA,
We begin with a centered $M$ (dim = $(n,d)$) of data. For this data, we compute the sample covariance : $S = frac{1}{n-1}M^TM$ where $n$ is the number of data points.
For this covariance matrix we find the eigenvectors and eigenvalues. Corresponding to the largest eigenvalues we select $l$ eigenvectors. Lets call the matrix consisting of these eigenvectors $W$.
$W$ will have dimensions $d times l$ . Then we can write $Z = MW$ and we understand that each row of Z is a lower dimensional embedding of $m_i$ a row of $M$.
OK now suppose we can write $M = U S V^T$. Then we notice that :
begin{align}
M^TM &= VS^TU^TUSV^T\
&= V(S^TS)V^T text{(since U is orthonormal)} \
&= VDV^T
end{align}
Where $D = S^2$ is a diagonal matrix containing the squares of singular values.
Thus we have that $(M^TM)V = VD$ since $V$ is also orthonormal. Aha! So we can see that the columns of V are the eigenvectors of $M^TM$ and $D$ contains the eigenvalues. This $V$ is precisely what we called $W$ above.
Hope that clears things up a bit. Sorry for the delay :)
answered May 14 '15 at 2:15
faith_in_factsfaith_in_facts
1,47221530
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f195385%2fwhat-is-the-principle-components-matrix-in-pca-with-svd%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Ok - Ill give it a shot. Let me start from the top and try to recap PCA, and then show the connection to SVD.
Recall: For PCA,
We begin with a centered $M$ (dim = $(n,d)$) of data. For this data, we compute the sample covariance : $S = frac{1}{n-1}M^TM$ where $n$ is the number of data points.
For this covariance matrix we find the eigenvectors and eigenvalues. Corresponding to the largest eigenvalues we select $l$ eigenvectors. Lets call the matrix consisting of these eigenvectors $W$.
$W$ will have dimensions $d times l$ . Then we can write $Z = MW$ and we understand that each row of Z is a lower dimensional embedding of $m_i$ a row of $M$.
OK now suppose we can write $M = U S V^T$. Then we notice that :
begin{align}
M^TM &= VS^TU^TUSV^T\
&= V(S^TS)V^T text{(since U is orthonormal)} \
&= VDV^T
end{align}
Where $D = S^2$ is a diagonal matrix containing the squares of singular values.
Thus we have that $(M^TM)V = VD$ since $V$ is also orthonormal. Aha! So we can see that the columns of V are the eigenvectors of $M^TM$ and $D$ contains the eigenvalues. This $V$ is precisely what we called $W$ above.
Hope that clears things up a bit. Sorry for the delay :)
answered May 14 '15 at 2:15
faith_in_factsfaith_in_facts
1,47221530
$endgroup$
add a comment |
$begingroup$
Ok - Ill give it a shot. Let me start from the top and try to recap PCA, and then show the connection to SVD.
Recall: For PCA,
We begin with a centered $M$ (dim = $(n,d)$) of data. For this data, we compute the sample covariance : $S = frac{1}{n-1}M^TM$ where $n$ is the number of data points.
For this covariance matrix we find the eigenvectors and eigenvalues. Corresponding to the largest eigenvalues we select $l$ eigenvectors. Lets call the matrix consisting of these eigenvectors $W$.
$W$ will have dimensions $d times l$ . Then we can write $Z = MW$ and we understand that each row of Z is a lower dimensional embedding of $m_i$ a row of $M$.
OK now suppose we can write $M = U S V^T$. Then we notice that :
begin{align}
M^TM &= VS^TU^TUSV^T\
&= V(S^TS)V^T text{(since U is orthonormal)} \
&= VDV^T
end{align}
Where $D = S^2$ is a diagonal matrix containing the squares of singular values.
Thus we have that $(M^TM)V = VD$ since $V$ is also orthonormal. Aha! So we can see that the columns of V are the eigenvectors of $M^TM$ and $D$ contains the eigenvalues. This $V$ is precisely what we called $W$ above.
Hope that clears things up a bit. Sorry for the delay :)
answered May 14 '15 at 2:15
faith_in_factsfaith_in_facts
1,47221530
$endgroup$
add a comment |
$begingroup$
Ok - Ill give it a shot. Let me start from the top and try to recap PCA, and then show the connection to SVD.
Recall: For PCA,
We begin with a centered $M$ (dim = $(n,d)$) of data. For this data, we compute the sample covariance : $S = frac{1}{n-1}M^TM$ where $n$ is the number of data points.
For this covariance matrix we find the eigenvectors and eigenvalues. Corresponding to the largest eigenvalues we select $l$ eigenvectors. Lets call the matrix consisting of these eigenvectors $W$.
$W$ will have dimensions $d times l$ . Then we can write $Z = MW$ and we understand that each row of Z is a lower dimensional embedding of $m_i$ a row of $M$.
OK now suppose we can write $M = U S V^T$. Then we notice that :
begin{align}
M^TM &= VS^TU^TUSV^T\
&= V(S^TS)V^T text{(since U is orthonormal)} \
&= VDV^T
end{align}
Where $D = S^2$ is a diagonal matrix containing the squares of singular values.
Thus we have that $(M^TM)V = VD$ since $V$ is also orthonormal. Aha! So we can see that the columns of V are the eigenvectors of $M^TM$ and $D$ contains the eigenvalues. This $V$ is precisely what we called $W$ above.
Hope that clears things up a bit. Sorry for the delay :)
answered May 14 '15 at 2:15
faith_in_factsfaith_in_facts
1,47221530
$endgroup$
Ok - Ill give it a shot. Let me start from the top and try to recap PCA, and then show the connection to SVD.
Recall: For PCA,
We begin with a centered $M$ (dim = $(n,d)$) of data. For this data, we compute the sample covariance : $S = frac{1}{n-1}M^TM$ where $n$ is the number of data points.
For this covariance matrix we find the eigenvectors and eigenvalues. Corresponding to the largest eigenvalues we select $l$ eigenvectors. Lets call the matrix consisting of these eigenvectors $W$.
$W$ will have dimensions $d times l$ . Then we can write $Z = MW$ and we understand that each row of Z is a lower dimensional embedding of $m_i$ a row of $M$.
OK now suppose we can write $M = U S V^T$. Then we notice that :
begin{align}
M^TM &= VS^TU^TUSV^T\
&= V(S^TS)V^T text{(since U is orthonormal)} \
&= VDV^T
end{align}
Where $D = S^2$ is a diagonal matrix containing the squares of singular values.
Thus we have that $(M^TM)V = VD$ since $V$ is also orthonormal. Aha! So we can see that the columns of V are the eigenvectors of $M^TM$ and $D$ contains the eigenvalues. This $V$ is precisely what we called $W$ above.
Hope that clears things up a bit. Sorry for the delay :)
answered May 14 '15 at 2:15
faith_in_factsfaith_in_facts
1,47221530
answered May 14 '15 at 2:15
faith_in_factsfaith_in_facts
1,47221530
answered May 14 '15 at 2:15
faith_in_factsfaith_in_facts
1,47221530
answered May 14 '15 at 2:15
faith_in_factsfaith_in_facts
1,47221530
1,47221530
add a comment |
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f195385%2fwhat-is-the-principle-components-matrix-in-pca-with-svd%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
$begingroup$
question is not very clear. What do you wanna learn exactly?
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 20:04
$begingroup$
I apologize for the vagueness of my question, which probably reflects my vague understanding of the subject matter. I specifically wish to learn if what I calculate to be matrix Z contains the principal components of my PCA analysis. I.e. would column one of Z be the first principal component of my data matrix M?
$endgroup$
– Paul Hunter
Sep 13 '12 at 20:10
1
$begingroup$
You are right. Basically you obtain your principle components by multiplying it with the eigen matrices that you obtained after $SVD$ The most significant components after multiplication are called principle components.
$endgroup$
– Seyhmus Güngören
Sep 13 '12 at 21:37