Mixing
adding a small constant to the diagonals of a matrix to stabilize
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
$begingroup$
I have a large correlation matrix (110x110) with some small eigenvalues (about 20 < 0.1). It has been suggested that adding a constant (about 0.1) to the diagonals will help to stabilize the matrix. My ultimate goal is to run dimensionality reduction on this matrix (PCA or Factor analysis). So when I do that, should I be using the "stable" matrix? How would one expect the results to change due to adding the constants?
matrix eigenvalues numerics
edited Feb 3 at 11:27
kjetil b halvorsen
32.4k985241
asked Feb 3 at 4:21
undertheskyunderthesky
162
$endgroup$
add a comment |
$begingroup$
I have a large correlation matrix (110x110) with some small eigenvalues (about 20 < 0.1). It has been suggested that adding a constant (about 0.1) to the diagonals will help to stabilize the matrix. My ultimate goal is to run dimensionality reduction on this matrix (PCA or Factor analysis). So when I do that, should I be using the "stable" matrix? How would one expect the results to change due to adding the constants?
matrix eigenvalues numerics
edited Feb 3 at 11:27
kjetil b halvorsen
32.4k985241
asked Feb 3 at 4:21
undertheskyunderthesky
162
$endgroup$
add a comment |
$begingroup$
I have a large correlation matrix (110x110) with some small eigenvalues (about 20 < 0.1). It has been suggested that adding a constant (about 0.1) to the diagonals will help to stabilize the matrix. My ultimate goal is to run dimensionality reduction on this matrix (PCA or Factor analysis). So when I do that, should I be using the "stable" matrix? How would one expect the results to change due to adding the constants?
matrix eigenvalues numerics
edited Feb 3 at 11:27
kjetil b halvorsen
32.4k985241
asked Feb 3 at 4:21
undertheskyunderthesky
162
$endgroup$
I have a large correlation matrix (110x110) with some small eigenvalues (about 20 < 0.1). It has been suggested that adding a constant (about 0.1) to the diagonals will help to stabilize the matrix. My ultimate goal is to run dimensionality reduction on this matrix (PCA or Factor analysis). So when I do that, should I be using the "stable" matrix? How would one expect the results to change due to adding the constants?
matrix eigenvalues numerics
matrix eigenvalues numerics
edited Feb 3 at 11:27
kjetil b halvorsen
32.4k985241
asked Feb 3 at 4:21
undertheskyunderthesky
162
edited Feb 3 at 11:27
kjetil b halvorsen
32.4k985241
asked Feb 3 at 4:21
undertheskyunderthesky
162
edited Feb 3 at 11:27
kjetil b halvorsen
32.4k985241
edited Feb 3 at 11:27
kjetil b halvorsen
32.4k985241
edited Feb 3 at 11:27
kjetil b halvorsen
32.4k985241
32.4k985241
asked Feb 3 at 4:21
undertheskyunderthesky
162
asked Feb 3 at 4:21
undertheskyunderthesky
162
asked Feb 3 at 4:21
undertheskyunderthesky
162
162
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
This is in addition to OmG's good answer (+1).
Numerically, the size of the eigenvalues that is not directly a problem but rather the condition number of the matrix (i.e. the ratio between the largest and the smallest singular value) as this is what it relates to the "stability" of the system. What you propose (adding a small constant along the diagonal of the covariance/correlation matrix) is effectively a ridge regression/regularisation solution. (I think it is actually a very good solution). Other options would be: 1. Get more data, 2. Eliminate certain explanatory variables or 3. combine some of the explanatory variables to form new ones.
Regarding the potential influence the results of a PCA analysis might experience due to this regularisation step: Given the $110 times 110$ initial correlation matrix $A$ you described, qualitatively we should see very small differences in the estimated principal components of the major orders (e.g. $k = {1,...,5}$, etc.) but we should expect more pronounced differences in the minor orders (e.g. $k = {60+}$). Notice that if we use a truncated SVD approach to do our PCA in our original sample $Y$ (not the correlation matrix $A$) we will never worry for those minor order principal components anyway. We would only care for the top $k$ components and the lower magnitude components that would be potentially affect by near-collinearity would be excluded from the analysis automatically.
I recently read Spanos (2018) "Near-collinearity in linear regression revisited: The numerical vs. the statistical perspective" and I think it will greatly assist one's understanding about the possible implication of near-collinearity can have in an analysis (sign-reversal, changes in magnitude or loss of significance when testing, etc.) The paper takes the position of treating near-collinearity, "numerically" as a data problem (effectively what you describe when looking at eigenvalues, norms, etc.) and "statistically" as a parameter estimation problem. (effectively why we worry about it). (A free copy can be found here.)
answered Feb 3 at 11:48
usεr11852usεr11852
19.9k14477
$endgroup$
$begingroup$
Thanks! This is very useful. I will look up the reference.
$endgroup$
– underthesky
Feb 3 at 17:53
add a comment |
$begingroup$
Because the eigenvalues come from $det(A-lambda I) = 0$. Hence, if you add some constant $c$ to the elements of the diagonal of $A$, you will have $det(A + cI - lambda I) = det (A-(lambda -c)I)$. Hence, the eigenvalue of the $A' = A + cI$, as $c$ is greater than $lambda$ and $lambda$ is small, is near to $c$ and the problem of small eigenvalues will be solved for $A'$.
edited Feb 3 at 13:03
Karolis Koncevičius
2,35341529
answered Feb 3 at 6:52
OmGOmG
36629
$endgroup$
$begingroup$
@underthesky my pleasure.
$endgroup$
– OmG
Feb 4 at 9:40
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "65"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f390532%2fadding-a-small-constant-to-the-diagonals-of-a-matrix-to-stabilize%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
This is in addition to OmG's good answer (+1).
Numerically, the size of the eigenvalues that is not directly a problem but rather the condition number of the matrix (i.e. the ratio between the largest and the smallest singular value) as this is what it relates to the "stability" of the system. What you propose (adding a small constant along the diagonal of the covariance/correlation matrix) is effectively a ridge regression/regularisation solution. (I think it is actually a very good solution). Other options would be: 1. Get more data, 2. Eliminate certain explanatory variables or 3. combine some of the explanatory variables to form new ones.
Regarding the potential influence the results of a PCA analysis might experience due to this regularisation step: Given the $110 times 110$ initial correlation matrix $A$ you described, qualitatively we should see very small differences in the estimated principal components of the major orders (e.g. $k = {1,...,5}$, etc.) but we should expect more pronounced differences in the minor orders (e.g. $k = {60+}$). Notice that if we use a truncated SVD approach to do our PCA in our original sample $Y$ (not the correlation matrix $A$) we will never worry for those minor order principal components anyway. We would only care for the top $k$ components and the lower magnitude components that would be potentially affect by near-collinearity would be excluded from the analysis automatically.
I recently read Spanos (2018) "Near-collinearity in linear regression revisited: The numerical vs. the statistical perspective" and I think it will greatly assist one's understanding about the possible implication of near-collinearity can have in an analysis (sign-reversal, changes in magnitude or loss of significance when testing, etc.) The paper takes the position of treating near-collinearity, "numerically" as a data problem (effectively what you describe when looking at eigenvalues, norms, etc.) and "statistically" as a parameter estimation problem. (effectively why we worry about it). (A free copy can be found here.)
answered Feb 3 at 11:48
usεr11852usεr11852
19.9k14477
$endgroup$
$begingroup$
Thanks! This is very useful. I will look up the reference.
$endgroup$
– underthesky
Feb 3 at 17:53
add a comment |
$begingroup$
This is in addition to OmG's good answer (+1).
Numerically, the size of the eigenvalues that is not directly a problem but rather the condition number of the matrix (i.e. the ratio between the largest and the smallest singular value) as this is what it relates to the "stability" of the system. What you propose (adding a small constant along the diagonal of the covariance/correlation matrix) is effectively a ridge regression/regularisation solution. (I think it is actually a very good solution). Other options would be: 1. Get more data, 2. Eliminate certain explanatory variables or 3. combine some of the explanatory variables to form new ones.
Regarding the potential influence the results of a PCA analysis might experience due to this regularisation step: Given the $110 times 110$ initial correlation matrix $A$ you described, qualitatively we should see very small differences in the estimated principal components of the major orders (e.g. $k = {1,...,5}$, etc.) but we should expect more pronounced differences in the minor orders (e.g. $k = {60+}$). Notice that if we use a truncated SVD approach to do our PCA in our original sample $Y$ (not the correlation matrix $A$) we will never worry for those minor order principal components anyway. We would only care for the top $k$ components and the lower magnitude components that would be potentially affect by near-collinearity would be excluded from the analysis automatically.
I recently read Spanos (2018) "Near-collinearity in linear regression revisited: The numerical vs. the statistical perspective" and I think it will greatly assist one's understanding about the possible implication of near-collinearity can have in an analysis (sign-reversal, changes in magnitude or loss of significance when testing, etc.) The paper takes the position of treating near-collinearity, "numerically" as a data problem (effectively what you describe when looking at eigenvalues, norms, etc.) and "statistically" as a parameter estimation problem. (effectively why we worry about it). (A free copy can be found here.)
answered Feb 3 at 11:48
usεr11852usεr11852
19.9k14477
$endgroup$
$begingroup$
Thanks! This is very useful. I will look up the reference.
$endgroup$
– underthesky
Feb 3 at 17:53
add a comment |
$begingroup$
This is in addition to OmG's good answer (+1).
Numerically, the size of the eigenvalues that is not directly a problem but rather the condition number of the matrix (i.e. the ratio between the largest and the smallest singular value) as this is what it relates to the "stability" of the system. What you propose (adding a small constant along the diagonal of the covariance/correlation matrix) is effectively a ridge regression/regularisation solution. (I think it is actually a very good solution). Other options would be: 1. Get more data, 2. Eliminate certain explanatory variables or 3. combine some of the explanatory variables to form new ones.
Regarding the potential influence the results of a PCA analysis might experience due to this regularisation step: Given the $110 times 110$ initial correlation matrix $A$ you described, qualitatively we should see very small differences in the estimated principal components of the major orders (e.g. $k = {1,...,5}$, etc.) but we should expect more pronounced differences in the minor orders (e.g. $k = {60+}$). Notice that if we use a truncated SVD approach to do our PCA in our original sample $Y$ (not the correlation matrix $A$) we will never worry for those minor order principal components anyway. We would only care for the top $k$ components and the lower magnitude components that would be potentially affect by near-collinearity would be excluded from the analysis automatically.
I recently read Spanos (2018) "Near-collinearity in linear regression revisited: The numerical vs. the statistical perspective" and I think it will greatly assist one's understanding about the possible implication of near-collinearity can have in an analysis (sign-reversal, changes in magnitude or loss of significance when testing, etc.) The paper takes the position of treating near-collinearity, "numerically" as a data problem (effectively what you describe when looking at eigenvalues, norms, etc.) and "statistically" as a parameter estimation problem. (effectively why we worry about it). (A free copy can be found here.)
answered Feb 3 at 11:48
usεr11852usεr11852
19.9k14477
$endgroup$
This is in addition to OmG's good answer (+1).
Numerically, the size of the eigenvalues that is not directly a problem but rather the condition number of the matrix (i.e. the ratio between the largest and the smallest singular value) as this is what it relates to the "stability" of the system. What you propose (adding a small constant along the diagonal of the covariance/correlation matrix) is effectively a ridge regression/regularisation solution. (I think it is actually a very good solution). Other options would be: 1. Get more data, 2. Eliminate certain explanatory variables or 3. combine some of the explanatory variables to form new ones.
Regarding the potential influence the results of a PCA analysis might experience due to this regularisation step: Given the $110 times 110$ initial correlation matrix $A$ you described, qualitatively we should see very small differences in the estimated principal components of the major orders (e.g. $k = {1,...,5}$, etc.) but we should expect more pronounced differences in the minor orders (e.g. $k = {60+}$). Notice that if we use a truncated SVD approach to do our PCA in our original sample $Y$ (not the correlation matrix $A$) we will never worry for those minor order principal components anyway. We would only care for the top $k$ components and the lower magnitude components that would be potentially affect by near-collinearity would be excluded from the analysis automatically.
I recently read Spanos (2018) "Near-collinearity in linear regression revisited: The numerical vs. the statistical perspective" and I think it will greatly assist one's understanding about the possible implication of near-collinearity can have in an analysis (sign-reversal, changes in magnitude or loss of significance when testing, etc.) The paper takes the position of treating near-collinearity, "numerically" as a data problem (effectively what you describe when looking at eigenvalues, norms, etc.) and "statistically" as a parameter estimation problem. (effectively why we worry about it). (A free copy can be found here.)
answered Feb 3 at 11:48
usεr11852usεr11852
19.9k14477
edited Feb 3 at 11:58
answered Feb 3 at 11:48
usεr11852usεr11852
19.9k14477
answered Feb 3 at 11:48
usεr11852usεr11852
19.9k14477
answered Feb 3 at 11:48
usεr11852usεr11852
19.9k14477
19.9k14477
$begingroup$
Thanks! This is very useful. I will look up the reference.
$endgroup$
– underthesky
Feb 3 at 17:53
add a comment |
$begingroup$
Thanks! This is very useful. I will look up the reference.
$endgroup$
– underthesky
Feb 3 at 17:53
$begingroup$
Thanks! This is very useful. I will look up the reference.
$endgroup$
– underthesky
Feb 3 at 17:53
$begingroup$
Thanks! This is very useful. I will look up the reference.
$endgroup$
– underthesky
Feb 3 at 17:53
add a comment |
$begingroup$
Because the eigenvalues come from $det(A-lambda I) = 0$. Hence, if you add some constant $c$ to the elements of the diagonal of $A$, you will have $det(A + cI - lambda I) = det (A-(lambda -c)I)$. Hence, the eigenvalue of the $A' = A + cI$, as $c$ is greater than $lambda$ and $lambda$ is small, is near to $c$ and the problem of small eigenvalues will be solved for $A'$.
edited Feb 3 at 13:03
Karolis Koncevičius
2,35341529
answered Feb 3 at 6:52
OmGOmG
36629
$endgroup$
$begingroup$
@underthesky my pleasure.
$endgroup$
– OmG
Feb 4 at 9:40
add a comment |
$begingroup$
Because the eigenvalues come from $det(A-lambda I) = 0$. Hence, if you add some constant $c$ to the elements of the diagonal of $A$, you will have $det(A + cI - lambda I) = det (A-(lambda -c)I)$. Hence, the eigenvalue of the $A' = A + cI$, as $c$ is greater than $lambda$ and $lambda$ is small, is near to $c$ and the problem of small eigenvalues will be solved for $A'$.
edited Feb 3 at 13:03
Karolis Koncevičius
2,35341529
answered Feb 3 at 6:52
OmGOmG
36629
$endgroup$
$begingroup$
@underthesky my pleasure.
$endgroup$
– OmG
Feb 4 at 9:40
add a comment |
$begingroup$
Because the eigenvalues come from $det(A-lambda I) = 0$. Hence, if you add some constant $c$ to the elements of the diagonal of $A$, you will have $det(A + cI - lambda I) = det (A-(lambda -c)I)$. Hence, the eigenvalue of the $A' = A + cI$, as $c$ is greater than $lambda$ and $lambda$ is small, is near to $c$ and the problem of small eigenvalues will be solved for $A'$.
edited Feb 3 at 13:03
Karolis Koncevičius
2,35341529
answered Feb 3 at 6:52
OmGOmG
36629
$endgroup$
Because the eigenvalues come from $det(A-lambda I) = 0$. Hence, if you add some constant $c$ to the elements of the diagonal of $A$, you will have $det(A + cI - lambda I) = det (A-(lambda -c)I)$. Hence, the eigenvalue of the $A' = A + cI$, as $c$ is greater than $lambda$ and $lambda$ is small, is near to $c$ and the problem of small eigenvalues will be solved for $A'$.
edited Feb 3 at 13:03
Karolis Koncevičius
2,35341529
answered Feb 3 at 6:52
OmGOmG
36629
edited Feb 3 at 13:03
Karolis Koncevičius
2,35341529
edited Feb 3 at 13:03
Karolis Koncevičius
2,35341529
edited Feb 3 at 13:03
Karolis Koncevičius
2,35341529
2,35341529
answered Feb 3 at 6:52
OmGOmG
36629
answered Feb 3 at 6:52
OmGOmG
36629
answered Feb 3 at 6:52
OmGOmG
36629
36629
$begingroup$
@underthesky my pleasure.
$endgroup$
– OmG
Feb 4 at 9:40
add a comment |
$begingroup$
@underthesky my pleasure.
$endgroup$
– OmG
Feb 4 at 9:40
$begingroup$
@underthesky my pleasure.
$endgroup$
– OmG
Feb 4 at 9:40
$begingroup$
@underthesky my pleasure.
$endgroup$
– OmG
Feb 4 at 9:40
add a comment |
Thanks for contributing an answer to Cross Validated!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f390532%2fadding-a-small-constant-to-the-diagonals-of-a-matrix-to-stabilize%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
