Mixing
Can we compute confidence intervals for the variance of an unknown distributions from sample variances?
$begingroup$
Assume $X_1,ldots,X_n$ are i.i.d. with unknown distribution $mathcal D$ - we only know it is not normal and has finite variance.
Is there a way to give confidence intervals for the variance of $mathcal D$?
Can we base the confidence interval on the sample variance $hat sigma^2 = frac1{n-1}sum_{i=1}^n (X_i - bar X)^2$ with $bar X = frac1nsum_{i=1}^n X_i$ the sample mean?
If it helps, we can restrict ourselves to distributions with mean $0$. (From my application, I can compute the exact mean $mu$ of $mathcal D$ and then consider $Y_i := X_i - mu$ to estimate the variance.)
I know the approaches via $chi^2$-distribution if we assume a normal distribution $mathcal D = mathcal N(mu,sigma^2)$, but I as noted above $mathcal D$ is not normal.
statistics estimation-theory
asked Mar 24 '13 at 15:48
SebastianSebastian
17718
$endgroup$
add a comment |
$begingroup$
Assume $X_1,ldots,X_n$ are i.i.d. with unknown distribution $mathcal D$ - we only know it is not normal and has finite variance.
Is there a way to give confidence intervals for the variance of $mathcal D$?
Can we base the confidence interval on the sample variance $hat sigma^2 = frac1{n-1}sum_{i=1}^n (X_i - bar X)^2$ with $bar X = frac1nsum_{i=1}^n X_i$ the sample mean?
If it helps, we can restrict ourselves to distributions with mean $0$. (From my application, I can compute the exact mean $mu$ of $mathcal D$ and then consider $Y_i := X_i - mu$ to estimate the variance.)
I know the approaches via $chi^2$-distribution if we assume a normal distribution $mathcal D = mathcal N(mu,sigma^2)$, but I as noted above $mathcal D$ is not normal.
statistics estimation-theory
asked Mar 24 '13 at 15:48
SebastianSebastian
17718
$endgroup$
add a comment |
$begingroup$
Assume $X_1,ldots,X_n$ are i.i.d. with unknown distribution $mathcal D$ - we only know it is not normal and has finite variance.
Is there a way to give confidence intervals for the variance of $mathcal D$?
Can we base the confidence interval on the sample variance $hat sigma^2 = frac1{n-1}sum_{i=1}^n (X_i - bar X)^2$ with $bar X = frac1nsum_{i=1}^n X_i$ the sample mean?
If it helps, we can restrict ourselves to distributions with mean $0$. (From my application, I can compute the exact mean $mu$ of $mathcal D$ and then consider $Y_i := X_i - mu$ to estimate the variance.)
I know the approaches via $chi^2$-distribution if we assume a normal distribution $mathcal D = mathcal N(mu,sigma^2)$, but I as noted above $mathcal D$ is not normal.
statistics estimation-theory
asked Mar 24 '13 at 15:48
SebastianSebastian
17718
$endgroup$
Assume $X_1,ldots,X_n$ are i.i.d. with unknown distribution $mathcal D$ - we only know it is not normal and has finite variance.
Is there a way to give confidence intervals for the variance of $mathcal D$?
Can we base the confidence interval on the sample variance $hat sigma^2 = frac1{n-1}sum_{i=1}^n (X_i - bar X)^2$ with $bar X = frac1nsum_{i=1}^n X_i$ the sample mean?
If it helps, we can restrict ourselves to distributions with mean $0$. (From my application, I can compute the exact mean $mu$ of $mathcal D$ and then consider $Y_i := X_i - mu$ to estimate the variance.)
I know the approaches via $chi^2$-distribution if we assume a normal distribution $mathcal D = mathcal N(mu,sigma^2)$, but I as noted above $mathcal D$ is not normal.
statistics estimation-theory
statistics estimation-theory
asked Mar 24 '13 at 15:48
SebastianSebastian
17718
asked Mar 24 '13 at 15:48
SebastianSebastian
17718
edited Mar 24 '13 at 15:56
Sebastian
asked Mar 24 '13 at 15:48
SebastianSebastian
17718
asked Mar 24 '13 at 15:48
SebastianSebastian
17718
asked Mar 24 '13 at 15:48
SebastianSebastian
17718
17718
add a comment |
add a comment |
3 Answers
3
active
oldest
votes
$begingroup$
In practical terms, if the distribution is unknown and one has a lot of data, one can assume that the distribution of the sample variance converges to a Gaussian one (e.g. see here). Then the confidence interval can be computed from this.
One can also do bootstrapping to approximate the statistic's distribution, and use it to estimate the confidence interval.
This is (asymptotically) quite accurate.
Another method might be to use a Bayesian approach (see: A Bayesian perspective on estimating mean, variance, and standard-deviation from data by Oliphant). (This method is built into scipy already :].) Essentially, it finds that, with an "ignorant" Bayesian prior, the sample variance follows an inverted Gamma distribution, from which confidence intervals can be constructed.
See also this question about the distribution of the sample variance and this one, which is also related.
edited Apr 13 '17 at 12:44
Community♦
1
answered Feb 8 '17 at 22:36
user3658307user3658307
4,8033946
$endgroup$
add a comment |
$begingroup$
If the distribution is unknown then it will be a problem of Non-parametric or Distribution-Free.And in non-parametric we go with median or p-th quartile. So, I think there is no confidence intervals for the variance you define.
answered Mar 24 '13 at 15:59
A.DA.D
3,9631235
$endgroup$
$begingroup$
>I think there is no confidence intervals for the variance you define. Do you mean, it can proven that one cannot give such intervals (in general) or are there just no results known?
$endgroup$
– Sebastian
Mar 24 '13 at 16:28
$begingroup$
@Sebastian:I am not familiar with such result.
$endgroup$
– A.D
Mar 24 '13 at 17:43
add a comment |
$begingroup$
This is late, but I was also looking into this and figured it would be worth answering the question for other people who is also googling for the answer. You can derive concentration inequalities (and thus confidence intervals) for the sample variance through the use of $U$-statistics since the sample variance is a $U$-statistic. See https://arxiv.org/abs/1712.06160 for more details.
edited Jan 22 at 5:47
Daniele Tampieri
2,3972922
answered Jan 22 at 1:21
Ojash NeopaneOjash Neopane
111
$endgroup$
$begingroup$
Your answer basically comes down to a link to an off-site source. Such answers are generally frowned upon here, as answers on MSE should be as self-contained as possible, and off-site links can suffer from "rot". Please take the time to explain why that link is relevant and to summarize the major results or ideas.
$endgroup$
– Xander Henderson
Jan 22 at 1:52
$begingroup$
@Ojash Thanks for digging this out; since you seem to have gone through this, would you summarize the result here?
$endgroup$
– Sebastian
Feb 18 at 8:31
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f339702%2fcan-we-compute-confidence-intervals-for-the-variance-of-an-unknown-distributions%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
In practical terms, if the distribution is unknown and one has a lot of data, one can assume that the distribution of the sample variance converges to a Gaussian one (e.g. see here). Then the confidence interval can be computed from this.
One can also do bootstrapping to approximate the statistic's distribution, and use it to estimate the confidence interval.
This is (asymptotically) quite accurate.
Another method might be to use a Bayesian approach (see: A Bayesian perspective on estimating mean, variance, and standard-deviation from data by Oliphant). (This method is built into scipy already :].) Essentially, it finds that, with an "ignorant" Bayesian prior, the sample variance follows an inverted Gamma distribution, from which confidence intervals can be constructed.
See also this question about the distribution of the sample variance and this one, which is also related.
edited Apr 13 '17 at 12:44
Community♦
1
answered Feb 8 '17 at 22:36
user3658307user3658307
4,8033946
$endgroup$
add a comment |
$begingroup$
In practical terms, if the distribution is unknown and one has a lot of data, one can assume that the distribution of the sample variance converges to a Gaussian one (e.g. see here). Then the confidence interval can be computed from this.
One can also do bootstrapping to approximate the statistic's distribution, and use it to estimate the confidence interval.
This is (asymptotically) quite accurate.
Another method might be to use a Bayesian approach (see: A Bayesian perspective on estimating mean, variance, and standard-deviation from data by Oliphant). (This method is built into scipy already :].) Essentially, it finds that, with an "ignorant" Bayesian prior, the sample variance follows an inverted Gamma distribution, from which confidence intervals can be constructed.
See also this question about the distribution of the sample variance and this one, which is also related.
edited Apr 13 '17 at 12:44
Community♦
1
answered Feb 8 '17 at 22:36
user3658307user3658307
4,8033946
$endgroup$
add a comment |
$begingroup$
In practical terms, if the distribution is unknown and one has a lot of data, one can assume that the distribution of the sample variance converges to a Gaussian one (e.g. see here). Then the confidence interval can be computed from this.
One can also do bootstrapping to approximate the statistic's distribution, and use it to estimate the confidence interval.
This is (asymptotically) quite accurate.
Another method might be to use a Bayesian approach (see: A Bayesian perspective on estimating mean, variance, and standard-deviation from data by Oliphant). (This method is built into scipy already :].) Essentially, it finds that, with an "ignorant" Bayesian prior, the sample variance follows an inverted Gamma distribution, from which confidence intervals can be constructed.
See also this question about the distribution of the sample variance and this one, which is also related.
edited Apr 13 '17 at 12:44
Community♦
1
answered Feb 8 '17 at 22:36
user3658307user3658307
4,8033946
$endgroup$
In practical terms, if the distribution is unknown and one has a lot of data, one can assume that the distribution of the sample variance converges to a Gaussian one (e.g. see here). Then the confidence interval can be computed from this.
One can also do bootstrapping to approximate the statistic's distribution, and use it to estimate the confidence interval.
This is (asymptotically) quite accurate.
Another method might be to use a Bayesian approach (see: A Bayesian perspective on estimating mean, variance, and standard-deviation from data by Oliphant). (This method is built into scipy already :].) Essentially, it finds that, with an "ignorant" Bayesian prior, the sample variance follows an inverted Gamma distribution, from which confidence intervals can be constructed.
See also this question about the distribution of the sample variance and this one, which is also related.
edited Apr 13 '17 at 12:44
Community♦
1
answered Feb 8 '17 at 22:36
user3658307user3658307
4,8033946
edited Apr 13 '17 at 12:44
Community♦
1
edited Apr 13 '17 at 12:44
Community♦
1
edited Apr 13 '17 at 12:44
Community♦
1
1
answered Feb 8 '17 at 22:36
user3658307user3658307
4,8033946
answered Feb 8 '17 at 22:36
user3658307user3658307
4,8033946
answered Feb 8 '17 at 22:36
user3658307user3658307
4,8033946
4,8033946
add a comment |
add a comment |
$begingroup$
If the distribution is unknown then it will be a problem of Non-parametric or Distribution-Free.And in non-parametric we go with median or p-th quartile. So, I think there is no confidence intervals for the variance you define.
answered Mar 24 '13 at 15:59
A.DA.D
3,9631235
$endgroup$
$begingroup$
>I think there is no confidence intervals for the variance you define. Do you mean, it can proven that one cannot give such intervals (in general) or are there just no results known?
$endgroup$
– Sebastian
Mar 24 '13 at 16:28
$begingroup$
@Sebastian:I am not familiar with such result.
$endgroup$
– A.D
Mar 24 '13 at 17:43
add a comment |
$begingroup$
If the distribution is unknown then it will be a problem of Non-parametric or Distribution-Free.And in non-parametric we go with median or p-th quartile. So, I think there is no confidence intervals for the variance you define.
answered Mar 24 '13 at 15:59
A.DA.D
3,9631235
$endgroup$
$begingroup$
>I think there is no confidence intervals for the variance you define. Do you mean, it can proven that one cannot give such intervals (in general) or are there just no results known?
$endgroup$
– Sebastian
Mar 24 '13 at 16:28
$begingroup$
@Sebastian:I am not familiar with such result.
$endgroup$
– A.D
Mar 24 '13 at 17:43
add a comment |
$begingroup$
If the distribution is unknown then it will be a problem of Non-parametric or Distribution-Free.And in non-parametric we go with median or p-th quartile. So, I think there is no confidence intervals for the variance you define.
answered Mar 24 '13 at 15:59
A.DA.D
3,9631235
$endgroup$
If the distribution is unknown then it will be a problem of Non-parametric or Distribution-Free.And in non-parametric we go with median or p-th quartile. So, I think there is no confidence intervals for the variance you define.
answered Mar 24 '13 at 15:59
A.DA.D
3,9631235
answered Mar 24 '13 at 15:59
A.DA.D
3,9631235
answered Mar 24 '13 at 15:59
A.DA.D
3,9631235
answered Mar 24 '13 at 15:59
A.DA.D
3,9631235
3,9631235
$begingroup$
>I think there is no confidence intervals for the variance you define. Do you mean, it can proven that one cannot give such intervals (in general) or are there just no results known?
$endgroup$
– Sebastian
Mar 24 '13 at 16:28
$begingroup$
@Sebastian:I am not familiar with such result.
$endgroup$
– A.D
Mar 24 '13 at 17:43
add a comment |
$begingroup$
>I think there is no confidence intervals for the variance you define. Do you mean, it can proven that one cannot give such intervals (in general) or are there just no results known?
$endgroup$
– Sebastian
Mar 24 '13 at 16:28
$begingroup$
@Sebastian:I am not familiar with such result.
$endgroup$
– A.D
Mar 24 '13 at 17:43
$begingroup$
>I think there is no confidence intervals for the variance you define. Do you mean, it can proven that one cannot give such intervals (in general) or are there just no results known?
$endgroup$
– Sebastian
Mar 24 '13 at 16:28
$begingroup$
>I think there is no confidence intervals for the variance you define. Do you mean, it can proven that one cannot give such intervals (in general) or are there just no results known?
$endgroup$
– Sebastian
Mar 24 '13 at 16:28
$begingroup$
@Sebastian:I am not familiar with such result.
$endgroup$
– A.D
Mar 24 '13 at 17:43
$begingroup$
@Sebastian:I am not familiar with such result.
$endgroup$
– A.D
Mar 24 '13 at 17:43
add a comment |
$begingroup$
This is late, but I was also looking into this and figured it would be worth answering the question for other people who is also googling for the answer. You can derive concentration inequalities (and thus confidence intervals) for the sample variance through the use of $U$-statistics since the sample variance is a $U$-statistic. See https://arxiv.org/abs/1712.06160 for more details.
edited Jan 22 at 5:47
Daniele Tampieri
2,3972922
answered Jan 22 at 1:21
Ojash NeopaneOjash Neopane
111
$endgroup$
$begingroup$
Your answer basically comes down to a link to an off-site source. Such answers are generally frowned upon here, as answers on MSE should be as self-contained as possible, and off-site links can suffer from "rot". Please take the time to explain why that link is relevant and to summarize the major results or ideas.
$endgroup$
– Xander Henderson
Jan 22 at 1:52
$begingroup$
@Ojash Thanks for digging this out; since you seem to have gone through this, would you summarize the result here?
$endgroup$
– Sebastian
Feb 18 at 8:31
add a comment |
$begingroup$
This is late, but I was also looking into this and figured it would be worth answering the question for other people who is also googling for the answer. You can derive concentration inequalities (and thus confidence intervals) for the sample variance through the use of $U$-statistics since the sample variance is a $U$-statistic. See https://arxiv.org/abs/1712.06160 for more details.
edited Jan 22 at 5:47
Daniele Tampieri
2,3972922
answered Jan 22 at 1:21
Ojash NeopaneOjash Neopane
111
$endgroup$
$begingroup$
Your answer basically comes down to a link to an off-site source. Such answers are generally frowned upon here, as answers on MSE should be as self-contained as possible, and off-site links can suffer from "rot". Please take the time to explain why that link is relevant and to summarize the major results or ideas.
$endgroup$
– Xander Henderson
Jan 22 at 1:52
$begingroup$
@Ojash Thanks for digging this out; since you seem to have gone through this, would you summarize the result here?
$endgroup$
– Sebastian
Feb 18 at 8:31
add a comment |
$begingroup$
This is late, but I was also looking into this and figured it would be worth answering the question for other people who is also googling for the answer. You can derive concentration inequalities (and thus confidence intervals) for the sample variance through the use of $U$-statistics since the sample variance is a $U$-statistic. See https://arxiv.org/abs/1712.06160 for more details.
edited Jan 22 at 5:47
Daniele Tampieri
2,3972922
answered Jan 22 at 1:21
Ojash NeopaneOjash Neopane
111
$endgroup$
This is late, but I was also looking into this and figured it would be worth answering the question for other people who is also googling for the answer. You can derive concentration inequalities (and thus confidence intervals) for the sample variance through the use of $U$-statistics since the sample variance is a $U$-statistic. See https://arxiv.org/abs/1712.06160 for more details.
edited Jan 22 at 5:47
Daniele Tampieri
2,3972922
answered Jan 22 at 1:21
Ojash NeopaneOjash Neopane
111
edited Jan 22 at 5:47
Daniele Tampieri
2,3972922
edited Jan 22 at 5:47
Daniele Tampieri
2,3972922
edited Jan 22 at 5:47
Daniele Tampieri
2,3972922
2,3972922
answered Jan 22 at 1:21
Ojash NeopaneOjash Neopane
111
answered Jan 22 at 1:21
Ojash NeopaneOjash Neopane
111
answered Jan 22 at 1:21
Ojash NeopaneOjash Neopane
111
111
$begingroup$
Your answer basically comes down to a link to an off-site source. Such answers are generally frowned upon here, as answers on MSE should be as self-contained as possible, and off-site links can suffer from "rot". Please take the time to explain why that link is relevant and to summarize the major results or ideas.
$endgroup$
– Xander Henderson
Jan 22 at 1:52
$begingroup$
@Ojash Thanks for digging this out; since you seem to have gone through this, would you summarize the result here?
$endgroup$
– Sebastian
Feb 18 at 8:31
add a comment |
$begingroup$
Your answer basically comes down to a link to an off-site source. Such answers are generally frowned upon here, as answers on MSE should be as self-contained as possible, and off-site links can suffer from "rot". Please take the time to explain why that link is relevant and to summarize the major results or ideas.
$endgroup$
– Xander Henderson
Jan 22 at 1:52
$begingroup$
@Ojash Thanks for digging this out; since you seem to have gone through this, would you summarize the result here?
$endgroup$
– Sebastian
Feb 18 at 8:31
$begingroup$
Your answer basically comes down to a link to an off-site source. Such answers are generally frowned upon here, as answers on MSE should be as self-contained as possible, and off-site links can suffer from "rot". Please take the time to explain why that link is relevant and to summarize the major results or ideas.
$endgroup$
– Xander Henderson
Jan 22 at 1:52
$begingroup$
Your answer basically comes down to a link to an off-site source. Such answers are generally frowned upon here, as answers on MSE should be as self-contained as possible, and off-site links can suffer from "rot". Please take the time to explain why that link is relevant and to summarize the major results or ideas.
$endgroup$
– Xander Henderson
Jan 22 at 1:52
$begingroup$
@Ojash Thanks for digging this out; since you seem to have gone through this, would you summarize the result here?
$endgroup$
– Sebastian
Feb 18 at 8:31
$begingroup$
@Ojash Thanks for digging this out; since you seem to have gone through this, would you summarize the result here?
$endgroup$
– Sebastian
Feb 18 at 8:31
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f339702%2fcan-we-compute-confidence-intervals-for-the-variance-of-an-unknown-distributions%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
