Mixing
higher moments of entropy… does the variance of $ log x $ have any operational meaning?
$begingroup$
The Shannon entropy is the average of the negative log of a list of probabilities $ { x_1 , dots , x_d} $, i.e. $$ H(x)= -sumlimits_{i=1}^d x_i log x_i $$ there are of course lots of nice interpretations of the Shannon entropy. What about the variance of $ -log x_i $ ? $$ sigma^2 (-log x)=sumlimits_i x_i (log x_i )^2-left( sumlimits_i x_i log x_i right)^2 $$ does this have any meaning / has it been used in the literature?
probability information-theory coding-theory entropy
edited Jan 19 at 2:31
user549397
1,5081418
asked Jan 25 '16 at 16:34
jdizzlejdizzle
154112
$endgroup$
add a comment |
$begingroup$
The Shannon entropy is the average of the negative log of a list of probabilities $ { x_1 , dots , x_d} $, i.e. $$ H(x)= -sumlimits_{i=1}^d x_i log x_i $$ there are of course lots of nice interpretations of the Shannon entropy. What about the variance of $ -log x_i $ ? $$ sigma^2 (-log x)=sumlimits_i x_i (log x_i )^2-left( sumlimits_i x_i log x_i right)^2 $$ does this have any meaning / has it been used in the literature?
probability information-theory coding-theory entropy
edited Jan 19 at 2:31
user549397
1,5081418
asked Jan 25 '16 at 16:34
jdizzlejdizzle
154112
$endgroup$
add a comment |
$begingroup$
The Shannon entropy is the average of the negative log of a list of probabilities $ { x_1 , dots , x_d} $, i.e. $$ H(x)= -sumlimits_{i=1}^d x_i log x_i $$ there are of course lots of nice interpretations of the Shannon entropy. What about the variance of $ -log x_i $ ? $$ sigma^2 (-log x)=sumlimits_i x_i (log x_i )^2-left( sumlimits_i x_i log x_i right)^2 $$ does this have any meaning / has it been used in the literature?
probability information-theory coding-theory entropy
edited Jan 19 at 2:31
user549397
1,5081418
asked Jan 25 '16 at 16:34
jdizzlejdizzle
154112
$endgroup$
The Shannon entropy is the average of the negative log of a list of probabilities $ { x_1 , dots , x_d} $, i.e. $$ H(x)= -sumlimits_{i=1}^d x_i log x_i $$ there are of course lots of nice interpretations of the Shannon entropy. What about the variance of $ -log x_i $ ? $$ sigma^2 (-log x)=sumlimits_i x_i (log x_i )^2-left( sumlimits_i x_i log x_i right)^2 $$ does this have any meaning / has it been used in the literature?
probability information-theory coding-theory entropy
probability information-theory coding-theory entropy
edited Jan 19 at 2:31
user549397
1,5081418
asked Jan 25 '16 at 16:34
jdizzlejdizzle
154112
edited Jan 19 at 2:31
user549397
1,5081418
asked Jan 25 '16 at 16:34
jdizzlejdizzle
154112
edited Jan 19 at 2:31
user549397
1,5081418
edited Jan 19 at 2:31
user549397
1,5081418
edited Jan 19 at 2:31
user549397
1,5081418
1,5081418
asked Jan 25 '16 at 16:34
jdizzlejdizzle
154112
asked Jan 25 '16 at 16:34
jdizzlejdizzle
154112
asked Jan 25 '16 at 16:34
jdizzlejdizzle
154112
154112
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
$log 1/x_i$ is sometimes known as the 'surprise' (e.g. in units of bits) of drawing the symbol $x_i$, and $log 1/X$, being a random variable, has all the operational meanings that come with any random variable, namely, entropy is the average 'surprise'; similarly, higher moments are simply higher moments of the surprise measure of $X$.
There is indeed a literature on using the variance of information measures (not of surprise in this case, but of divergence), here are two good places to get started on a concept called 'dispersion':
http://people.lids.mit.edu/yp/homepage/data/gauss_isit.pdf
http://arxiv.org/pdf/1109.6310v2.pdf
The application is clear. When you only know the expected value of a random variable, you know it at first order. But when you need to get tighter bounds you need to use higher moments.
answered Jan 25 '16 at 16:53
NimrodNimrod
63647
$endgroup$
add a comment |
$begingroup$
More generally, this paper does talk about higher moments of information (though there does not seem to be that much follow-up work on this):
H. Jürgensen, D. E. Matthews, "Entropy and Higher Moments of Information", Journal of Universal Computer Science vol 16, nr. 5 (2010)
Link here: http://www.jucs.org/jucs_16_5/entropy_and_higher_moments/jucs_16_05_0749_0794_juergensen.pdf
answered Jan 19 at 2:00
VeechVeech
1184
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f1626522%2fhigher-moments-of-entropy-does-the-variance-of-log-x-have-any-operationa%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
$log 1/x_i$ is sometimes known as the 'surprise' (e.g. in units of bits) of drawing the symbol $x_i$, and $log 1/X$, being a random variable, has all the operational meanings that come with any random variable, namely, entropy is the average 'surprise'; similarly, higher moments are simply higher moments of the surprise measure of $X$.
There is indeed a literature on using the variance of information measures (not of surprise in this case, but of divergence), here are two good places to get started on a concept called 'dispersion':
http://people.lids.mit.edu/yp/homepage/data/gauss_isit.pdf
http://arxiv.org/pdf/1109.6310v2.pdf
The application is clear. When you only know the expected value of a random variable, you know it at first order. But when you need to get tighter bounds you need to use higher moments.
answered Jan 25 '16 at 16:53
NimrodNimrod
63647
$endgroup$
add a comment |
$begingroup$
$log 1/x_i$ is sometimes known as the 'surprise' (e.g. in units of bits) of drawing the symbol $x_i$, and $log 1/X$, being a random variable, has all the operational meanings that come with any random variable, namely, entropy is the average 'surprise'; similarly, higher moments are simply higher moments of the surprise measure of $X$.
There is indeed a literature on using the variance of information measures (not of surprise in this case, but of divergence), here are two good places to get started on a concept called 'dispersion':
http://people.lids.mit.edu/yp/homepage/data/gauss_isit.pdf
http://arxiv.org/pdf/1109.6310v2.pdf
The application is clear. When you only know the expected value of a random variable, you know it at first order. But when you need to get tighter bounds you need to use higher moments.
answered Jan 25 '16 at 16:53
NimrodNimrod
63647
$endgroup$
add a comment |
$begingroup$
$log 1/x_i$ is sometimes known as the 'surprise' (e.g. in units of bits) of drawing the symbol $x_i$, and $log 1/X$, being a random variable, has all the operational meanings that come with any random variable, namely, entropy is the average 'surprise'; similarly, higher moments are simply higher moments of the surprise measure of $X$.
There is indeed a literature on using the variance of information measures (not of surprise in this case, but of divergence), here are two good places to get started on a concept called 'dispersion':
http://people.lids.mit.edu/yp/homepage/data/gauss_isit.pdf
http://arxiv.org/pdf/1109.6310v2.pdf
The application is clear. When you only know the expected value of a random variable, you know it at first order. But when you need to get tighter bounds you need to use higher moments.
answered Jan 25 '16 at 16:53
NimrodNimrod
63647
$endgroup$
$log 1/x_i$ is sometimes known as the 'surprise' (e.g. in units of bits) of drawing the symbol $x_i$, and $log 1/X$, being a random variable, has all the operational meanings that come with any random variable, namely, entropy is the average 'surprise'; similarly, higher moments are simply higher moments of the surprise measure of $X$.
There is indeed a literature on using the variance of information measures (not of surprise in this case, but of divergence), here are two good places to get started on a concept called 'dispersion':
http://people.lids.mit.edu/yp/homepage/data/gauss_isit.pdf
http://arxiv.org/pdf/1109.6310v2.pdf
The application is clear. When you only know the expected value of a random variable, you know it at first order. But when you need to get tighter bounds you need to use higher moments.
answered Jan 25 '16 at 16:53
NimrodNimrod
63647
answered Jan 25 '16 at 16:53
NimrodNimrod
63647
answered Jan 25 '16 at 16:53
NimrodNimrod
63647
answered Jan 25 '16 at 16:53
NimrodNimrod
63647
63647
add a comment |
add a comment |
$begingroup$
More generally, this paper does talk about higher moments of information (though there does not seem to be that much follow-up work on this):
H. Jürgensen, D. E. Matthews, "Entropy and Higher Moments of Information", Journal of Universal Computer Science vol 16, nr. 5 (2010)
Link here: http://www.jucs.org/jucs_16_5/entropy_and_higher_moments/jucs_16_05_0749_0794_juergensen.pdf
answered Jan 19 at 2:00
VeechVeech
1184
$endgroup$
add a comment |
$begingroup$
More generally, this paper does talk about higher moments of information (though there does not seem to be that much follow-up work on this):
H. Jürgensen, D. E. Matthews, "Entropy and Higher Moments of Information", Journal of Universal Computer Science vol 16, nr. 5 (2010)
Link here: http://www.jucs.org/jucs_16_5/entropy_and_higher_moments/jucs_16_05_0749_0794_juergensen.pdf
answered Jan 19 at 2:00
VeechVeech
1184
$endgroup$
add a comment |
$begingroup$
More generally, this paper does talk about higher moments of information (though there does not seem to be that much follow-up work on this):
H. Jürgensen, D. E. Matthews, "Entropy and Higher Moments of Information", Journal of Universal Computer Science vol 16, nr. 5 (2010)
Link here: http://www.jucs.org/jucs_16_5/entropy_and_higher_moments/jucs_16_05_0749_0794_juergensen.pdf
answered Jan 19 at 2:00
VeechVeech
1184
$endgroup$
More generally, this paper does talk about higher moments of information (though there does not seem to be that much follow-up work on this):
H. Jürgensen, D. E. Matthews, "Entropy and Higher Moments of Information", Journal of Universal Computer Science vol 16, nr. 5 (2010)
Link here: http://www.jucs.org/jucs_16_5/entropy_and_higher_moments/jucs_16_05_0749_0794_juergensen.pdf
answered Jan 19 at 2:00
VeechVeech
1184
answered Jan 19 at 2:00
VeechVeech
1184
answered Jan 19 at 2:00
VeechVeech
1184
answered Jan 19 at 2:00
VeechVeech
1184
1184
add a comment |
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f1626522%2fhigher-moments-of-entropy-does-the-variance-of-log-x-have-any-operationa%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
