Mixing
Can I derive a well-known distribution from another divergence?
$begingroup$
A large class of distributions can be derived from
$max_{p(x)} H(p)$
s.t.
$E_x{x}=mu$
$E_x{x^n}=c_n$
where $H(p)$ denotes the Shannon (differential) entropy and are called maximum entropy. E.g., a Gaussian in canonical form. Do you know any well-known distribution that is derived from another entropic measure? E.g., Renyi Entropy, etc?
probability-distributions entropy
asked Jan 10 at 21:09
J. ReinhardJ. Reinhard
184
$endgroup$
add a comment |
$begingroup$
A large class of distributions can be derived from
$max_{p(x)} H(p)$
s.t.
$E_x{x}=mu$
$E_x{x^n}=c_n$
where $H(p)$ denotes the Shannon (differential) entropy and are called maximum entropy. E.g., a Gaussian in canonical form. Do you know any well-known distribution that is derived from another entropic measure? E.g., Renyi Entropy, etc?
probability-distributions entropy
asked Jan 10 at 21:09
J. ReinhardJ. Reinhard
184
$endgroup$
add a comment |
$begingroup$
A large class of distributions can be derived from
$max_{p(x)} H(p)$
s.t.
$E_x{x}=mu$
$E_x{x^n}=c_n$
where $H(p)$ denotes the Shannon (differential) entropy and are called maximum entropy. E.g., a Gaussian in canonical form. Do you know any well-known distribution that is derived from another entropic measure? E.g., Renyi Entropy, etc?
probability-distributions entropy
asked Jan 10 at 21:09
J. ReinhardJ. Reinhard
184
$endgroup$
A large class of distributions can be derived from
$max_{p(x)} H(p)$
s.t.
$E_x{x}=mu$
$E_x{x^n}=c_n$
where $H(p)$ denotes the Shannon (differential) entropy and are called maximum entropy. E.g., a Gaussian in canonical form. Do you know any well-known distribution that is derived from another entropic measure? E.g., Renyi Entropy, etc?
probability-distributions entropy
probability-distributions entropy
asked Jan 10 at 21:09
J. ReinhardJ. Reinhard
184
asked Jan 10 at 21:09
J. ReinhardJ. Reinhard
184
asked Jan 10 at 21:09
J. ReinhardJ. Reinhard
184
asked Jan 10 at 21:09
J. ReinhardJ. Reinhard
184
asked Jan 10 at 21:09
J. ReinhardJ. Reinhard
184
184
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
There's a proof here that the Shannon entropy $H$ is maximal when $ln p$ is linear in the functions of $X$ (be they powers thereof or otherwise) whose means are specified as constraints. But you already knew that. If you adapt that proof, you can similarly show other results of the kind you seek. I'll talk about one outcome from doing this, but if you want to consider other alternatives to $H$ I'll leave you to try the calculations yourself. (The example I'll discuss is actually a bit hard if you're new to functional calculus, but I'll only present the result.)
For example, you can show the Rényi entropy $H_alpha$ with $alphane 1$ is maximised when $frac{p^{1-alpha}-1}{1-alpha}$ is linear in the constrained functions. This bizarre-looking quantity, which by L'Hôpital's rule becomes $ln p$ in the $alphato 1$ limit, is called the Tsallis-$alpha$ logarithm of $p$ (although with that terminology the parameter is usually called $q$).
The idea is that this function differentiates to $p^{-alpha}$, and equals $0$ when $p=1$. (Yes, I realise $p=1$ has no special meaning for probability densities, but the Tsallis-$q$ logarithm is often worth computing for quantities other than probabilities or densities thereof; in fact, you'd be amazed how many approximate but imperfect power laws in the applied sciences can be expressed in terms of Tsallis statistics, though I don't think they usually are.)
For example, what maximises $H_alpha$ for known mean and variance? Well, $p$ would have to be the Tsallis $alpha$-exponential of a quadratic, and this is (up to a linear transformation of $X$) just a Student's-$t$ distribution, with $nu$ satisfying, if my mental arithmetic is right, $alpha=frac{nu+3}{nu+1}$. Unsurprisingly, we recover the usual Gaussian in the $alphato 1,,nutoinfty$ limit.
answered Jan 10 at 22:42
J.G.J.G.
25.7k22539
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3069175%2fcan-i-derive-a-well-known-distribution-from-another-divergence%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
There's a proof here that the Shannon entropy $H$ is maximal when $ln p$ is linear in the functions of $X$ (be they powers thereof or otherwise) whose means are specified as constraints. But you already knew that. If you adapt that proof, you can similarly show other results of the kind you seek. I'll talk about one outcome from doing this, but if you want to consider other alternatives to $H$ I'll leave you to try the calculations yourself. (The example I'll discuss is actually a bit hard if you're new to functional calculus, but I'll only present the result.)
For example, you can show the Rényi entropy $H_alpha$ with $alphane 1$ is maximised when $frac{p^{1-alpha}-1}{1-alpha}$ is linear in the constrained functions. This bizarre-looking quantity, which by L'Hôpital's rule becomes $ln p$ in the $alphato 1$ limit, is called the Tsallis-$alpha$ logarithm of $p$ (although with that terminology the parameter is usually called $q$).
The idea is that this function differentiates to $p^{-alpha}$, and equals $0$ when $p=1$. (Yes, I realise $p=1$ has no special meaning for probability densities, but the Tsallis-$q$ logarithm is often worth computing for quantities other than probabilities or densities thereof; in fact, you'd be amazed how many approximate but imperfect power laws in the applied sciences can be expressed in terms of Tsallis statistics, though I don't think they usually are.)
For example, what maximises $H_alpha$ for known mean and variance? Well, $p$ would have to be the Tsallis $alpha$-exponential of a quadratic, and this is (up to a linear transformation of $X$) just a Student's-$t$ distribution, with $nu$ satisfying, if my mental arithmetic is right, $alpha=frac{nu+3}{nu+1}$. Unsurprisingly, we recover the usual Gaussian in the $alphato 1,,nutoinfty$ limit.
answered Jan 10 at 22:42
J.G.J.G.
25.7k22539
$endgroup$
add a comment |
$begingroup$
There's a proof here that the Shannon entropy $H$ is maximal when $ln p$ is linear in the functions of $X$ (be they powers thereof or otherwise) whose means are specified as constraints. But you already knew that. If you adapt that proof, you can similarly show other results of the kind you seek. I'll talk about one outcome from doing this, but if you want to consider other alternatives to $H$ I'll leave you to try the calculations yourself. (The example I'll discuss is actually a bit hard if you're new to functional calculus, but I'll only present the result.)
For example, you can show the Rényi entropy $H_alpha$ with $alphane 1$ is maximised when $frac{p^{1-alpha}-1}{1-alpha}$ is linear in the constrained functions. This bizarre-looking quantity, which by L'Hôpital's rule becomes $ln p$ in the $alphato 1$ limit, is called the Tsallis-$alpha$ logarithm of $p$ (although with that terminology the parameter is usually called $q$).
The idea is that this function differentiates to $p^{-alpha}$, and equals $0$ when $p=1$. (Yes, I realise $p=1$ has no special meaning for probability densities, but the Tsallis-$q$ logarithm is often worth computing for quantities other than probabilities or densities thereof; in fact, you'd be amazed how many approximate but imperfect power laws in the applied sciences can be expressed in terms of Tsallis statistics, though I don't think they usually are.)
For example, what maximises $H_alpha$ for known mean and variance? Well, $p$ would have to be the Tsallis $alpha$-exponential of a quadratic, and this is (up to a linear transformation of $X$) just a Student's-$t$ distribution, with $nu$ satisfying, if my mental arithmetic is right, $alpha=frac{nu+3}{nu+1}$. Unsurprisingly, we recover the usual Gaussian in the $alphato 1,,nutoinfty$ limit.
answered Jan 10 at 22:42
J.G.J.G.
25.7k22539
$endgroup$
add a comment |
$begingroup$
There's a proof here that the Shannon entropy $H$ is maximal when $ln p$ is linear in the functions of $X$ (be they powers thereof or otherwise) whose means are specified as constraints. But you already knew that. If you adapt that proof, you can similarly show other results of the kind you seek. I'll talk about one outcome from doing this, but if you want to consider other alternatives to $H$ I'll leave you to try the calculations yourself. (The example I'll discuss is actually a bit hard if you're new to functional calculus, but I'll only present the result.)
For example, you can show the Rényi entropy $H_alpha$ with $alphane 1$ is maximised when $frac{p^{1-alpha}-1}{1-alpha}$ is linear in the constrained functions. This bizarre-looking quantity, which by L'Hôpital's rule becomes $ln p$ in the $alphato 1$ limit, is called the Tsallis-$alpha$ logarithm of $p$ (although with that terminology the parameter is usually called $q$).
The idea is that this function differentiates to $p^{-alpha}$, and equals $0$ when $p=1$. (Yes, I realise $p=1$ has no special meaning for probability densities, but the Tsallis-$q$ logarithm is often worth computing for quantities other than probabilities or densities thereof; in fact, you'd be amazed how many approximate but imperfect power laws in the applied sciences can be expressed in terms of Tsallis statistics, though I don't think they usually are.)
For example, what maximises $H_alpha$ for known mean and variance? Well, $p$ would have to be the Tsallis $alpha$-exponential of a quadratic, and this is (up to a linear transformation of $X$) just a Student's-$t$ distribution, with $nu$ satisfying, if my mental arithmetic is right, $alpha=frac{nu+3}{nu+1}$. Unsurprisingly, we recover the usual Gaussian in the $alphato 1,,nutoinfty$ limit.
answered Jan 10 at 22:42
J.G.J.G.
25.7k22539
$endgroup$
There's a proof here that the Shannon entropy $H$ is maximal when $ln p$ is linear in the functions of $X$ (be they powers thereof or otherwise) whose means are specified as constraints. But you already knew that. If you adapt that proof, you can similarly show other results of the kind you seek. I'll talk about one outcome from doing this, but if you want to consider other alternatives to $H$ I'll leave you to try the calculations yourself. (The example I'll discuss is actually a bit hard if you're new to functional calculus, but I'll only present the result.)
For example, you can show the Rényi entropy $H_alpha$ with $alphane 1$ is maximised when $frac{p^{1-alpha}-1}{1-alpha}$ is linear in the constrained functions. This bizarre-looking quantity, which by L'Hôpital's rule becomes $ln p$ in the $alphato 1$ limit, is called the Tsallis-$alpha$ logarithm of $p$ (although with that terminology the parameter is usually called $q$).
The idea is that this function differentiates to $p^{-alpha}$, and equals $0$ when $p=1$. (Yes, I realise $p=1$ has no special meaning for probability densities, but the Tsallis-$q$ logarithm is often worth computing for quantities other than probabilities or densities thereof; in fact, you'd be amazed how many approximate but imperfect power laws in the applied sciences can be expressed in terms of Tsallis statistics, though I don't think they usually are.)
For example, what maximises $H_alpha$ for known mean and variance? Well, $p$ would have to be the Tsallis $alpha$-exponential of a quadratic, and this is (up to a linear transformation of $X$) just a Student's-$t$ distribution, with $nu$ satisfying, if my mental arithmetic is right, $alpha=frac{nu+3}{nu+1}$. Unsurprisingly, we recover the usual Gaussian in the $alphato 1,,nutoinfty$ limit.
answered Jan 10 at 22:42
J.G.J.G.
25.7k22539
answered Jan 10 at 22:42
J.G.J.G.
25.7k22539
answered Jan 10 at 22:42
J.G.J.G.
25.7k22539
answered Jan 10 at 22:42
J.G.J.G.
25.7k22539
25.7k22539
add a comment |
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3069175%2fcan-i-derive-a-well-known-distribution-from-another-divergence%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
