Mixing
Modelling conditional distribution based on multiple variables of various types?
$begingroup$
I have a looking basic statistics problem: basing on a large sample of multivariate data, model conditional probability distribution (continuous) of one variable based on the remaining ones: a few dozens of various types (continuous, discrete, binary) - I have some own approach (below), but need to compare it with some standard methods and honestly don't know any appropiate?
The only standard approach able to model such complex densities I know is KDE, but it doesn't seem useful in such high dimensional situation with discrete variables (?)
Specifically, I recently got dataset for 37000 households with declared income and a few dozens of other variables of various types: continuous, discrete, binary. The task is to automatically (unsupervised) evaluate credibility of declared income based on the remaining variables: evaluate if it agrees with statistics of the sample.
Approach I have used ( https://arxiv.org/pdf/1812.08040 , general slides):
Normalize the income to uniform distribution on [0,1] using empirical distribution function like in copula theory. Thanks of it, modeled density of conditional distribution of this variable seems a proper way to evaluate credibility (?) Here are examples of pairwise dependencies of such normalized variables - would be rho=1 if independent, inhomogeneity allows to predict different conditional distribution of income based on the second variable (e.g. for 70 year old, extreme value is less credible):
To combine predictions from multiple variables, I just used linear regression: of cumulant-like polynomial coefficients of predicted variable, as a linear combinations of features of the remaining variables (e.g. their contribution to j-th moment) - works nicely, here are some predicted densities:
I need to compare it with some standard methods - which one can I use?
statistics probability-distributions statistical-inference conditional-probability estimation
asked Jan 16 at 9:41
Jarek DudaJarek Duda
506312
$endgroup$
add a comment |
$begingroup$
I have a looking basic statistics problem: basing on a large sample of multivariate data, model conditional probability distribution (continuous) of one variable based on the remaining ones: a few dozens of various types (continuous, discrete, binary) - I have some own approach (below), but need to compare it with some standard methods and honestly don't know any appropiate?
The only standard approach able to model such complex densities I know is KDE, but it doesn't seem useful in such high dimensional situation with discrete variables (?)
Specifically, I recently got dataset for 37000 households with declared income and a few dozens of other variables of various types: continuous, discrete, binary. The task is to automatically (unsupervised) evaluate credibility of declared income based on the remaining variables: evaluate if it agrees with statistics of the sample.
Approach I have used ( https://arxiv.org/pdf/1812.08040 , general slides):
Normalize the income to uniform distribution on [0,1] using empirical distribution function like in copula theory. Thanks of it, modeled density of conditional distribution of this variable seems a proper way to evaluate credibility (?) Here are examples of pairwise dependencies of such normalized variables - would be rho=1 if independent, inhomogeneity allows to predict different conditional distribution of income based on the second variable (e.g. for 70 year old, extreme value is less credible):
To combine predictions from multiple variables, I just used linear regression: of cumulant-like polynomial coefficients of predicted variable, as a linear combinations of features of the remaining variables (e.g. their contribution to j-th moment) - works nicely, here are some predicted densities:
I need to compare it with some standard methods - which one can I use?
statistics probability-distributions statistical-inference conditional-probability estimation
asked Jan 16 at 9:41
Jarek DudaJarek Duda
506312
$endgroup$
add a comment |
$begingroup$
I have a looking basic statistics problem: basing on a large sample of multivariate data, model conditional probability distribution (continuous) of one variable based on the remaining ones: a few dozens of various types (continuous, discrete, binary) - I have some own approach (below), but need to compare it with some standard methods and honestly don't know any appropiate?
The only standard approach able to model such complex densities I know is KDE, but it doesn't seem useful in such high dimensional situation with discrete variables (?)
Specifically, I recently got dataset for 37000 households with declared income and a few dozens of other variables of various types: continuous, discrete, binary. The task is to automatically (unsupervised) evaluate credibility of declared income based on the remaining variables: evaluate if it agrees with statistics of the sample.
Approach I have used ( https://arxiv.org/pdf/1812.08040 , general slides):
Normalize the income to uniform distribution on [0,1] using empirical distribution function like in copula theory. Thanks of it, modeled density of conditional distribution of this variable seems a proper way to evaluate credibility (?) Here are examples of pairwise dependencies of such normalized variables - would be rho=1 if independent, inhomogeneity allows to predict different conditional distribution of income based on the second variable (e.g. for 70 year old, extreme value is less credible):
To combine predictions from multiple variables, I just used linear regression: of cumulant-like polynomial coefficients of predicted variable, as a linear combinations of features of the remaining variables (e.g. their contribution to j-th moment) - works nicely, here are some predicted densities:
I need to compare it with some standard methods - which one can I use?
statistics probability-distributions statistical-inference conditional-probability estimation
asked Jan 16 at 9:41
Jarek DudaJarek Duda
506312
$endgroup$
I have a looking basic statistics problem: basing on a large sample of multivariate data, model conditional probability distribution (continuous) of one variable based on the remaining ones: a few dozens of various types (continuous, discrete, binary) - I have some own approach (below), but need to compare it with some standard methods and honestly don't know any appropiate?
The only standard approach able to model such complex densities I know is KDE, but it doesn't seem useful in such high dimensional situation with discrete variables (?)
Specifically, I recently got dataset for 37000 households with declared income and a few dozens of other variables of various types: continuous, discrete, binary. The task is to automatically (unsupervised) evaluate credibility of declared income based on the remaining variables: evaluate if it agrees with statistics of the sample.
Approach I have used ( https://arxiv.org/pdf/1812.08040 , general slides):
Normalize the income to uniform distribution on [0,1] using empirical distribution function like in copula theory. Thanks of it, modeled density of conditional distribution of this variable seems a proper way to evaluate credibility (?) Here are examples of pairwise dependencies of such normalized variables - would be rho=1 if independent, inhomogeneity allows to predict different conditional distribution of income based on the second variable (e.g. for 70 year old, extreme value is less credible):
To combine predictions from multiple variables, I just used linear regression: of cumulant-like polynomial coefficients of predicted variable, as a linear combinations of features of the remaining variables (e.g. their contribution to j-th moment) - works nicely, here are some predicted densities:
I need to compare it with some standard methods - which one can I use?
statistics probability-distributions statistical-inference conditional-probability estimation
statistics probability-distributions statistical-inference conditional-probability estimation
asked Jan 16 at 9:41
Jarek DudaJarek Duda
506312
asked Jan 16 at 9:41
Jarek DudaJarek Duda
506312
asked Jan 16 at 9:41
Jarek DudaJarek Duda
506312
asked Jan 16 at 9:41
Jarek DudaJarek Duda
506312
asked Jan 16 at 9:41
Jarek DudaJarek Duda
506312
506312
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3075531%2fmodelling-conditional-distribution-based-on-multiple-variables-of-various-types%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3075531%2fmodelling-conditional-distribution-based-on-multiple-variables-of-various-types%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
