Mixing
Handle high cardinality for one column in time-series database
I have a very high cardinality time-series database. Suppose, I have 4 columns in my time-series database (A,B,C and D) whose individual cardinalities are (10, 100, 50, 10,000,000). So, in total I have a database of (10*100*50*10,000,000) cardinality. I want to know following questions:
- Which alerting system should I use to monitor high cardinality
(say 5 million cardinality in last one hour of data) database. - What is the best way to handle if 1 column in time-series database
is of very high cardinality?
time-series monitor prometheus
edited Nov 20 '18 at 22:10
Jan Garaj
2,602618
asked Nov 19 '18 at 18:34
Utkarsh SrivastavUtkarsh Srivastav
69551943
add a comment |
I have a very high cardinality time-series database. Suppose, I have 4 columns in my time-series database (A,B,C and D) whose individual cardinalities are (10, 100, 50, 10,000,000). So, in total I have a database of (10*100*50*10,000,000) cardinality. I want to know following questions:
- Which alerting system should I use to monitor high cardinality
(say 5 million cardinality in last one hour of data) database. - What is the best way to handle if 1 column in time-series database
is of very high cardinality?
time-series monitor prometheus
edited Nov 20 '18 at 22:10
Jan Garaj
2,602618
asked Nov 19 '18 at 18:34
Utkarsh SrivastavUtkarsh Srivastav
69551943
add a comment |
I have a very high cardinality time-series database. Suppose, I have 4 columns in my time-series database (A,B,C and D) whose individual cardinalities are (10, 100, 50, 10,000,000). So, in total I have a database of (10*100*50*10,000,000) cardinality. I want to know following questions:
- Which alerting system should I use to monitor high cardinality
(say 5 million cardinality in last one hour of data) database. - What is the best way to handle if 1 column in time-series database
is of very high cardinality?
time-series monitor prometheus
edited Nov 20 '18 at 22:10
Jan Garaj
2,602618
asked Nov 19 '18 at 18:34
Utkarsh SrivastavUtkarsh Srivastav
69551943
I have a very high cardinality time-series database. Suppose, I have 4 columns in my time-series database (A,B,C and D) whose individual cardinalities are (10, 100, 50, 10,000,000). So, in total I have a database of (10*100*50*10,000,000) cardinality. I want to know following questions:
- Which alerting system should I use to monitor high cardinality
(say 5 million cardinality in last one hour of data) database. - What is the best way to handle if 1 column in time-series database
is of very high cardinality?
time-series monitor prometheus
time-series monitor prometheus
edited Nov 20 '18 at 22:10
Jan Garaj
2,602618
asked Nov 19 '18 at 18:34
Utkarsh SrivastavUtkarsh Srivastav
69551943
edited Nov 20 '18 at 22:10
Jan Garaj
2,602618
asked Nov 19 '18 at 18:34
Utkarsh SrivastavUtkarsh Srivastav
69551943
edited Nov 20 '18 at 22:10
Jan Garaj
2,602618
edited Nov 20 '18 at 22:10
Jan Garaj
2,602618
edited Nov 20 '18 at 22:10
Jan Garaj
2,602618
2,602618
asked Nov 19 '18 at 18:34
Utkarsh SrivastavUtkarsh Srivastav
69551943
asked Nov 19 '18 at 18:34
Utkarsh SrivastavUtkarsh Srivastav
69551943
asked Nov 19 '18 at 18:34
Utkarsh SrivastavUtkarsh Srivastav
69551943
69551943
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
I'm assuming you want use some sort of monitoring system where upon some events the system is triggered to alarm about a certain service right? like a anomaly detection system.
So, my question to you is, are you looking a monitoring tool, just to have reports overs the features, or use the time-series for machine learning for example?
I'll answer this as if it was oriented to Machine learning. I'm sorry if this is not your intention:
==> In ML features with high cardinality are usually handled through bining if you need usem as dummy variables. In orther words, for each level of the feature a new binary column is created. (Example: http code: 200, 200, 201, 404, 409, 500 ==> 2xx, 3xx, 4xx).
==> However, if you are using tree-based algorithms to handle high cardinality, no need for dummy variables to handle de cardinality.
Many more approaches can be used, but i need to know if this is what you are looking for in order for me to deepen the answer.
answered Dec 11 '18 at 18:08
Pedro Tourais PereiraPedro Tourais Pereira
112
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53380716%2fhandle-high-cardinality-for-one-column-in-time-series-database%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I'm assuming you want use some sort of monitoring system where upon some events the system is triggered to alarm about a certain service right? like a anomaly detection system.
So, my question to you is, are you looking a monitoring tool, just to have reports overs the features, or use the time-series for machine learning for example?
I'll answer this as if it was oriented to Machine learning. I'm sorry if this is not your intention:
==> In ML features with high cardinality are usually handled through bining if you need usem as dummy variables. In orther words, for each level of the feature a new binary column is created. (Example: http code: 200, 200, 201, 404, 409, 500 ==> 2xx, 3xx, 4xx).
==> However, if you are using tree-based algorithms to handle high cardinality, no need for dummy variables to handle de cardinality.
Many more approaches can be used, but i need to know if this is what you are looking for in order for me to deepen the answer.
answered Dec 11 '18 at 18:08
Pedro Tourais PereiraPedro Tourais Pereira
112
add a comment |
I'm assuming you want use some sort of monitoring system where upon some events the system is triggered to alarm about a certain service right? like a anomaly detection system.
So, my question to you is, are you looking a monitoring tool, just to have reports overs the features, or use the time-series for machine learning for example?
I'll answer this as if it was oriented to Machine learning. I'm sorry if this is not your intention:
==> In ML features with high cardinality are usually handled through bining if you need usem as dummy variables. In orther words, for each level of the feature a new binary column is created. (Example: http code: 200, 200, 201, 404, 409, 500 ==> 2xx, 3xx, 4xx).
==> However, if you are using tree-based algorithms to handle high cardinality, no need for dummy variables to handle de cardinality.
Many more approaches can be used, but i need to know if this is what you are looking for in order for me to deepen the answer.
answered Dec 11 '18 at 18:08
Pedro Tourais PereiraPedro Tourais Pereira
112
add a comment |
I'm assuming you want use some sort of monitoring system where upon some events the system is triggered to alarm about a certain service right? like a anomaly detection system.
So, my question to you is, are you looking a monitoring tool, just to have reports overs the features, or use the time-series for machine learning for example?
I'll answer this as if it was oriented to Machine learning. I'm sorry if this is not your intention:
==> In ML features with high cardinality are usually handled through bining if you need usem as dummy variables. In orther words, for each level of the feature a new binary column is created. (Example: http code: 200, 200, 201, 404, 409, 500 ==> 2xx, 3xx, 4xx).
==> However, if you are using tree-based algorithms to handle high cardinality, no need for dummy variables to handle de cardinality.
Many more approaches can be used, but i need to know if this is what you are looking for in order for me to deepen the answer.
answered Dec 11 '18 at 18:08
Pedro Tourais PereiraPedro Tourais Pereira
112
I'm assuming you want use some sort of monitoring system where upon some events the system is triggered to alarm about a certain service right? like a anomaly detection system.
So, my question to you is, are you looking a monitoring tool, just to have reports overs the features, or use the time-series for machine learning for example?
I'll answer this as if it was oriented to Machine learning. I'm sorry if this is not your intention:
==> In ML features with high cardinality are usually handled through bining if you need usem as dummy variables. In orther words, for each level of the feature a new binary column is created. (Example: http code: 200, 200, 201, 404, 409, 500 ==> 2xx, 3xx, 4xx).
==> However, if you are using tree-based algorithms to handle high cardinality, no need for dummy variables to handle de cardinality.
Many more approaches can be used, but i need to know if this is what you are looking for in order for me to deepen the answer.
answered Dec 11 '18 at 18:08
Pedro Tourais PereiraPedro Tourais Pereira
112
answered Dec 11 '18 at 18:08
Pedro Tourais PereiraPedro Tourais Pereira
112
answered Dec 11 '18 at 18:08
Pedro Tourais PereiraPedro Tourais Pereira
112
answered Dec 11 '18 at 18:08
Pedro Tourais PereiraPedro Tourais Pereira
112
112
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53380716%2fhandle-high-cardinality-for-one-column-in-time-series-database%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
