Mixing
Microservice for notifications
Most of us are familiar with the notifications that pop-up on top right on FB page. Then we ack them, we can still see them with latest first ordering etc.
I have a question on designing such a thing(trying to do some reading also on the side on the arch for this). So at a high level let us say there are microservices running into the system
- S1
- S2
- S3
So let us say I have services like that. I am thinking of creating a new Service say S4 that can receive messages from these services. I am less worried about how these services will talk to S4. There are ways like SQS, kafka, etc.
My main Q is how can S4 do the following
- Maintain these notifications in like what? PSQL?
- Have a column with timestamp?
- Do I need a timeseries DB for it?
- How can I store them based on severity? Fatal, Info, critical?
- How can someone ack a notification?
web-services architecture microservices publish-subscribe distributed-system
edited Jan 2 at 23:40
Oswin Noetzelmann
3,9681529
asked Jan 2 at 21:36
curiousengineercuriousengineer
556926
add a comment |
Most of us are familiar with the notifications that pop-up on top right on FB page. Then we ack them, we can still see them with latest first ordering etc.
I have a question on designing such a thing(trying to do some reading also on the side on the arch for this). So at a high level let us say there are microservices running into the system
- S1
- S2
- S3
So let us say I have services like that. I am thinking of creating a new Service say S4 that can receive messages from these services. I am less worried about how these services will talk to S4. There are ways like SQS, kafka, etc.
My main Q is how can S4 do the following
- Maintain these notifications in like what? PSQL?
- Have a column with timestamp?
- Do I need a timeseries DB for it?
- How can I store them based on severity? Fatal, Info, critical?
- How can someone ack a notification?
web-services architecture microservices publish-subscribe distributed-system
edited Jan 2 at 23:40
Oswin Noetzelmann
3,9681529
asked Jan 2 at 21:36
curiousengineercuriousengineer
556926
Some aspects to consider to help determine what data layer will fit best for "S4" are: required throughput/TPS, how the notifications will be consumed from S4 (and that TPS there as well), persistence requirements, and tolerance for data loss.
– John Fantastico
Jan 2 at 21:42
add a comment |
Most of us are familiar with the notifications that pop-up on top right on FB page. Then we ack them, we can still see them with latest first ordering etc.
I have a question on designing such a thing(trying to do some reading also on the side on the arch for this). So at a high level let us say there are microservices running into the system
- S1
- S2
- S3
So let us say I have services like that. I am thinking of creating a new Service say S4 that can receive messages from these services. I am less worried about how these services will talk to S4. There are ways like SQS, kafka, etc.
My main Q is how can S4 do the following
- Maintain these notifications in like what? PSQL?
- Have a column with timestamp?
- Do I need a timeseries DB for it?
- How can I store them based on severity? Fatal, Info, critical?
- How can someone ack a notification?
web-services architecture microservices publish-subscribe distributed-system
edited Jan 2 at 23:40
Oswin Noetzelmann
3,9681529
asked Jan 2 at 21:36
curiousengineercuriousengineer
556926
Most of us are familiar with the notifications that pop-up on top right on FB page. Then we ack them, we can still see them with latest first ordering etc.
I have a question on designing such a thing(trying to do some reading also on the side on the arch for this). So at a high level let us say there are microservices running into the system
- S1
- S2
- S3
So let us say I have services like that. I am thinking of creating a new Service say S4 that can receive messages from these services. I am less worried about how these services will talk to S4. There are ways like SQS, kafka, etc.
My main Q is how can S4 do the following
- Maintain these notifications in like what? PSQL?
- Have a column with timestamp?
- Do I need a timeseries DB for it?
- How can I store them based on severity? Fatal, Info, critical?
- How can someone ack a notification?
web-services architecture microservices publish-subscribe distributed-system
web-services architecture microservices publish-subscribe distributed-system
edited Jan 2 at 23:40
Oswin Noetzelmann
3,9681529
asked Jan 2 at 21:36
curiousengineercuriousengineer
556926
edited Jan 2 at 23:40
Oswin Noetzelmann
3,9681529
asked Jan 2 at 21:36
curiousengineercuriousengineer
556926
edited Jan 2 at 23:40
Oswin Noetzelmann
3,9681529
edited Jan 2 at 23:40
Oswin Noetzelmann
3,9681529
edited Jan 2 at 23:40
Oswin Noetzelmann
3,9681529
3,9681529
asked Jan 2 at 21:36
curiousengineercuriousengineer
556926
asked Jan 2 at 21:36
curiousengineercuriousengineer
556926
asked Jan 2 at 21:36
curiousengineercuriousengineer
556926
556926
Some aspects to consider to help determine what data layer will fit best for "S4" are: required throughput/TPS, how the notifications will be consumed from S4 (and that TPS there as well), persistence requirements, and tolerance for data loss.
– John Fantastico
Jan 2 at 21:42
add a comment |
Some aspects to consider to help determine what data layer will fit best for "S4" are: required throughput/TPS, how the notifications will be consumed from S4 (and that TPS there as well), persistence requirements, and tolerance for data loss.
– John Fantastico
Jan 2 at 21:42
Some aspects to consider to help determine what data layer will fit best for "S4" are: required throughput/TPS, how the notifications will be consumed from S4 (and that TPS there as well), persistence requirements, and tolerance for data loss.
– John Fantastico
Jan 2 at 21:42
Some aspects to consider to help determine what data layer will fit best for "S4" are: required throughput/TPS, how the notifications will be consumed from S4 (and that TPS there as well), persistence requirements, and tolerance for data loss.
– John Fantastico
Jan 2 at 21:42
add a comment |
1 Answer
1
active
oldest
votes
Below are some thoughts, but I don't think there is a one size fit all approach. It really depends on the rest of your architecture, domain requirements, scalability requirements, etc.
Maintain these notifications in like what? PSQL?
Do you really need to store the notifications additionally to your event queues? It may depend on if your queue design does allow for a just-in-time access based on topics and message content. Many queue systems would allow you to store the message indefinitely/until processed (check out Apache Pulsar). In this case there is a backlog of unacknowledged messages that can be accessed and processed whenever ready (e.g. client confirms message was read). After acknowledgement you can archive or delete the events.
If you decide that doesn't work and you need an additional database, the usual suspects come into play: Key-Value-Stores, such as MongoDB, or Relational DB, such as CockroachDB.
Have a column with timestamp?
Most queue systems would have a time stamp recorded by default. Sometimes the queue order could be enough. Depending on your query requirements a schema-less format (e.g. json or blob) for the message content and some explicit identifiers for finding the messages (by user etc) would be required as a minimum in most cases.
Do I need a timeseries DB for it?
Possibly useful if you need to track a large number of values changing over time, e.g. IOT related sensor data such as a temperature measurement. If you just have a handful of facebook-style notifications per user that seems not like a good fit.
How can I store them based on severity? Fatal, Info, critical?
Here you could utilize separation by topics in event queues or when using a separate DB it depends again if you use a schema or schema-less DB.
How can someone ack a notification?
As previously mentioned, modern event queue systems have that built in, or if using a separate DB you'll have to add that to the schema (or use a schema-less DB). In either case you need to define or implement the behavior for message retention after acknowledgment, e.g. whether to delete or archive.
answered Jan 2 at 23:13
Oswin NoetzelmannOswin Noetzelmann
3,9681529
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54013470%2fmicroservice-for-notifications%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Below are some thoughts, but I don't think there is a one size fit all approach. It really depends on the rest of your architecture, domain requirements, scalability requirements, etc.
Maintain these notifications in like what? PSQL?
Do you really need to store the notifications additionally to your event queues? It may depend on if your queue design does allow for a just-in-time access based on topics and message content. Many queue systems would allow you to store the message indefinitely/until processed (check out Apache Pulsar). In this case there is a backlog of unacknowledged messages that can be accessed and processed whenever ready (e.g. client confirms message was read). After acknowledgement you can archive or delete the events.
If you decide that doesn't work and you need an additional database, the usual suspects come into play: Key-Value-Stores, such as MongoDB, or Relational DB, such as CockroachDB.
Have a column with timestamp?
Most queue systems would have a time stamp recorded by default. Sometimes the queue order could be enough. Depending on your query requirements a schema-less format (e.g. json or blob) for the message content and some explicit identifiers for finding the messages (by user etc) would be required as a minimum in most cases.
Do I need a timeseries DB for it?
Possibly useful if you need to track a large number of values changing over time, e.g. IOT related sensor data such as a temperature measurement. If you just have a handful of facebook-style notifications per user that seems not like a good fit.
How can I store them based on severity? Fatal, Info, critical?
Here you could utilize separation by topics in event queues or when using a separate DB it depends again if you use a schema or schema-less DB.
How can someone ack a notification?
As previously mentioned, modern event queue systems have that built in, or if using a separate DB you'll have to add that to the schema (or use a schema-less DB). In either case you need to define or implement the behavior for message retention after acknowledgment, e.g. whether to delete or archive.
answered Jan 2 at 23:13
Oswin NoetzelmannOswin Noetzelmann
3,9681529
add a comment |
Below are some thoughts, but I don't think there is a one size fit all approach. It really depends on the rest of your architecture, domain requirements, scalability requirements, etc.
Maintain these notifications in like what? PSQL?
Do you really need to store the notifications additionally to your event queues? It may depend on if your queue design does allow for a just-in-time access based on topics and message content. Many queue systems would allow you to store the message indefinitely/until processed (check out Apache Pulsar). In this case there is a backlog of unacknowledged messages that can be accessed and processed whenever ready (e.g. client confirms message was read). After acknowledgement you can archive or delete the events.
If you decide that doesn't work and you need an additional database, the usual suspects come into play: Key-Value-Stores, such as MongoDB, or Relational DB, such as CockroachDB.
Have a column with timestamp?
Most queue systems would have a time stamp recorded by default. Sometimes the queue order could be enough. Depending on your query requirements a schema-less format (e.g. json or blob) for the message content and some explicit identifiers for finding the messages (by user etc) would be required as a minimum in most cases.
Do I need a timeseries DB for it?
Possibly useful if you need to track a large number of values changing over time, e.g. IOT related sensor data such as a temperature measurement. If you just have a handful of facebook-style notifications per user that seems not like a good fit.
How can I store them based on severity? Fatal, Info, critical?
Here you could utilize separation by topics in event queues or when using a separate DB it depends again if you use a schema or schema-less DB.
How can someone ack a notification?
As previously mentioned, modern event queue systems have that built in, or if using a separate DB you'll have to add that to the schema (or use a schema-less DB). In either case you need to define or implement the behavior for message retention after acknowledgment, e.g. whether to delete or archive.
answered Jan 2 at 23:13
Oswin NoetzelmannOswin Noetzelmann
3,9681529
add a comment |
Below are some thoughts, but I don't think there is a one size fit all approach. It really depends on the rest of your architecture, domain requirements, scalability requirements, etc.
Maintain these notifications in like what? PSQL?
Do you really need to store the notifications additionally to your event queues? It may depend on if your queue design does allow for a just-in-time access based on topics and message content. Many queue systems would allow you to store the message indefinitely/until processed (check out Apache Pulsar). In this case there is a backlog of unacknowledged messages that can be accessed and processed whenever ready (e.g. client confirms message was read). After acknowledgement you can archive or delete the events.
If you decide that doesn't work and you need an additional database, the usual suspects come into play: Key-Value-Stores, such as MongoDB, or Relational DB, such as CockroachDB.
Have a column with timestamp?
Most queue systems would have a time stamp recorded by default. Sometimes the queue order could be enough. Depending on your query requirements a schema-less format (e.g. json or blob) for the message content and some explicit identifiers for finding the messages (by user etc) would be required as a minimum in most cases.
Do I need a timeseries DB for it?
Possibly useful if you need to track a large number of values changing over time, e.g. IOT related sensor data such as a temperature measurement. If you just have a handful of facebook-style notifications per user that seems not like a good fit.
How can I store them based on severity? Fatal, Info, critical?
Here you could utilize separation by topics in event queues or when using a separate DB it depends again if you use a schema or schema-less DB.
How can someone ack a notification?
As previously mentioned, modern event queue systems have that built in, or if using a separate DB you'll have to add that to the schema (or use a schema-less DB). In either case you need to define or implement the behavior for message retention after acknowledgment, e.g. whether to delete or archive.
answered Jan 2 at 23:13
Oswin NoetzelmannOswin Noetzelmann
3,9681529
Below are some thoughts, but I don't think there is a one size fit all approach. It really depends on the rest of your architecture, domain requirements, scalability requirements, etc.
Maintain these notifications in like what? PSQL?
Do you really need to store the notifications additionally to your event queues? It may depend on if your queue design does allow for a just-in-time access based on topics and message content. Many queue systems would allow you to store the message indefinitely/until processed (check out Apache Pulsar). In this case there is a backlog of unacknowledged messages that can be accessed and processed whenever ready (e.g. client confirms message was read). After acknowledgement you can archive or delete the events.
If you decide that doesn't work and you need an additional database, the usual suspects come into play: Key-Value-Stores, such as MongoDB, or Relational DB, such as CockroachDB.
Have a column with timestamp?
Most queue systems would have a time stamp recorded by default. Sometimes the queue order could be enough. Depending on your query requirements a schema-less format (e.g. json or blob) for the message content and some explicit identifiers for finding the messages (by user etc) would be required as a minimum in most cases.
Do I need a timeseries DB for it?
Possibly useful if you need to track a large number of values changing over time, e.g. IOT related sensor data such as a temperature measurement. If you just have a handful of facebook-style notifications per user that seems not like a good fit.
How can I store them based on severity? Fatal, Info, critical?
Here you could utilize separation by topics in event queues or when using a separate DB it depends again if you use a schema or schema-less DB.
How can someone ack a notification?
As previously mentioned, modern event queue systems have that built in, or if using a separate DB you'll have to add that to the schema (or use a schema-less DB). In either case you need to define or implement the behavior for message retention after acknowledgment, e.g. whether to delete or archive.
answered Jan 2 at 23:13
Oswin NoetzelmannOswin Noetzelmann
3,9681529
answered Jan 2 at 23:13
Oswin NoetzelmannOswin Noetzelmann
3,9681529
answered Jan 2 at 23:13
Oswin NoetzelmannOswin Noetzelmann
3,9681529
answered Jan 2 at 23:13
Oswin NoetzelmannOswin Noetzelmann
3,9681529
3,9681529
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54013470%2fmicroservice-for-notifications%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Some aspects to consider to help determine what data layer will fit best for "S4" are: required throughput/TPS, how the notifications will be consumed from S4 (and that TPS there as well), persistence requirements, and tolerance for data loss.
– John Fantastico
Jan 2 at 21:42