Mixing
Using Step Function for Synchronizing Primay Data Source with Secondary Data Sources and trigger Actions...
I have an application which acts as Data Layer and is responsible for performing all the CRUD operation on the database. Due to availability issue, we maintain Redis cache over it, which will ultimately act as the secondary data source. Hitting primary data sources frequently leads to throttling and latency issues.
Certain actions also gets triggered on the basis of updates published to the DataBase.
I am trying to abstract out the secondary data source updates and actions triggered on the basis of these updates from the data access layer to serverless architecture.
Is the step-function best suited for the mentioned usecase? Or I should think of different architecture?
Current Architecture
design-patterns redis architecture aws-lambda aws-step-functions
asked Jan 2 at 9:53
Kajol KumariKajol Kumari
11
add a comment |
I have an application which acts as Data Layer and is responsible for performing all the CRUD operation on the database. Due to availability issue, we maintain Redis cache over it, which will ultimately act as the secondary data source. Hitting primary data sources frequently leads to throttling and latency issues.
Certain actions also gets triggered on the basis of updates published to the DataBase.
I am trying to abstract out the secondary data source updates and actions triggered on the basis of these updates from the data access layer to serverless architecture.
Is the step-function best suited for the mentioned usecase? Or I should think of different architecture?
Current Architecture
design-patterns redis architecture aws-lambda aws-step-functions
asked Jan 2 at 9:53
Kajol KumariKajol Kumari
11
Can you give us more information about the primary Data Source? What kind of Database is it? - If you're lucky enough to use DynamoDB, DynamoDB streams and/or DAX might me a suitable option.
– Maurice
Jan 2 at 11:46
Primay Data Source is not AWS based and is internal to my company. @Maurice I am planning to pass on the changes done to Primary Data Source to Step Function using SQS-Lambda
– Kajol Kumari
Jan 2 at 12:07
I'm still not sure how your architecture currently looks... - A solution for this might be some kind of write-through-caching behavior, which means your application would write all changes to Redis and the DB, which means all new updates are always in present in Redis - although this may waste memory if changed data is infrequently accessed. If you can give us more information about your use case we might be able to provide better solutions - if possible a small diagram would probably help.
– Maurice
Jan 2 at 12:32
@Maurice I have updated the question with the current architecture as well as the architecture I am thinking of.
– Kajol Kumari
Jan 5 at 9:57
add a comment |
I have an application which acts as Data Layer and is responsible for performing all the CRUD operation on the database. Due to availability issue, we maintain Redis cache over it, which will ultimately act as the secondary data source. Hitting primary data sources frequently leads to throttling and latency issues.
Certain actions also gets triggered on the basis of updates published to the DataBase.
I am trying to abstract out the secondary data source updates and actions triggered on the basis of these updates from the data access layer to serverless architecture.
Is the step-function best suited for the mentioned usecase? Or I should think of different architecture?
Current Architecture
design-patterns redis architecture aws-lambda aws-step-functions
asked Jan 2 at 9:53
Kajol KumariKajol Kumari
11
I have an application which acts as Data Layer and is responsible for performing all the CRUD operation on the database. Due to availability issue, we maintain Redis cache over it, which will ultimately act as the secondary data source. Hitting primary data sources frequently leads to throttling and latency issues.
Certain actions also gets triggered on the basis of updates published to the DataBase.
I am trying to abstract out the secondary data source updates and actions triggered on the basis of these updates from the data access layer to serverless architecture.
Is the step-function best suited for the mentioned usecase? Or I should think of different architecture?
Current Architecture
design-patterns redis architecture aws-lambda aws-step-functions
design-patterns redis architecture aws-lambda aws-step-functions
asked Jan 2 at 9:53
Kajol KumariKajol Kumari
11
asked Jan 2 at 9:53
Kajol KumariKajol Kumari
11
edited Jan 5 at 9:56
Kajol Kumari
asked Jan 2 at 9:53
Kajol KumariKajol Kumari
11
asked Jan 2 at 9:53
Kajol KumariKajol Kumari
11
asked Jan 2 at 9:53
Kajol KumariKajol Kumari
11
11
Can you give us more information about the primary Data Source? What kind of Database is it? - If you're lucky enough to use DynamoDB, DynamoDB streams and/or DAX might me a suitable option.
– Maurice
Jan 2 at 11:46
Primay Data Source is not AWS based and is internal to my company. @Maurice I am planning to pass on the changes done to Primary Data Source to Step Function using SQS-Lambda
– Kajol Kumari
Jan 2 at 12:07
I'm still not sure how your architecture currently looks... - A solution for this might be some kind of write-through-caching behavior, which means your application would write all changes to Redis and the DB, which means all new updates are always in present in Redis - although this may waste memory if changed data is infrequently accessed. If you can give us more information about your use case we might be able to provide better solutions - if possible a small diagram would probably help.
– Maurice
Jan 2 at 12:32
@Maurice I have updated the question with the current architecture as well as the architecture I am thinking of.
– Kajol Kumari
Jan 5 at 9:57
add a comment |
Can you give us more information about the primary Data Source? What kind of Database is it? - If you're lucky enough to use DynamoDB, DynamoDB streams and/or DAX might me a suitable option.
– Maurice
Jan 2 at 11:46
Primay Data Source is not AWS based and is internal to my company. @Maurice I am planning to pass on the changes done to Primary Data Source to Step Function using SQS-Lambda
– Kajol Kumari
Jan 2 at 12:07
I'm still not sure how your architecture currently looks... - A solution for this might be some kind of write-through-caching behavior, which means your application would write all changes to Redis and the DB, which means all new updates are always in present in Redis - although this may waste memory if changed data is infrequently accessed. If you can give us more information about your use case we might be able to provide better solutions - if possible a small diagram would probably help.
– Maurice
Jan 2 at 12:32
@Maurice I have updated the question with the current architecture as well as the architecture I am thinking of.
– Kajol Kumari
Jan 5 at 9:57
Can you give us more information about the primary Data Source? What kind of Database is it? - If you're lucky enough to use DynamoDB, DynamoDB streams and/or DAX might me a suitable option.
– Maurice
Jan 2 at 11:46
Can you give us more information about the primary Data Source? What kind of Database is it? - If you're lucky enough to use DynamoDB, DynamoDB streams and/or DAX might me a suitable option.
– Maurice
Jan 2 at 11:46
Primay Data Source is not AWS based and is internal to my company. @Maurice I am planning to pass on the changes done to Primary Data Source to Step Function using SQS-Lambda
– Kajol Kumari
Jan 2 at 12:07
Primay Data Source is not AWS based and is internal to my company. @Maurice I am planning to pass on the changes done to Primary Data Source to Step Function using SQS-Lambda
– Kajol Kumari
Jan 2 at 12:07
I'm still not sure how your architecture currently looks... - A solution for this might be some kind of write-through-caching behavior, which means your application would write all changes to Redis and the DB, which means all new updates are always in present in Redis - although this may waste memory if changed data is infrequently accessed. If you can give us more information about your use case we might be able to provide better solutions - if possible a small diagram would probably help.
– Maurice
Jan 2 at 12:32
I'm still not sure how your architecture currently looks... - A solution for this might be some kind of write-through-caching behavior, which means your application would write all changes to Redis and the DB, which means all new updates are always in present in Redis - although this may waste memory if changed data is infrequently accessed. If you can give us more information about your use case we might be able to provide better solutions - if possible a small diagram would probably help.
– Maurice
Jan 2 at 12:32
@Maurice I have updated the question with the current architecture as well as the architecture I am thinking of.
– Kajol Kumari
Jan 5 at 9:57
@Maurice I have updated the question with the current architecture as well as the architecture I am thinking of.
– Kajol Kumari
Jan 5 at 9:57
add a comment |
1 Answer
1
active
oldest
votes
Based on your proposed architecture, which looks as follows, I don't think step functions are necessary.
I'd use SNS to fan out updates to multiple SQS queues:
- Queue #1 triggers a lambda function, that updates your data in Redis
- Queue #2 is used for Client-Updates (whatever this means in your case)
I don't see the reason to use Step Functions here as SQS makes sure, that each message gets processed at least once.
+----------+
+---->Primary DB|
| +----------+ +----------------+ +------------+ +-----+
+ | | | | | |
Change +-----> SQS-Queue #1 +------> Lambda +---->Redis|
+ | | | | | | |
| | +----------------+ +------------+ +-----+
| +-----------+
| | |
+----> SNS |
| |
+-----------+
| +----------------+
| | |
+-----> SQS-Queue #2 | <------ Clients
| |
+----------------+
I think SNS might be more suited to push updates to your clients as well, as SQS is pull-based.
answered Jan 5 at 20:42
MauriceMaurice
1,0082824
For the status tracking of each updates and retrying failed execution I thought of using Step Function.
– Kajol Kumari
Jan 6 at 8:53
You could do that and it may be suitable depending on how complex your data structures are. But it may be sufficient to use the Visibility Timeout of SQS to ensure, that messages get processed again, if the first attempt fails. To be clear: Step functions will work, but if you should use them depends on what you want to optimize for: Speed, Reliability, Cost, Maintainability,...
– Maurice
Jan 6 at 9:33
Speed, Reliability are what is expected out of the architecture and the messages put to the queue needs to be transformed into some structure before dumping to SQS or Redis. Thats why thought of using Step Function.
– Kajol Kumari
Jan 6 at 15:38
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54004230%2fusing-step-function-for-synchronizing-primay-data-source-with-secondary-data-sou%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Based on your proposed architecture, which looks as follows, I don't think step functions are necessary.
I'd use SNS to fan out updates to multiple SQS queues:
- Queue #1 triggers a lambda function, that updates your data in Redis
- Queue #2 is used for Client-Updates (whatever this means in your case)
I don't see the reason to use Step Functions here as SQS makes sure, that each message gets processed at least once.
+----------+
+---->Primary DB|
| +----------+ +----------------+ +------------+ +-----+
+ | | | | | |
Change +-----> SQS-Queue #1 +------> Lambda +---->Redis|
+ | | | | | | |
| | +----------------+ +------------+ +-----+
| +-----------+
| | |
+----> SNS |
| |
+-----------+
| +----------------+
| | |
+-----> SQS-Queue #2 | <------ Clients
| |
+----------------+
I think SNS might be more suited to push updates to your clients as well, as SQS is pull-based.
answered Jan 5 at 20:42
MauriceMaurice
1,0082824
For the status tracking of each updates and retrying failed execution I thought of using Step Function.
– Kajol Kumari
Jan 6 at 8:53
You could do that and it may be suitable depending on how complex your data structures are. But it may be sufficient to use the Visibility Timeout of SQS to ensure, that messages get processed again, if the first attempt fails. To be clear: Step functions will work, but if you should use them depends on what you want to optimize for: Speed, Reliability, Cost, Maintainability,...
– Maurice
Jan 6 at 9:33
Speed, Reliability are what is expected out of the architecture and the messages put to the queue needs to be transformed into some structure before dumping to SQS or Redis. Thats why thought of using Step Function.
– Kajol Kumari
Jan 6 at 15:38
add a comment |
Based on your proposed architecture, which looks as follows, I don't think step functions are necessary.
I'd use SNS to fan out updates to multiple SQS queues:
- Queue #1 triggers a lambda function, that updates your data in Redis
- Queue #2 is used for Client-Updates (whatever this means in your case)
I don't see the reason to use Step Functions here as SQS makes sure, that each message gets processed at least once.
+----------+
+---->Primary DB|
| +----------+ +----------------+ +------------+ +-----+
+ | | | | | |
Change +-----> SQS-Queue #1 +------> Lambda +---->Redis|
+ | | | | | | |
| | +----------------+ +------------+ +-----+
| +-----------+
| | |
+----> SNS |
| |
+-----------+
| +----------------+
| | |
+-----> SQS-Queue #2 | <------ Clients
| |
+----------------+
I think SNS might be more suited to push updates to your clients as well, as SQS is pull-based.
answered Jan 5 at 20:42
MauriceMaurice
1,0082824
For the status tracking of each updates and retrying failed execution I thought of using Step Function.
– Kajol Kumari
Jan 6 at 8:53
You could do that and it may be suitable depending on how complex your data structures are. But it may be sufficient to use the Visibility Timeout of SQS to ensure, that messages get processed again, if the first attempt fails. To be clear: Step functions will work, but if you should use them depends on what you want to optimize for: Speed, Reliability, Cost, Maintainability,...
– Maurice
Jan 6 at 9:33
Speed, Reliability are what is expected out of the architecture and the messages put to the queue needs to be transformed into some structure before dumping to SQS or Redis. Thats why thought of using Step Function.
– Kajol Kumari
Jan 6 at 15:38
add a comment |
Based on your proposed architecture, which looks as follows, I don't think step functions are necessary.
I'd use SNS to fan out updates to multiple SQS queues:
- Queue #1 triggers a lambda function, that updates your data in Redis
- Queue #2 is used for Client-Updates (whatever this means in your case)
I don't see the reason to use Step Functions here as SQS makes sure, that each message gets processed at least once.
+----------+
+---->Primary DB|
| +----------+ +----------------+ +------------+ +-----+
+ | | | | | |
Change +-----> SQS-Queue #1 +------> Lambda +---->Redis|
+ | | | | | | |
| | +----------------+ +------------+ +-----+
| +-----------+
| | |
+----> SNS |
| |
+-----------+
| +----------------+
| | |
+-----> SQS-Queue #2 | <------ Clients
| |
+----------------+
I think SNS might be more suited to push updates to your clients as well, as SQS is pull-based.
answered Jan 5 at 20:42
MauriceMaurice
1,0082824
Based on your proposed architecture, which looks as follows, I don't think step functions are necessary.
I'd use SNS to fan out updates to multiple SQS queues:
- Queue #1 triggers a lambda function, that updates your data in Redis
- Queue #2 is used for Client-Updates (whatever this means in your case)
I don't see the reason to use Step Functions here as SQS makes sure, that each message gets processed at least once.
+----------+
+---->Primary DB|
| +----------+ +----------------+ +------------+ +-----+
+ | | | | | |
Change +-----> SQS-Queue #1 +------> Lambda +---->Redis|
+ | | | | | | |
| | +----------------+ +------------+ +-----+
| +-----------+
| | |
+----> SNS |
| |
+-----------+
| +----------------+
| | |
+-----> SQS-Queue #2 | <------ Clients
| |
+----------------+
I think SNS might be more suited to push updates to your clients as well, as SQS is pull-based.
answered Jan 5 at 20:42
MauriceMaurice
1,0082824
answered Jan 5 at 20:42
MauriceMaurice
1,0082824
answered Jan 5 at 20:42
MauriceMaurice
1,0082824
answered Jan 5 at 20:42
MauriceMaurice
1,0082824
1,0082824
For the status tracking of each updates and retrying failed execution I thought of using Step Function.
– Kajol Kumari
Jan 6 at 8:53
You could do that and it may be suitable depending on how complex your data structures are. But it may be sufficient to use the Visibility Timeout of SQS to ensure, that messages get processed again, if the first attempt fails. To be clear: Step functions will work, but if you should use them depends on what you want to optimize for: Speed, Reliability, Cost, Maintainability,...
– Maurice
Jan 6 at 9:33
Speed, Reliability are what is expected out of the architecture and the messages put to the queue needs to be transformed into some structure before dumping to SQS or Redis. Thats why thought of using Step Function.
– Kajol Kumari
Jan 6 at 15:38
add a comment |
For the status tracking of each updates and retrying failed execution I thought of using Step Function.
– Kajol Kumari
Jan 6 at 8:53
You could do that and it may be suitable depending on how complex your data structures are. But it may be sufficient to use the Visibility Timeout of SQS to ensure, that messages get processed again, if the first attempt fails. To be clear: Step functions will work, but if you should use them depends on what you want to optimize for: Speed, Reliability, Cost, Maintainability,...
– Maurice
Jan 6 at 9:33
Speed, Reliability are what is expected out of the architecture and the messages put to the queue needs to be transformed into some structure before dumping to SQS or Redis. Thats why thought of using Step Function.
– Kajol Kumari
Jan 6 at 15:38
For the status tracking of each updates and retrying failed execution I thought of using Step Function.
– Kajol Kumari
Jan 6 at 8:53
For the status tracking of each updates and retrying failed execution I thought of using Step Function.
– Kajol Kumari
Jan 6 at 8:53
You could do that and it may be suitable depending on how complex your data structures are. But it may be sufficient to use the Visibility Timeout of SQS to ensure, that messages get processed again, if the first attempt fails. To be clear: Step functions will work, but if you should use them depends on what you want to optimize for: Speed, Reliability, Cost, Maintainability,...
– Maurice
Jan 6 at 9:33
You could do that and it may be suitable depending on how complex your data structures are. But it may be sufficient to use the Visibility Timeout of SQS to ensure, that messages get processed again, if the first attempt fails. To be clear: Step functions will work, but if you should use them depends on what you want to optimize for: Speed, Reliability, Cost, Maintainability,...
– Maurice
Jan 6 at 9:33
Speed, Reliability are what is expected out of the architecture and the messages put to the queue needs to be transformed into some structure before dumping to SQS or Redis. Thats why thought of using Step Function.
– Kajol Kumari
Jan 6 at 15:38
Speed, Reliability are what is expected out of the architecture and the messages put to the queue needs to be transformed into some structure before dumping to SQS or Redis. Thats why thought of using Step Function.
– Kajol Kumari
Jan 6 at 15:38
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54004230%2fusing-step-function-for-synchronizing-primay-data-source-with-secondary-data-sou%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can you give us more information about the primary Data Source? What kind of Database is it? - If you're lucky enough to use DynamoDB, DynamoDB streams and/or DAX might me a suitable option.
– Maurice
Jan 2 at 11:46
Primay Data Source is not AWS based and is internal to my company. @Maurice I am planning to pass on the changes done to Primary Data Source to Step Function using SQS-Lambda
– Kajol Kumari
Jan 2 at 12:07
I'm still not sure how your architecture currently looks... - A solution for this might be some kind of write-through-caching behavior, which means your application would write all changes to Redis and the DB, which means all new updates are always in present in Redis - although this may waste memory if changed data is infrequently accessed. If you can give us more information about your use case we might be able to provide better solutions - if possible a small diagram would probably help.
– Maurice
Jan 2 at 12:32
@Maurice I have updated the question with the current architecture as well as the architecture I am thinking of.
– Kajol Kumari
Jan 5 at 9:57