Mixing
Distributed Lock Manager with Azure SQL database
We have Web API using Azure SQL database. Database model has Customers and Managers. Customers can add appointments. We can't allow overlapping appointments from 2 or more Customers for same Manager. Because we are working in a distributed environment (multiple instances of web server can insert records into database at the same time), there is a possibility that appointments that are not valid will be saved. As an example, Customer 1 wants an appointment between 10:00 - 10: 30. Customer 2 wants an appointment between 10:15 - 10:45. If both appointments happen during the same time, then the validation code in Web API, will not catch an error. That's why we need something like distributed lock manager. We read about Redlock from Redis and Zookeeper. My questions is: Is Redlock or Zookeeper good choise for our use case or there is some better solution?
If we would use Redlock than we would go with Azure Redis Cache because we already use Azure Cloud to host our Web API. We plan to identify shared resource (resource we want to lock) by using ManagerId + Date. This would result in lock for Manager on one date, so it would be possible to have other locks for same Manager on some other date. We plan to use one instance of Azure Redis Cache, is this safe enough?
azure redis locking apache-zookeeper asked Apr 11 '16 at 13:08
Vukasin SpasicVukasin Spasic
113
|
show 4 more comments
We have Web API using Azure SQL database. Database model has Customers and Managers. Customers can add appointments. We can't allow overlapping appointments from 2 or more Customers for same Manager. Because we are working in a distributed environment (multiple instances of web server can insert records into database at the same time), there is a possibility that appointments that are not valid will be saved. As an example, Customer 1 wants an appointment between 10:00 - 10: 30. Customer 2 wants an appointment between 10:15 - 10:45. If both appointments happen during the same time, then the validation code in Web API, will not catch an error. That's why we need something like distributed lock manager. We read about Redlock from Redis and Zookeeper. My questions is: Is Redlock or Zookeeper good choise for our use case or there is some better solution?
If we would use Redlock than we would go with Azure Redis Cache because we already use Azure Cloud to host our Web API. We plan to identify shared resource (resource we want to lock) by using ManagerId + Date. This would result in lock for Manager on one date, so it would be possible to have other locks for same Manager on some other date. We plan to use one instance of Azure Redis Cache, is this safe enough?
azure redis locking apache-zookeeper asked Apr 11 '16 at 13:08
Vukasin SpasicVukasin Spasic
113
1
I would not solve this by locking the whole thing down. This is a first-come, first-serve scenario, right? Before persisting the appointment, check that the appointment slot is still available. If not - return an HTTP 409 or something - and notify the user to find another slot - else return a 200 from your WebAPI.
– Jochen van Wylick
Apr 11 '16 at 14:21
We are using Entity Framework ORM so there is possibility that while one instance checks for appointment free slot and determines it is free, before it inserts new appointment to database, another instance of web service also determines that appointment slot is free and also insert appointment = we get appointments overlap, so we are trying to solve this problem
– Vukasin Spasic
Apr 11 '16 at 19:18
Sure I see your point - but still there are ways around this without the locking. For instance: use WebAPI to put the messages on a queue and use just 1 worker to pick up the messages, check for overlap and store in the database. A way to solve concurrency problems is to eliminate concurrency. I got this from Mark Seamann's course on Functional Architecture on Pluralsight: pluralsight.com/courses/functional-architecture-fsharp
– Jochen van Wylick
Apr 11 '16 at 20:08
Hey @JochenvanWylick, I'm one of Vukasins colleagues. Your right with the message queue that it can eliminate concurrency, but this brings two issues: You can't scale out the worker role and second, the user needs to be notified right away if the appointment was created successfully which would mean that we have to create a more complex notification system for the UI
– Alex Maie
Apr 12 '16 at 9:48
Yes, indeed, both points are correct. It's a tradeoff - but certainly possible. That's why I stronly suggest that course on pluralsight. A famous case of this is the Amazon inventory system. If you present '1 item left' to 2 users at the same time - and one buys the thing quicker, the second buyer could get an error. But amazon doesn't lock down buying that item as soon as someone looks at it - they compensate in case they sold more than they have in inventory ( which happens only a few times ).
– Jochen van Wylick
Apr 12 '16 at 10:28
|
show 4 more comments
We have Web API using Azure SQL database. Database model has Customers and Managers. Customers can add appointments. We can't allow overlapping appointments from 2 or more Customers for same Manager. Because we are working in a distributed environment (multiple instances of web server can insert records into database at the same time), there is a possibility that appointments that are not valid will be saved. As an example, Customer 1 wants an appointment between 10:00 - 10: 30. Customer 2 wants an appointment between 10:15 - 10:45. If both appointments happen during the same time, then the validation code in Web API, will not catch an error. That's why we need something like distributed lock manager. We read about Redlock from Redis and Zookeeper. My questions is: Is Redlock or Zookeeper good choise for our use case or there is some better solution?
If we would use Redlock than we would go with Azure Redis Cache because we already use Azure Cloud to host our Web API. We plan to identify shared resource (resource we want to lock) by using ManagerId + Date. This would result in lock for Manager on one date, so it would be possible to have other locks for same Manager on some other date. We plan to use one instance of Azure Redis Cache, is this safe enough?
azure redis locking apache-zookeeper asked Apr 11 '16 at 13:08
Vukasin SpasicVukasin Spasic
113
We have Web API using Azure SQL database. Database model has Customers and Managers. Customers can add appointments. We can't allow overlapping appointments from 2 or more Customers for same Manager. Because we are working in a distributed environment (multiple instances of web server can insert records into database at the same time), there is a possibility that appointments that are not valid will be saved. As an example, Customer 1 wants an appointment between 10:00 - 10: 30. Customer 2 wants an appointment between 10:15 - 10:45. If both appointments happen during the same time, then the validation code in Web API, will not catch an error. That's why we need something like distributed lock manager. We read about Redlock from Redis and Zookeeper. My questions is: Is Redlock or Zookeeper good choise for our use case or there is some better solution?
If we would use Redlock than we would go with Azure Redis Cache because we already use Azure Cloud to host our Web API. We plan to identify shared resource (resource we want to lock) by using ManagerId + Date. This would result in lock for Manager on one date, so it would be possible to have other locks for same Manager on some other date. We plan to use one instance of Azure Redis Cache, is this safe enough?
azure redis locking apache-zookeeper azure redis locking apache-zookeeper asked Apr 11 '16 at 13:08
Vukasin SpasicVukasin Spasic
113
asked Apr 11 '16 at 13:08
Vukasin SpasicVukasin Spasic
113
edited Apr 12 '16 at 9:59
Vukasin Spasic
asked Apr 11 '16 at 13:08
Vukasin SpasicVukasin Spasic
113
asked Apr 11 '16 at 13:08
Vukasin SpasicVukasin Spasic
113
asked Apr 11 '16 at 13:08
Vukasin SpasicVukasin Spasic
113
113
1
I would not solve this by locking the whole thing down. This is a first-come, first-serve scenario, right? Before persisting the appointment, check that the appointment slot is still available. If not - return an HTTP 409 or something - and notify the user to find another slot - else return a 200 from your WebAPI.
– Jochen van Wylick
Apr 11 '16 at 14:21
We are using Entity Framework ORM so there is possibility that while one instance checks for appointment free slot and determines it is free, before it inserts new appointment to database, another instance of web service also determines that appointment slot is free and also insert appointment = we get appointments overlap, so we are trying to solve this problem
– Vukasin Spasic
Apr 11 '16 at 19:18
Sure I see your point - but still there are ways around this without the locking. For instance: use WebAPI to put the messages on a queue and use just 1 worker to pick up the messages, check for overlap and store in the database. A way to solve concurrency problems is to eliminate concurrency. I got this from Mark Seamann's course on Functional Architecture on Pluralsight: pluralsight.com/courses/functional-architecture-fsharp
– Jochen van Wylick
Apr 11 '16 at 20:08
Hey @JochenvanWylick, I'm one of Vukasins colleagues. Your right with the message queue that it can eliminate concurrency, but this brings two issues: You can't scale out the worker role and second, the user needs to be notified right away if the appointment was created successfully which would mean that we have to create a more complex notification system for the UI
– Alex Maie
Apr 12 '16 at 9:48
Yes, indeed, both points are correct. It's a tradeoff - but certainly possible. That's why I stronly suggest that course on pluralsight. A famous case of this is the Amazon inventory system. If you present '1 item left' to 2 users at the same time - and one buys the thing quicker, the second buyer could get an error. But amazon doesn't lock down buying that item as soon as someone looks at it - they compensate in case they sold more than they have in inventory ( which happens only a few times ).
– Jochen van Wylick
Apr 12 '16 at 10:28
|
show 4 more comments
1
I would not solve this by locking the whole thing down. This is a first-come, first-serve scenario, right? Before persisting the appointment, check that the appointment slot is still available. If not - return an HTTP 409 or something - and notify the user to find another slot - else return a 200 from your WebAPI.
– Jochen van Wylick
Apr 11 '16 at 14:21
We are using Entity Framework ORM so there is possibility that while one instance checks for appointment free slot and determines it is free, before it inserts new appointment to database, another instance of web service also determines that appointment slot is free and also insert appointment = we get appointments overlap, so we are trying to solve this problem
– Vukasin Spasic
Apr 11 '16 at 19:18
Sure I see your point - but still there are ways around this without the locking. For instance: use WebAPI to put the messages on a queue and use just 1 worker to pick up the messages, check for overlap and store in the database. A way to solve concurrency problems is to eliminate concurrency. I got this from Mark Seamann's course on Functional Architecture on Pluralsight: pluralsight.com/courses/functional-architecture-fsharp
– Jochen van Wylick
Apr 11 '16 at 20:08
Hey @JochenvanWylick, I'm one of Vukasins colleagues. Your right with the message queue that it can eliminate concurrency, but this brings two issues: You can't scale out the worker role and second, the user needs to be notified right away if the appointment was created successfully which would mean that we have to create a more complex notification system for the UI
– Alex Maie
Apr 12 '16 at 9:48
Yes, indeed, both points are correct. It's a tradeoff - but certainly possible. That's why I stronly suggest that course on pluralsight. A famous case of this is the Amazon inventory system. If you present '1 item left' to 2 users at the same time - and one buys the thing quicker, the second buyer could get an error. But amazon doesn't lock down buying that item as soon as someone looks at it - they compensate in case they sold more than they have in inventory ( which happens only a few times ).
– Jochen van Wylick
Apr 12 '16 at 10:28
1
1
I would not solve this by locking the whole thing down. This is a first-come, first-serve scenario, right? Before persisting the appointment, check that the appointment slot is still available. If not - return an HTTP 409 or something - and notify the user to find another slot - else return a 200 from your WebAPI.
– Jochen van Wylick
Apr 11 '16 at 14:21
I would not solve this by locking the whole thing down. This is a first-come, first-serve scenario, right? Before persisting the appointment, check that the appointment slot is still available. If not - return an HTTP 409 or something - and notify the user to find another slot - else return a 200 from your WebAPI.
– Jochen van Wylick
Apr 11 '16 at 14:21
We are using Entity Framework ORM so there is possibility that while one instance checks for appointment free slot and determines it is free, before it inserts new appointment to database, another instance of web service also determines that appointment slot is free and also insert appointment = we get appointments overlap, so we are trying to solve this problem
– Vukasin Spasic
Apr 11 '16 at 19:18
We are using Entity Framework ORM so there is possibility that while one instance checks for appointment free slot and determines it is free, before it inserts new appointment to database, another instance of web service also determines that appointment slot is free and also insert appointment = we get appointments overlap, so we are trying to solve this problem
– Vukasin Spasic
Apr 11 '16 at 19:18
Sure I see your point - but still there are ways around this without the locking. For instance: use WebAPI to put the messages on a queue and use just 1 worker to pick up the messages, check for overlap and store in the database. A way to solve concurrency problems is to eliminate concurrency. I got this from Mark Seamann's course on Functional Architecture on Pluralsight: pluralsight.com/courses/functional-architecture-fsharp
– Jochen van Wylick
Apr 11 '16 at 20:08
Sure I see your point - but still there are ways around this without the locking. For instance: use WebAPI to put the messages on a queue and use just 1 worker to pick up the messages, check for overlap and store in the database. A way to solve concurrency problems is to eliminate concurrency. I got this from Mark Seamann's course on Functional Architecture on Pluralsight: pluralsight.com/courses/functional-architecture-fsharp
– Jochen van Wylick
Apr 11 '16 at 20:08
Hey @JochenvanWylick, I'm one of Vukasins colleagues. Your right with the message queue that it can eliminate concurrency, but this brings two issues: You can't scale out the worker role and second, the user needs to be notified right away if the appointment was created successfully which would mean that we have to create a more complex notification system for the UI
– Alex Maie
Apr 12 '16 at 9:48
Hey @JochenvanWylick, I'm one of Vukasins colleagues. Your right with the message queue that it can eliminate concurrency, but this brings two issues: You can't scale out the worker role and second, the user needs to be notified right away if the appointment was created successfully which would mean that we have to create a more complex notification system for the UI
– Alex Maie
Apr 12 '16 at 9:48
Yes, indeed, both points are correct. It's a tradeoff - but certainly possible. That's why I stronly suggest that course on pluralsight. A famous case of this is the Amazon inventory system. If you present '1 item left' to 2 users at the same time - and one buys the thing quicker, the second buyer could get an error. But amazon doesn't lock down buying that item as soon as someone looks at it - they compensate in case they sold more than they have in inventory ( which happens only a few times ).
– Jochen van Wylick
Apr 12 '16 at 10:28
Yes, indeed, both points are correct. It's a tradeoff - but certainly possible. That's why I stronly suggest that course on pluralsight. A famous case of this is the Amazon inventory system. If you present '1 item left' to 2 users at the same time - and one buys the thing quicker, the second buyer could get an error. But amazon doesn't lock down buying that item as soon as someone looks at it - they compensate in case they sold more than they have in inventory ( which happens only a few times ).
– Jochen van Wylick
Apr 12 '16 at 10:28
|
show 4 more comments
1 Answer
1
active
oldest
votes
Q1: Is Redlock or Zookeeper good choise for our use case or there is some better solution?
I consider Redlock as not the best choice for your use case because:
a) its guarantees are for a specific amount of time (TTL) set before using the DB operation. If for some reason (talk to DevOps for incredible ones and also check How to do distributed locking) the DB operation takes longer than TTL you loose the guarantee for lock validity (see lock validity time in the official documentation). You could use large TTL (minutes) or you could try to extend its validity with another thread which would monitor the DB operation time - but this gets incredibly complicated. On the other hands with zookeeper (ZK) your lock is there till you remove it or the process dies; it could be the situation when your DB operation hangs which would lead to the lock also to hang but these kind of problems are easily spotted by DevOps tools which will kill the hanging process which in turn will free the ZK lock (there's also the option to have a monitoring process which to also do this faster and in a more specific to you business fashion).
b) while trying to lock the processes must “fight” to win a lock; the “fighting” suppose for them to wait then retry getting the lock. These could lead to retry-count to overflow which would lead to a fail to get the lock. This seems to me a less important issue but with ZK the solution is far better: there’s no “fight” but all processes will get in a line of ones waiting their turn to get the lock (check ZK lock recipe).
c) Redlock is based on time measures which is incredible tricky; check at least the paragraph containing “feeling smug” at How to do distributed locking (Conclusion paragraph too) then think again how large should be that TTL value in order to be sure about your RedLock (time) based locking.
For these reasons I consider RedLock a risky solution while Zookeeper a good solution for your use case. Other better distributed locking solution fit for your case I don’t know but other distributed locking solutions do exist, e.g. just check Apache ZooKeeper vs. etcd3.
Q2: We plan to use one instance of Azure Redis Cache, is this safe enough?
It could be safe for your use case because the TTL seems to be predictable (if we really trust the time measuring - see the warn below) but only if the slave taking over a failed master could be delayed (not sure if possible, you should check the Redis configuration capabilities). In case you loose the master before a lock is synchronized to the slave than another process could just acquire the same lock. Redlock recommends to use delayed restarts (check Performance, crash-recovery and fsync in official documentation) with a period at least of 1 TTL. If for the Q1:a+c reason your TTL is a very long one than your system won’t be able to lock for maybe an unacceptable large period (because the only 1 Redis master you have must be replaced by the slave in a delayed fashion).
PS: I stress again to read Martin Kleppmann's opinion on Redlock where you’ll find incredible reasons for a DB operation to be delayed (search for before reaching the storage service) and also incredible reasons for not relaying on time measuring when locking (and also an interesting argumentation against using Redlock)
answered Nov 21 '18 at 21:12
adrhcadrhc
495314
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f36549656%2fdistributed-lock-manager-with-azure-sql-database%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Q1: Is Redlock or Zookeeper good choise for our use case or there is some better solution?
I consider Redlock as not the best choice for your use case because:
a) its guarantees are for a specific amount of time (TTL) set before using the DB operation. If for some reason (talk to DevOps for incredible ones and also check How to do distributed locking) the DB operation takes longer than TTL you loose the guarantee for lock validity (see lock validity time in the official documentation). You could use large TTL (minutes) or you could try to extend its validity with another thread which would monitor the DB operation time - but this gets incredibly complicated. On the other hands with zookeeper (ZK) your lock is there till you remove it or the process dies; it could be the situation when your DB operation hangs which would lead to the lock also to hang but these kind of problems are easily spotted by DevOps tools which will kill the hanging process which in turn will free the ZK lock (there's also the option to have a monitoring process which to also do this faster and in a more specific to you business fashion).
b) while trying to lock the processes must “fight” to win a lock; the “fighting” suppose for them to wait then retry getting the lock. These could lead to retry-count to overflow which would lead to a fail to get the lock. This seems to me a less important issue but with ZK the solution is far better: there’s no “fight” but all processes will get in a line of ones waiting their turn to get the lock (check ZK lock recipe).
c) Redlock is based on time measures which is incredible tricky; check at least the paragraph containing “feeling smug” at How to do distributed locking (Conclusion paragraph too) then think again how large should be that TTL value in order to be sure about your RedLock (time) based locking.
For these reasons I consider RedLock a risky solution while Zookeeper a good solution for your use case. Other better distributed locking solution fit for your case I don’t know but other distributed locking solutions do exist, e.g. just check Apache ZooKeeper vs. etcd3.
Q2: We plan to use one instance of Azure Redis Cache, is this safe enough?
It could be safe for your use case because the TTL seems to be predictable (if we really trust the time measuring - see the warn below) but only if the slave taking over a failed master could be delayed (not sure if possible, you should check the Redis configuration capabilities). In case you loose the master before a lock is synchronized to the slave than another process could just acquire the same lock. Redlock recommends to use delayed restarts (check Performance, crash-recovery and fsync in official documentation) with a period at least of 1 TTL. If for the Q1:a+c reason your TTL is a very long one than your system won’t be able to lock for maybe an unacceptable large period (because the only 1 Redis master you have must be replaced by the slave in a delayed fashion).
PS: I stress again to read Martin Kleppmann's opinion on Redlock where you’ll find incredible reasons for a DB operation to be delayed (search for before reaching the storage service) and also incredible reasons for not relaying on time measuring when locking (and also an interesting argumentation against using Redlock)
answered Nov 21 '18 at 21:12
adrhcadrhc
495314
add a comment |
Q1: Is Redlock or Zookeeper good choise for our use case or there is some better solution?
I consider Redlock as not the best choice for your use case because:
a) its guarantees are for a specific amount of time (TTL) set before using the DB operation. If for some reason (talk to DevOps for incredible ones and also check How to do distributed locking) the DB operation takes longer than TTL you loose the guarantee for lock validity (see lock validity time in the official documentation). You could use large TTL (minutes) or you could try to extend its validity with another thread which would monitor the DB operation time - but this gets incredibly complicated. On the other hands with zookeeper (ZK) your lock is there till you remove it or the process dies; it could be the situation when your DB operation hangs which would lead to the lock also to hang but these kind of problems are easily spotted by DevOps tools which will kill the hanging process which in turn will free the ZK lock (there's also the option to have a monitoring process which to also do this faster and in a more specific to you business fashion).
b) while trying to lock the processes must “fight” to win a lock; the “fighting” suppose for them to wait then retry getting the lock. These could lead to retry-count to overflow which would lead to a fail to get the lock. This seems to me a less important issue but with ZK the solution is far better: there’s no “fight” but all processes will get in a line of ones waiting their turn to get the lock (check ZK lock recipe).
c) Redlock is based on time measures which is incredible tricky; check at least the paragraph containing “feeling smug” at How to do distributed locking (Conclusion paragraph too) then think again how large should be that TTL value in order to be sure about your RedLock (time) based locking.
For these reasons I consider RedLock a risky solution while Zookeeper a good solution for your use case. Other better distributed locking solution fit for your case I don’t know but other distributed locking solutions do exist, e.g. just check Apache ZooKeeper vs. etcd3.
Q2: We plan to use one instance of Azure Redis Cache, is this safe enough?
It could be safe for your use case because the TTL seems to be predictable (if we really trust the time measuring - see the warn below) but only if the slave taking over a failed master could be delayed (not sure if possible, you should check the Redis configuration capabilities). In case you loose the master before a lock is synchronized to the slave than another process could just acquire the same lock. Redlock recommends to use delayed restarts (check Performance, crash-recovery and fsync in official documentation) with a period at least of 1 TTL. If for the Q1:a+c reason your TTL is a very long one than your system won’t be able to lock for maybe an unacceptable large period (because the only 1 Redis master you have must be replaced by the slave in a delayed fashion).
PS: I stress again to read Martin Kleppmann's opinion on Redlock where you’ll find incredible reasons for a DB operation to be delayed (search for before reaching the storage service) and also incredible reasons for not relaying on time measuring when locking (and also an interesting argumentation against using Redlock)
answered Nov 21 '18 at 21:12
adrhcadrhc
495314
add a comment |
Q1: Is Redlock or Zookeeper good choise for our use case or there is some better solution?
I consider Redlock as not the best choice for your use case because:
a) its guarantees are for a specific amount of time (TTL) set before using the DB operation. If for some reason (talk to DevOps for incredible ones and also check How to do distributed locking) the DB operation takes longer than TTL you loose the guarantee for lock validity (see lock validity time in the official documentation). You could use large TTL (minutes) or you could try to extend its validity with another thread which would monitor the DB operation time - but this gets incredibly complicated. On the other hands with zookeeper (ZK) your lock is there till you remove it or the process dies; it could be the situation when your DB operation hangs which would lead to the lock also to hang but these kind of problems are easily spotted by DevOps tools which will kill the hanging process which in turn will free the ZK lock (there's also the option to have a monitoring process which to also do this faster and in a more specific to you business fashion).
b) while trying to lock the processes must “fight” to win a lock; the “fighting” suppose for them to wait then retry getting the lock. These could lead to retry-count to overflow which would lead to a fail to get the lock. This seems to me a less important issue but with ZK the solution is far better: there’s no “fight” but all processes will get in a line of ones waiting their turn to get the lock (check ZK lock recipe).
c) Redlock is based on time measures which is incredible tricky; check at least the paragraph containing “feeling smug” at How to do distributed locking (Conclusion paragraph too) then think again how large should be that TTL value in order to be sure about your RedLock (time) based locking.
For these reasons I consider RedLock a risky solution while Zookeeper a good solution for your use case. Other better distributed locking solution fit for your case I don’t know but other distributed locking solutions do exist, e.g. just check Apache ZooKeeper vs. etcd3.
Q2: We plan to use one instance of Azure Redis Cache, is this safe enough?
It could be safe for your use case because the TTL seems to be predictable (if we really trust the time measuring - see the warn below) but only if the slave taking over a failed master could be delayed (not sure if possible, you should check the Redis configuration capabilities). In case you loose the master before a lock is synchronized to the slave than another process could just acquire the same lock. Redlock recommends to use delayed restarts (check Performance, crash-recovery and fsync in official documentation) with a period at least of 1 TTL. If for the Q1:a+c reason your TTL is a very long one than your system won’t be able to lock for maybe an unacceptable large period (because the only 1 Redis master you have must be replaced by the slave in a delayed fashion).
PS: I stress again to read Martin Kleppmann's opinion on Redlock where you’ll find incredible reasons for a DB operation to be delayed (search for before reaching the storage service) and also incredible reasons for not relaying on time measuring when locking (and also an interesting argumentation against using Redlock)
answered Nov 21 '18 at 21:12
adrhcadrhc
495314
Q1: Is Redlock or Zookeeper good choise for our use case or there is some better solution?
I consider Redlock as not the best choice for your use case because:
a) its guarantees are for a specific amount of time (TTL) set before using the DB operation. If for some reason (talk to DevOps for incredible ones and also check How to do distributed locking) the DB operation takes longer than TTL you loose the guarantee for lock validity (see lock validity time in the official documentation). You could use large TTL (minutes) or you could try to extend its validity with another thread which would monitor the DB operation time - but this gets incredibly complicated. On the other hands with zookeeper (ZK) your lock is there till you remove it or the process dies; it could be the situation when your DB operation hangs which would lead to the lock also to hang but these kind of problems are easily spotted by DevOps tools which will kill the hanging process which in turn will free the ZK lock (there's also the option to have a monitoring process which to also do this faster and in a more specific to you business fashion).
b) while trying to lock the processes must “fight” to win a lock; the “fighting” suppose for them to wait then retry getting the lock. These could lead to retry-count to overflow which would lead to a fail to get the lock. This seems to me a less important issue but with ZK the solution is far better: there’s no “fight” but all processes will get in a line of ones waiting their turn to get the lock (check ZK lock recipe).
c) Redlock is based on time measures which is incredible tricky; check at least the paragraph containing “feeling smug” at How to do distributed locking (Conclusion paragraph too) then think again how large should be that TTL value in order to be sure about your RedLock (time) based locking.
For these reasons I consider RedLock a risky solution while Zookeeper a good solution for your use case. Other better distributed locking solution fit for your case I don’t know but other distributed locking solutions do exist, e.g. just check Apache ZooKeeper vs. etcd3.
Q2: We plan to use one instance of Azure Redis Cache, is this safe enough?
It could be safe for your use case because the TTL seems to be predictable (if we really trust the time measuring - see the warn below) but only if the slave taking over a failed master could be delayed (not sure if possible, you should check the Redis configuration capabilities). In case you loose the master before a lock is synchronized to the slave than another process could just acquire the same lock. Redlock recommends to use delayed restarts (check Performance, crash-recovery and fsync in official documentation) with a period at least of 1 TTL. If for the Q1:a+c reason your TTL is a very long one than your system won’t be able to lock for maybe an unacceptable large period (because the only 1 Redis master you have must be replaced by the slave in a delayed fashion).
PS: I stress again to read Martin Kleppmann's opinion on Redlock where you’ll find incredible reasons for a DB operation to be delayed (search for before reaching the storage service) and also incredible reasons for not relaying on time measuring when locking (and also an interesting argumentation against using Redlock)
answered Nov 21 '18 at 21:12
adrhcadrhc
495314
edited Nov 28 '18 at 10:08
answered Nov 21 '18 at 21:12
adrhcadrhc
495314
answered Nov 21 '18 at 21:12
adrhcadrhc
495314
answered Nov 21 '18 at 21:12
adrhcadrhc
495314
495314
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f36549656%2fdistributed-lock-manager-with-azure-sql-database%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
I would not solve this by locking the whole thing down. This is a first-come, first-serve scenario, right? Before persisting the appointment, check that the appointment slot is still available. If not - return an HTTP 409 or something - and notify the user to find another slot - else return a 200 from your WebAPI.
– Jochen van Wylick
Apr 11 '16 at 14:21
We are using Entity Framework ORM so there is possibility that while one instance checks for appointment free slot and determines it is free, before it inserts new appointment to database, another instance of web service also determines that appointment slot is free and also insert appointment = we get appointments overlap, so we are trying to solve this problem
– Vukasin Spasic
Apr 11 '16 at 19:18
Sure I see your point - but still there are ways around this without the locking. For instance: use WebAPI to put the messages on a queue and use just 1 worker to pick up the messages, check for overlap and store in the database. A way to solve concurrency problems is to eliminate concurrency. I got this from Mark Seamann's course on Functional Architecture on Pluralsight: pluralsight.com/courses/functional-architecture-fsharp
– Jochen van Wylick
Apr 11 '16 at 20:08
Hey @JochenvanWylick, I'm one of Vukasins colleagues. Your right with the message queue that it can eliminate concurrency, but this brings two issues: You can't scale out the worker role and second, the user needs to be notified right away if the appointment was created successfully which would mean that we have to create a more complex notification system for the UI
– Alex Maie
Apr 12 '16 at 9:48
Yes, indeed, both points are correct. It's a tradeoff - but certainly possible. That's why I stronly suggest that course on pluralsight. A famous case of this is the Amazon inventory system. If you present '1 item left' to 2 users at the same time - and one buys the thing quicker, the second buyer could get an error. But amazon doesn't lock down buying that item as soon as someone looks at it - they compensate in case they sold more than they have in inventory ( which happens only a few times ).
– Jochen van Wylick
Apr 12 '16 at 10:28