Python 3: Will Queue.empty() be correct if, for some time, no threads are putting in or pulling out elements?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}

I'm using a queue.Queue() in Python 3.7 (Windows 10). Sometimes I need to check if the queue is empty or not. The documentation reads:

Queue.empty()
Return True if the queue is empty, False otherwise. If empty() returns True it doesn’t guarantee that a subsequent call to put() will not block. Similarly, if empty() returns False it doesn’t guarantee that a subsequent call to get() will not block.

In other words, one cannot rely full 100% on the returned value from Queue.empty(). That seems indeed natural to me. In in a multithreaded environment, some side-thread can sneak in and put an element on the queue the very moment you're asking if it's empty. Consequently, the returned answer is unreliable. I get that.

But what about the following scenario? A timer ticks: t1, t2, t3, ... On each tick of the timer, the queue is asked: "are you empty"?

   t1       t2       t3       t4       t5       t6

---|--------|--------|--------|--------|--------|--------->

  __________  __________/__________  ___________/

             /                      /

     Different threads            No action on

     put elements on the          the queue.

     queue and/or pull

     elements from it.

Now let's imagine that from timer tick t4 onwards, we are absolutely sure that the queue isn't getting touched by any thread - no elements get inserted, nor are any pulled out.

Perhaps on timer tick t4, the queue didn't have the time yet to stabilize internally, and the returned value from Queue.empty() is wrong. But will it stabilize after some time, say on timer tick t5 or t6?

Possibility 1: No, it never stabilizes

That's really horrible. I will never use this queue again. End of story.

Possibility 2: Yes, it stabilizes after x milliseconds

I get that you cannot give an exact answer here. It will depend on many factors: the operating system, the hardware, the amount of threads that are taking resources, ... But it would just be great to know that it does stabilize after some time. I'll just take enough margin.

NOTE:

If Queue.empty() stabilizes (or not) after some time, I suppose the same is true for Queue.qsize(), right?

edited Jan 3 at 16:30

asked Jan 3 at 16:23

K.Mulier

2,46012859

add a comment |

I'm using a queue.Queue() in Python 3.7 (Windows 10). Sometimes I need to check if the queue is empty or not. The documentation reads:

Queue.empty()
Return True if the queue is empty, False otherwise. If empty() returns True it doesn’t guarantee that a subsequent call to put() will not block. Similarly, if empty() returns False it doesn’t guarantee that a subsequent call to get() will not block.

But what about the following scenario? A timer ticks: t1, t2, t3, ... On each tick of the timer, the queue is asked: "are you empty"?

   t1       t2       t3       t4       t5       t6

---|--------|--------|--------|--------|--------|--------->

  __________  __________/__________  ___________/

             /                      /

     Different threads            No action on

     put elements on the          the queue.

     queue and/or pull

     elements from it.

Now let's imagine that from timer tick t4 onwards, we are absolutely sure that the queue isn't getting touched by any thread - no elements get inserted, nor are any pulled out.

Possibility 1: No, it never stabilizes

That's really horrible. I will never use this queue again. End of story.

NOTE:

If Queue.empty() stabilizes (or not) after some time, I suppose the same is true for Queue.qsize(), right?

edited Jan 3 at 16:30

asked Jan 3 at 16:23

K.Mulier

2,46012859

add a comment |

I'm using a queue.Queue() in Python 3.7 (Windows 10). Sometimes I need to check if the queue is empty or not. The documentation reads:

Queue.empty()
Return True if the queue is empty, False otherwise. If empty() returns True it doesn’t guarantee that a subsequent call to put() will not block. Similarly, if empty() returns False it doesn’t guarantee that a subsequent call to get() will not block.

But what about the following scenario? A timer ticks: t1, t2, t3, ... On each tick of the timer, the queue is asked: "are you empty"?

   t1       t2       t3       t4       t5       t6

---|--------|--------|--------|--------|--------|--------->

  __________  __________/__________  ___________/

             /                      /

     Different threads            No action on

     put elements on the          the queue.

     queue and/or pull

     elements from it.

Now let's imagine that from timer tick t4 onwards, we are absolutely sure that the queue isn't getting touched by any thread - no elements get inserted, nor are any pulled out.

Possibility 1: No, it never stabilizes

That's really horrible. I will never use this queue again. End of story.

NOTE:

If Queue.empty() stabilizes (or not) after some time, I suppose the same is true for Queue.qsize(), right?

edited Jan 3 at 16:30

asked Jan 3 at 16:23

K.Mulier

2,46012859

I'm using a queue.Queue() in Python 3.7 (Windows 10). Sometimes I need to check if the queue is empty or not. The documentation reads:

Queue.empty()
Return True if the queue is empty, False otherwise. If empty() returns True it doesn’t guarantee that a subsequent call to put() will not block. Similarly, if empty() returns False it doesn’t guarantee that a subsequent call to get() will not block.

But what about the following scenario? A timer ticks: t1, t2, t3, ... On each tick of the timer, the queue is asked: "are you empty"?

   t1       t2       t3       t4       t5       t6

---|--------|--------|--------|--------|--------|--------->

  __________  __________/__________  ___________/

             /                      /

     Different threads            No action on

     put elements on the          the queue.

     queue and/or pull

     elements from it.

Now let's imagine that from timer tick t4 onwards, we are absolutely sure that the queue isn't getting touched by any thread - no elements get inserted, nor are any pulled out.

Possibility 1: No, it never stabilizes

That's really horrible. I will never use this queue again. End of story.

NOTE:

If Queue.empty() stabilizes (or not) after some time, I suppose the same is true for Queue.qsize(), right?

python python-3.x multithreading queue

edited Jan 3 at 16:30

asked Jan 3 at 16:23

K.Mulier

2,46012859

edited Jan 3 at 16:30

asked Jan 3 at 16:23

K.Mulier

2,46012859

edited Jan 3 at 16:30

asked Jan 3 at 16:23

K.Mulier

2,46012859

asked Jan 3 at 16:23

K.Mulier

2,46012859

asked Jan 3 at 16:23

K.Mulier

2,46012859

add a comment |

1 Answer
1

active

oldest

votes

It will stabilize. When no threads interact with the queue there is no possibility for a race condition any more.

The Queue.empty() method returns True or False based on the current state of the queue. At issue is that that state can change at any moment when another thread is given control and interacts with the queue, so within one thread, barring other synchronisation primitives (locks, semaphores, events, conditions, barriers, what-have-you), you can't rely on the return value for any length of time.

There is therefor no timeframe in which you can count on the state stabilising. There is no 'unstable state' in the queue object, only in your application as a whole, because you are using threads.

The same applies to the Queue.qsize() method; the return value is not fuzzy or approximate, it is an accurate measure of the state of the queue at that moment in time. But since your code can and will be interrupted when control is transferred to another thread, you can't rely on that measurement to inform how your code can then act on the queue, because by the time the next instruction in your thread could act on that measurement, control may have been switched away and back again, another thread could have acted on the queue, and the measurement is no longer relevant.

Use other synchronisation primitives to communicate between your threads if you need something more definitive.

edited Jan 3 at 17:06

answered Jan 3 at 16:29

Martijn Pieters♦

727k14425532356

Thank you, this looks indeed also natural to me. But the Python docs don't explicitly explain this, it just says something like: it's an approximation, don't rely on it. That's why I got confused.

– K.Mulier
Jan 3 at 16:33

@K.Mulier: that's the documentation for Queue.qsize(), which suffers from the same problem: it is a precise measurement, but a measurement in a single moment of time, in a system where other threads are likely to change the state at unknown points in the near future.

– Martijn Pieters♦
Jan 3 at 16:35

@K.Mulier: the issue with threads is that your code can't know when a thread switch will occur. It can happen at any time, and your code is not informed of the switch, or will notice that control has been returned. So from one instruction to the next, the system state on which a thread is operating can have been changed.

– Martijn Pieters♦
Jan 3 at 16:37

Thank you for helping me out. I get the way it works now :-)

– K.Mulier
Jan 3 at 16:49

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54026122%2fpython-3-will-queue-empty-be-correct-if-for-some-time-no-threads-are-puttin%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

It will stabilize. When no threads interact with the queue there is no possibility for a race condition any more.

There is therefor no timeframe in which you can count on the state stabilising. There is no 'unstable state' in the queue object, only in your application as a whole, because you are using threads.

Use other synchronisation primitives to communicate between your threads if you need something more definitive.

edited Jan 3 at 17:06

answered Jan 3 at 16:29

Martijn Pieters♦

727k14425532356

Thank you, this looks indeed also natural to me. But the Python docs don't explicitly explain this, it just says something like: it's an approximation, don't rely on it. That's why I got confused.

– K.Mulier
Jan 3 at 16:33

@K.Mulier: that's the documentation for Queue.qsize(), which suffers from the same problem: it is a precise measurement, but a measurement in a single moment of time, in a system where other threads are likely to change the state at unknown points in the near future.

– Martijn Pieters♦
Jan 3 at 16:35

@K.Mulier: the issue with threads is that your code can't know when a thread switch will occur. It can happen at any time, and your code is not informed of the switch, or will notice that control has been returned. So from one instruction to the next, the system state on which a thread is operating can have been changed.

– Martijn Pieters♦
Jan 3 at 16:37

Thank you for helping me out. I get the way it works now :-)

– K.Mulier
Jan 3 at 16:49

add a comment |

It will stabilize. When no threads interact with the queue there is no possibility for a race condition any more.

There is therefor no timeframe in which you can count on the state stabilising. There is no 'unstable state' in the queue object, only in your application as a whole, because you are using threads.

Use other synchronisation primitives to communicate between your threads if you need something more definitive.

edited Jan 3 at 17:06

answered Jan 3 at 16:29

Martijn Pieters♦

727k14425532356

Thank you, this looks indeed also natural to me. But the Python docs don't explicitly explain this, it just says something like: it's an approximation, don't rely on it. That's why I got confused.

– K.Mulier
Jan 3 at 16:33

@K.Mulier: that's the documentation for Queue.qsize(), which suffers from the same problem: it is a precise measurement, but a measurement in a single moment of time, in a system where other threads are likely to change the state at unknown points in the near future.

– Martijn Pieters♦
Jan 3 at 16:35

@K.Mulier: the issue with threads is that your code can't know when a thread switch will occur. It can happen at any time, and your code is not informed of the switch, or will notice that control has been returned. So from one instruction to the next, the system state on which a thread is operating can have been changed.

– Martijn Pieters♦
Jan 3 at 16:37

Thank you for helping me out. I get the way it works now :-)

– K.Mulier
Jan 3 at 16:49

add a comment |

It will stabilize. When no threads interact with the queue there is no possibility for a race condition any more.

There is therefor no timeframe in which you can count on the state stabilising. There is no 'unstable state' in the queue object, only in your application as a whole, because you are using threads.

Use other synchronisation primitives to communicate between your threads if you need something more definitive.

edited Jan 3 at 17:06

answered Jan 3 at 16:29

Martijn Pieters♦

727k14425532356

It will stabilize. When no threads interact with the queue there is no possibility for a race condition any more.

There is therefor no timeframe in which you can count on the state stabilising. There is no 'unstable state' in the queue object, only in your application as a whole, because you are using threads.

Use other synchronisation primitives to communicate between your threads if you need something more definitive.

edited Jan 3 at 17:06

answered Jan 3 at 16:29

Martijn Pieters♦

727k14425532356

edited Jan 3 at 17:06

answered Jan 3 at 16:29

Martijn Pieters♦

727k14425532356

answered Jan 3 at 16:29

Martijn Pieters♦

727k14425532356

answered Jan 3 at 16:29

Martijn Pieters♦

727k14425532356

Thank you, this looks indeed also natural to me. But the Python docs don't explicitly explain this, it just says something like: it's an approximation, don't rely on it. That's why I got confused.

– K.Mulier
Jan 3 at 16:33

@K.Mulier: that's the documentation for Queue.qsize(), which suffers from the same problem: it is a precise measurement, but a measurement in a single moment of time, in a system where other threads are likely to change the state at unknown points in the near future.

– Martijn Pieters♦
Jan 3 at 16:35

@K.Mulier: the issue with threads is that your code can't know when a thread switch will occur. It can happen at any time, and your code is not informed of the switch, or will notice that control has been returned. So from one instruction to the next, the system state on which a thread is operating can have been changed.

– Martijn Pieters♦
Jan 3 at 16:37

Thank you for helping me out. I get the way it works now :-)

– K.Mulier
Jan 3 at 16:49

add a comment |

Thank you, this looks indeed also natural to me. But the Python docs don't explicitly explain this, it just says something like: it's an approximation, don't rely on it. That's why I got confused.

– K.Mulier
Jan 3 at 16:33

@K.Mulier: that's the documentation for Queue.qsize(), which suffers from the same problem: it is a precise measurement, but a measurement in a single moment of time, in a system where other threads are likely to change the state at unknown points in the near future.

– Martijn Pieters♦
Jan 3 at 16:35

@K.Mulier: the issue with threads is that your code can't know when a thread switch will occur. It can happen at any time, and your code is not informed of the switch, or will notice that control has been returned. So from one instruction to the next, the system state on which a thread is operating can have been changed.

– Martijn Pieters♦
Jan 3 at 16:37

Thank you for helping me out. I get the way it works now :-)

– K.Mulier
Jan 3 at 16:49

Thank you, this looks indeed also natural to me. But the Python docs don't explicitly explain this, it just says something like: it's an approximation, don't rely on it. That's why I got confused.

– K.Mulier
Jan 3 at 16:33

@K.Mulier: that's the documentation for Queue.qsize(), which suffers from the same problem: it is a precise measurement, but a measurement in a single moment of time, in a system where other threads are likely to change the state at unknown points in the near future.

– Martijn Pieters♦
Jan 3 at 16:35

@K.Mulier: the issue with threads is that your code can't know when a thread switch will occur. It can happen at any time, and your code is not informed of the switch, or will notice that control has been returned. So from one instruction to the next, the system state on which a thread is operating can have been changed.

– Martijn Pieters♦
Jan 3 at 16:37

Thank you for helping me out. I get the way it works now :-)

– K.Mulier
Jan 3 at 16:49

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu