Mixing
How to realize a queue in dart?
I want to set up a queue of functions in Darts. Queuing should be asynchronous, allowing multiple functions to run concurrently. However, a maximum of three functions should be executed simultaneously. How can I achieve this?
I already tied working off a list but i am struggeling at adding a limit on same time running functions
List<String> queue = new List();
main(){
queue.add("...");
queue.add("...");
queue.add("...");
for(String q in queue){
await crawl(q);
}
}
crawl(String) async{
...
}
dart
asked Jan 2 at 12:14
VelsemVelsem
1
add a comment |
I want to set up a queue of functions in Darts. Queuing should be asynchronous, allowing multiple functions to run concurrently. However, a maximum of three functions should be executed simultaneously. How can I achieve this?
I already tied working off a list but i am struggeling at adding a limit on same time running functions
List<String> queue = new List();
main(){
queue.add("...");
queue.add("...");
queue.add("...");
for(String q in queue){
await crawl(q);
}
}
crawl(String) async{
...
}
dart
asked Jan 2 at 12:14
VelsemVelsem
1
There is a queue class for Dart api.dartlang.org/stable/2.1.0/dart-collection/Queue-class.html and Dart is single-threaded so there is no way 2 functions can modify the queue at the same time. All methods of the queue are sync and therefore atomic. "Queuing should be asynchronous" not sure what that means.
– Günter Zöchbauer
Jan 2 at 12:21
"Queuing should be asynchronous". I'm trying to make a search engine in dart. Therefore i have to crawl through websites. For speeding up the process the program should crawl multiple websites at the same time.
– Velsem
Jan 2 at 12:30
"crawl multiple websites at the same time" that's fine. The adding/removing to the queue is very cheap and atomic and therefore no need for any specific precautions. Just add and remove to your desire. Is there any specific problem left?
– Günter Zöchbauer
Jan 2 at 12:32
Yes! :) "It is generally not allowed to modify the queue (add or remove entries) while an operation on the queue is being performed, for example during a call to forEach. Modifying the queue while it is being iterated will most likely break the iteration.". The crawler adds websites permanently
– Velsem
Jan 2 at 12:41
Then just copy the queue before iterating.queue.toList().forEach(...)You would need to elaborate how you would want conflicts to be handled.
– Günter Zöchbauer
Jan 2 at 12:44
add a comment |
I want to set up a queue of functions in Darts. Queuing should be asynchronous, allowing multiple functions to run concurrently. However, a maximum of three functions should be executed simultaneously. How can I achieve this?
I already tied working off a list but i am struggeling at adding a limit on same time running functions
List<String> queue = new List();
main(){
queue.add("...");
queue.add("...");
queue.add("...");
for(String q in queue){
await crawl(q);
}
}
crawl(String) async{
...
}
dart
asked Jan 2 at 12:14
VelsemVelsem
1
I want to set up a queue of functions in Darts. Queuing should be asynchronous, allowing multiple functions to run concurrently. However, a maximum of three functions should be executed simultaneously. How can I achieve this?
I already tied working off a list but i am struggeling at adding a limit on same time running functions
List<String> queue = new List();
main(){
queue.add("...");
queue.add("...");
queue.add("...");
for(String q in queue){
await crawl(q);
}
}
crawl(String) async{
...
}
dart
dart
asked Jan 2 at 12:14
VelsemVelsem
1
asked Jan 2 at 12:14
VelsemVelsem
1
asked Jan 2 at 12:14
VelsemVelsem
1
asked Jan 2 at 12:14
VelsemVelsem
1
asked Jan 2 at 12:14
VelsemVelsem
1
1
There is a queue class for Dart api.dartlang.org/stable/2.1.0/dart-collection/Queue-class.html and Dart is single-threaded so there is no way 2 functions can modify the queue at the same time. All methods of the queue are sync and therefore atomic. "Queuing should be asynchronous" not sure what that means.
– Günter Zöchbauer
Jan 2 at 12:21
"Queuing should be asynchronous". I'm trying to make a search engine in dart. Therefore i have to crawl through websites. For speeding up the process the program should crawl multiple websites at the same time.
– Velsem
Jan 2 at 12:30
"crawl multiple websites at the same time" that's fine. The adding/removing to the queue is very cheap and atomic and therefore no need for any specific precautions. Just add and remove to your desire. Is there any specific problem left?
– Günter Zöchbauer
Jan 2 at 12:32
Yes! :) "It is generally not allowed to modify the queue (add or remove entries) while an operation on the queue is being performed, for example during a call to forEach. Modifying the queue while it is being iterated will most likely break the iteration.". The crawler adds websites permanently
– Velsem
Jan 2 at 12:41
Then just copy the queue before iterating.queue.toList().forEach(...)You would need to elaborate how you would want conflicts to be handled.
– Günter Zöchbauer
Jan 2 at 12:44
add a comment |
There is a queue class for Dart api.dartlang.org/stable/2.1.0/dart-collection/Queue-class.html and Dart is single-threaded so there is no way 2 functions can modify the queue at the same time. All methods of the queue are sync and therefore atomic. "Queuing should be asynchronous" not sure what that means.
– Günter Zöchbauer
Jan 2 at 12:21
"Queuing should be asynchronous". I'm trying to make a search engine in dart. Therefore i have to crawl through websites. For speeding up the process the program should crawl multiple websites at the same time.
– Velsem
Jan 2 at 12:30
"crawl multiple websites at the same time" that's fine. The adding/removing to the queue is very cheap and atomic and therefore no need for any specific precautions. Just add and remove to your desire. Is there any specific problem left?
– Günter Zöchbauer
Jan 2 at 12:32
Yes! :) "It is generally not allowed to modify the queue (add or remove entries) while an operation on the queue is being performed, for example during a call to forEach. Modifying the queue while it is being iterated will most likely break the iteration.". The crawler adds websites permanently
– Velsem
Jan 2 at 12:41
Then just copy the queue before iterating.queue.toList().forEach(...)You would need to elaborate how you would want conflicts to be handled.
– Günter Zöchbauer
Jan 2 at 12:44
There is a queue class for Dart api.dartlang.org/stable/2.1.0/dart-collection/Queue-class.html and Dart is single-threaded so there is no way 2 functions can modify the queue at the same time. All methods of the queue are sync and therefore atomic. "Queuing should be asynchronous" not sure what that means.
– Günter Zöchbauer
Jan 2 at 12:21
There is a queue class for Dart api.dartlang.org/stable/2.1.0/dart-collection/Queue-class.html and Dart is single-threaded so there is no way 2 functions can modify the queue at the same time. All methods of the queue are sync and therefore atomic. "Queuing should be asynchronous" not sure what that means.
– Günter Zöchbauer
Jan 2 at 12:21
"Queuing should be asynchronous". I'm trying to make a search engine in dart. Therefore i have to crawl through websites. For speeding up the process the program should crawl multiple websites at the same time.
– Velsem
Jan 2 at 12:30
"Queuing should be asynchronous". I'm trying to make a search engine in dart. Therefore i have to crawl through websites. For speeding up the process the program should crawl multiple websites at the same time.
– Velsem
Jan 2 at 12:30
"crawl multiple websites at the same time" that's fine. The adding/removing to the queue is very cheap and atomic and therefore no need for any specific precautions. Just add and remove to your desire. Is there any specific problem left?
– Günter Zöchbauer
Jan 2 at 12:32
"crawl multiple websites at the same time" that's fine. The adding/removing to the queue is very cheap and atomic and therefore no need for any specific precautions. Just add and remove to your desire. Is there any specific problem left?
– Günter Zöchbauer
Jan 2 at 12:32
Yes! :) "It is generally not allowed to modify the queue (add or remove entries) while an operation on the queue is being performed, for example during a call to forEach. Modifying the queue while it is being iterated will most likely break the iteration.". The crawler adds websites permanently
– Velsem
Jan 2 at 12:41
Yes! :) "It is generally not allowed to modify the queue (add or remove entries) while an operation on the queue is being performed, for example during a call to forEach. Modifying the queue while it is being iterated will most likely break the iteration.". The crawler adds websites permanently
– Velsem
Jan 2 at 12:41
Then just copy the queue before iterating.
queue.toList().forEach(...) You would need to elaborate how you would want conflicts to be handled.– Günter Zöchbauer
Jan 2 at 12:44
Then just copy the queue before iterating.
queue.toList().forEach(...) You would need to elaborate how you would want conflicts to be handled.– Günter Zöchbauer
Jan 2 at 12:44
add a comment |
1 Answer
1
active
oldest
votes
I would use a queue:
import "dart:collection";
final queue = Queue<String>();
main() {
queue
..add("...")
..add("...")
..add("...");
while (queue.isNotEmpty) {
await crawl(queue.removeFirst());
}
}
crawl(String x) async {
.... queue.add(...) ...
}
This should work. It will not do concurrent crawling because await each operation. If you want concurrent crawling, I recommend being a little more clever. Look for worker pools or similar structures to ensure that you only have a certain number of operations running at the same time.
answered Jan 2 at 14:16
lrnlrn
11k1526
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54006188%2fhow-to-realize-a-queue-in-dart%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I would use a queue:
import "dart:collection";
final queue = Queue<String>();
main() {
queue
..add("...")
..add("...")
..add("...");
while (queue.isNotEmpty) {
await crawl(queue.removeFirst());
}
}
crawl(String x) async {
.... queue.add(...) ...
}
This should work. It will not do concurrent crawling because await each operation. If you want concurrent crawling, I recommend being a little more clever. Look for worker pools or similar structures to ensure that you only have a certain number of operations running at the same time.
answered Jan 2 at 14:16
lrnlrn
11k1526
add a comment |
I would use a queue:
import "dart:collection";
final queue = Queue<String>();
main() {
queue
..add("...")
..add("...")
..add("...");
while (queue.isNotEmpty) {
await crawl(queue.removeFirst());
}
}
crawl(String x) async {
.... queue.add(...) ...
}
This should work. It will not do concurrent crawling because await each operation. If you want concurrent crawling, I recommend being a little more clever. Look for worker pools or similar structures to ensure that you only have a certain number of operations running at the same time.
answered Jan 2 at 14:16
lrnlrn
11k1526
add a comment |
I would use a queue:
import "dart:collection";
final queue = Queue<String>();
main() {
queue
..add("...")
..add("...")
..add("...");
while (queue.isNotEmpty) {
await crawl(queue.removeFirst());
}
}
crawl(String x) async {
.... queue.add(...) ...
}
This should work. It will not do concurrent crawling because await each operation. If you want concurrent crawling, I recommend being a little more clever. Look for worker pools or similar structures to ensure that you only have a certain number of operations running at the same time.
answered Jan 2 at 14:16
lrnlrn
11k1526
I would use a queue:
import "dart:collection";
final queue = Queue<String>();
main() {
queue
..add("...")
..add("...")
..add("...");
while (queue.isNotEmpty) {
await crawl(queue.removeFirst());
}
}
crawl(String x) async {
.... queue.add(...) ...
}
This should work. It will not do concurrent crawling because await each operation. If you want concurrent crawling, I recommend being a little more clever. Look for worker pools or similar structures to ensure that you only have a certain number of operations running at the same time.
answered Jan 2 at 14:16
lrnlrn
11k1526
answered Jan 2 at 14:16
lrnlrn
11k1526
answered Jan 2 at 14:16
lrnlrn
11k1526
answered Jan 2 at 14:16
lrnlrn
11k1526
11k1526
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54006188%2fhow-to-realize-a-queue-in-dart%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
There is a queue class for Dart api.dartlang.org/stable/2.1.0/dart-collection/Queue-class.html and Dart is single-threaded so there is no way 2 functions can modify the queue at the same time. All methods of the queue are sync and therefore atomic. "Queuing should be asynchronous" not sure what that means.
– Günter Zöchbauer
Jan 2 at 12:21
"Queuing should be asynchronous". I'm trying to make a search engine in dart. Therefore i have to crawl through websites. For speeding up the process the program should crawl multiple websites at the same time.
– Velsem
Jan 2 at 12:30
"crawl multiple websites at the same time" that's fine. The adding/removing to the queue is very cheap and atomic and therefore no need for any specific precautions. Just add and remove to your desire. Is there any specific problem left?
– Günter Zöchbauer
Jan 2 at 12:32
Yes! :) "It is generally not allowed to modify the queue (add or remove entries) while an operation on the queue is being performed, for example during a call to forEach. Modifying the queue while it is being iterated will most likely break the iteration.". The crawler adds websites permanently
– Velsem
Jan 2 at 12:41
Then just copy the queue before iterating.
queue.toList().forEach(...)You would need to elaborate how you would want conflicts to be handled.– Günter Zöchbauer
Jan 2 at 12:44