Mixing
return early from range of a channel
Is it possible to implement something akin to yield from python in go? That is, I want a producer/consumer paradigm where the consumer can ignore values produced by the producer at any time without manifesting a memory leak.
In python I could have a producer that produces infinite values and the client can consume as many as they wish.
def producer():
i = 0
while True:
yield i
i += 1
def consumer():
for i in producer():
if i == 2:
return
The common advice for this is for the producer to return a channel that the client uses in a range expression, and the producer can put its values into the channel. This is fine as long as the client reads all the values in the channel, but if the client breaks control from the range loop early the producer channel will be left hanging forever using up memory.
package main
func yield() chan int {
out := make(chan int)
go func(){
out <- 1
out <- 2
close(out)
}()
return out
}
func test(){
for _ = range yield() {
// return
}
}
func main(){
for {
test()
}
}
If the return in test is uncommented this code will use memory without bound, because the channel returned by yield has a value that is waiting to be read and apparently neither the channel nor the goroutine can be garbage collected safely.
Some possible solutions (neither of which I like) are
Change
outinyieldto be a buffered channel with exactly the amount of space as values that will be produced. In that caseyieldcould be
func yield() chan int {
out := make(chan int, 2)
out <- 1
out <- 2
close(out)
return out
}
But this is suboptimal because I don't know in general how many items yield will produce. This style is similar to just returning an array.
- Have the
yieldmethod use a timeout for putting values into the channel, and if the timeout is reached then close the channel early. This makes assumptions about the execution time of the client, which seems like a bad idea.
go yield
asked Nov 21 '18 at 0:22
jonrjonr
7251519
add a comment |
Is it possible to implement something akin to yield from python in go? That is, I want a producer/consumer paradigm where the consumer can ignore values produced by the producer at any time without manifesting a memory leak.
In python I could have a producer that produces infinite values and the client can consume as many as they wish.
def producer():
i = 0
while True:
yield i
i += 1
def consumer():
for i in producer():
if i == 2:
return
The common advice for this is for the producer to return a channel that the client uses in a range expression, and the producer can put its values into the channel. This is fine as long as the client reads all the values in the channel, but if the client breaks control from the range loop early the producer channel will be left hanging forever using up memory.
package main
func yield() chan int {
out := make(chan int)
go func(){
out <- 1
out <- 2
close(out)
}()
return out
}
func test(){
for _ = range yield() {
// return
}
}
func main(){
for {
test()
}
}
If the return in test is uncommented this code will use memory without bound, because the channel returned by yield has a value that is waiting to be read and apparently neither the channel nor the goroutine can be garbage collected safely.
Some possible solutions (neither of which I like) are
Change
outinyieldto be a buffered channel with exactly the amount of space as values that will be produced. In that caseyieldcould be
func yield() chan int {
out := make(chan int, 2)
out <- 1
out <- 2
close(out)
return out
}
But this is suboptimal because I don't know in general how many items yield will produce. This style is similar to just returning an array.
- Have the
yieldmethod use a timeout for putting values into the channel, and if the timeout is reached then close the channel early. This makes assumptions about the execution time of the client, which seems like a bad idea.
go yield
asked Nov 21 '18 at 0:22
jonrjonr
7251519
Seems this was discussed at github.com/golang/go/issues/19702, and likely will not be fixed.
– jonr
Nov 21 '18 at 0:32
You can implement an iterator design pattern: it wouldn't be as elegant asyield, but semantically the same play.golang.org/p/7zw6iIwcDxH
– zerkms
Nov 21 '18 at 1:14
See Stopping Short in Go Concurrency Patterns: Pipelines and cancellation.
– ThunderCat
Nov 21 '18 at 1:39
add a comment |
Is it possible to implement something akin to yield from python in go? That is, I want a producer/consumer paradigm where the consumer can ignore values produced by the producer at any time without manifesting a memory leak.
In python I could have a producer that produces infinite values and the client can consume as many as they wish.
def producer():
i = 0
while True:
yield i
i += 1
def consumer():
for i in producer():
if i == 2:
return
The common advice for this is for the producer to return a channel that the client uses in a range expression, and the producer can put its values into the channel. This is fine as long as the client reads all the values in the channel, but if the client breaks control from the range loop early the producer channel will be left hanging forever using up memory.
package main
func yield() chan int {
out := make(chan int)
go func(){
out <- 1
out <- 2
close(out)
}()
return out
}
func test(){
for _ = range yield() {
// return
}
}
func main(){
for {
test()
}
}
If the return in test is uncommented this code will use memory without bound, because the channel returned by yield has a value that is waiting to be read and apparently neither the channel nor the goroutine can be garbage collected safely.
Some possible solutions (neither of which I like) are
Change
outinyieldto be a buffered channel with exactly the amount of space as values that will be produced. In that caseyieldcould be
func yield() chan int {
out := make(chan int, 2)
out <- 1
out <- 2
close(out)
return out
}
But this is suboptimal because I don't know in general how many items yield will produce. This style is similar to just returning an array.
- Have the
yieldmethod use a timeout for putting values into the channel, and if the timeout is reached then close the channel early. This makes assumptions about the execution time of the client, which seems like a bad idea.
go yield
asked Nov 21 '18 at 0:22
jonrjonr
7251519
Is it possible to implement something akin to yield from python in go? That is, I want a producer/consumer paradigm where the consumer can ignore values produced by the producer at any time without manifesting a memory leak.
In python I could have a producer that produces infinite values and the client can consume as many as they wish.
def producer():
i = 0
while True:
yield i
i += 1
def consumer():
for i in producer():
if i == 2:
return
The common advice for this is for the producer to return a channel that the client uses in a range expression, and the producer can put its values into the channel. This is fine as long as the client reads all the values in the channel, but if the client breaks control from the range loop early the producer channel will be left hanging forever using up memory.
package main
func yield() chan int {
out := make(chan int)
go func(){
out <- 1
out <- 2
close(out)
}()
return out
}
func test(){
for _ = range yield() {
// return
}
}
func main(){
for {
test()
}
}
If the return in test is uncommented this code will use memory without bound, because the channel returned by yield has a value that is waiting to be read and apparently neither the channel nor the goroutine can be garbage collected safely.
Some possible solutions (neither of which I like) are
Change
outinyieldto be a buffered channel with exactly the amount of space as values that will be produced. In that caseyieldcould be
func yield() chan int {
out := make(chan int, 2)
out <- 1
out <- 2
close(out)
return out
}
But this is suboptimal because I don't know in general how many items yield will produce. This style is similar to just returning an array.
- Have the
yieldmethod use a timeout for putting values into the channel, and if the timeout is reached then close the channel early. This makes assumptions about the execution time of the client, which seems like a bad idea.
go yield
go yield
asked Nov 21 '18 at 0:22
jonrjonr
7251519
asked Nov 21 '18 at 0:22
jonrjonr
7251519
asked Nov 21 '18 at 0:22
jonrjonr
7251519
asked Nov 21 '18 at 0:22
jonrjonr
7251519
asked Nov 21 '18 at 0:22
jonrjonr
7251519
7251519
Seems this was discussed at github.com/golang/go/issues/19702, and likely will not be fixed.
– jonr
Nov 21 '18 at 0:32
You can implement an iterator design pattern: it wouldn't be as elegant asyield, but semantically the same play.golang.org/p/7zw6iIwcDxH
– zerkms
Nov 21 '18 at 1:14
See Stopping Short in Go Concurrency Patterns: Pipelines and cancellation.
– ThunderCat
Nov 21 '18 at 1:39
add a comment |
Seems this was discussed at github.com/golang/go/issues/19702, and likely will not be fixed.
– jonr
Nov 21 '18 at 0:32
You can implement an iterator design pattern: it wouldn't be as elegant asyield, but semantically the same play.golang.org/p/7zw6iIwcDxH
– zerkms
Nov 21 '18 at 1:14
See Stopping Short in Go Concurrency Patterns: Pipelines and cancellation.
– ThunderCat
Nov 21 '18 at 1:39
Seems this was discussed at github.com/golang/go/issues/19702, and likely will not be fixed.
– jonr
Nov 21 '18 at 0:32
Seems this was discussed at github.com/golang/go/issues/19702, and likely will not be fixed.
– jonr
Nov 21 '18 at 0:32
You can implement an iterator design pattern: it wouldn't be as elegant as
yield, but semantically the same play.golang.org/p/7zw6iIwcDxH– zerkms
Nov 21 '18 at 1:14
You can implement an iterator design pattern: it wouldn't be as elegant as
yield, but semantically the same play.golang.org/p/7zw6iIwcDxH– zerkms
Nov 21 '18 at 1:14
See Stopping Short in Go Concurrency Patterns: Pipelines and cancellation.
– ThunderCat
Nov 21 '18 at 1:39
See Stopping Short in Go Concurrency Patterns: Pipelines and cancellation.
– ThunderCat
Nov 21 '18 at 1:39
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53403593%2freturn-early-from-range-of-a-channel%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53403593%2freturn-early-from-range-of-a-channel%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Seems this was discussed at github.com/golang/go/issues/19702, and likely will not be fixed.
– jonr
Nov 21 '18 at 0:32
You can implement an iterator design pattern: it wouldn't be as elegant as
yield, but semantically the same play.golang.org/p/7zw6iIwcDxH– zerkms
Nov 21 '18 at 1:14
See Stopping Short in Go Concurrency Patterns: Pipelines and cancellation.
– ThunderCat
Nov 21 '18 at 1:39