Mixing
Trying to setup consumer and producer in Kafka
I'm trying to make a simple producer-consumer flow over Kafka, using node-rdkafka
I'm using debug: 'all' mode, this is what I get from the logs:
Producer: test [0]: MessageSet with 1 message(s) delivered
Consumer: Fetch topic test [0] at offset 38 (v2)
The fact that the consumer is changing offset when message is produced making me believe the connection with the broker is setup and authenticated correctly.
However, for some reason I don't get the message itself in the consumer.
This event is never called:
consumer.on('data', function(m) {
console.log("consumed", m)
});
I created a demo project for testing, you need to have a Kafka broker that supports SASL_SSL protocol in order to use it:
https://github.com/guysegal/kafka-example
Specifically, this is the consumer code:
https://github.com/guysegal/kafka-example/blob/master/src/consumer.ts
and the producer code:
https://github.com/guysegal/kafka-example/blob/master/src/producer.ts
javascript node.js apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 20 '18 at 15:03
cricket_007
80.4k1142110
asked Nov 20 '18 at 6:27
Guy SegalGuy Segal
1951520
add a comment |
I'm trying to make a simple producer-consumer flow over Kafka, using node-rdkafka
I'm using debug: 'all' mode, this is what I get from the logs:
Producer: test [0]: MessageSet with 1 message(s) delivered
Consumer: Fetch topic test [0] at offset 38 (v2)
The fact that the consumer is changing offset when message is produced making me believe the connection with the broker is setup and authenticated correctly.
However, for some reason I don't get the message itself in the consumer.
This event is never called:
consumer.on('data', function(m) {
console.log("consumed", m)
});
I created a demo project for testing, you need to have a Kafka broker that supports SASL_SSL protocol in order to use it:
https://github.com/guysegal/kafka-example
Specifically, this is the consumer code:
https://github.com/guysegal/kafka-example/blob/master/src/consumer.ts
and the producer code:
https://github.com/guysegal/kafka-example/blob/master/src/producer.ts
javascript node.js apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 20 '18 at 15:03
cricket_007
80.4k1142110
asked Nov 20 '18 at 6:27
Guy SegalGuy Segal
1951520
add a comment |
I'm trying to make a simple producer-consumer flow over Kafka, using node-rdkafka
I'm using debug: 'all' mode, this is what I get from the logs:
Producer: test [0]: MessageSet with 1 message(s) delivered
Consumer: Fetch topic test [0] at offset 38 (v2)
The fact that the consumer is changing offset when message is produced making me believe the connection with the broker is setup and authenticated correctly.
However, for some reason I don't get the message itself in the consumer.
This event is never called:
consumer.on('data', function(m) {
console.log("consumed", m)
});
I created a demo project for testing, you need to have a Kafka broker that supports SASL_SSL protocol in order to use it:
https://github.com/guysegal/kafka-example
Specifically, this is the consumer code:
https://github.com/guysegal/kafka-example/blob/master/src/consumer.ts
and the producer code:
https://github.com/guysegal/kafka-example/blob/master/src/producer.ts
javascript node.js apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 20 '18 at 15:03
cricket_007
80.4k1142110
asked Nov 20 '18 at 6:27
Guy SegalGuy Segal
1951520
I'm trying to make a simple producer-consumer flow over Kafka, using node-rdkafka
I'm using debug: 'all' mode, this is what I get from the logs:
Producer: test [0]: MessageSet with 1 message(s) delivered
Consumer: Fetch topic test [0] at offset 38 (v2)
The fact that the consumer is changing offset when message is produced making me believe the connection with the broker is setup and authenticated correctly.
However, for some reason I don't get the message itself in the consumer.
This event is never called:
consumer.on('data', function(m) {
console.log("consumed", m)
});
I created a demo project for testing, you need to have a Kafka broker that supports SASL_SSL protocol in order to use it:
https://github.com/guysegal/kafka-example
Specifically, this is the consumer code:
https://github.com/guysegal/kafka-example/blob/master/src/consumer.ts
and the producer code:
https://github.com/guysegal/kafka-example/blob/master/src/producer.ts
javascript node.js apache-kafka kafka-consumer-api kafka-producer-api
javascript node.js apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 20 '18 at 15:03
cricket_007
80.4k1142110
asked Nov 20 '18 at 6:27
Guy SegalGuy Segal
1951520
edited Nov 20 '18 at 15:03
cricket_007
80.4k1142110
asked Nov 20 '18 at 6:27
Guy SegalGuy Segal
1951520
edited Nov 20 '18 at 15:03
cricket_007
80.4k1142110
edited Nov 20 '18 at 15:03
cricket_007
80.4k1142110
edited Nov 20 '18 at 15:03
cricket_007
80.4k1142110
80.4k1142110
asked Nov 20 '18 at 6:27
Guy SegalGuy Segal
1951520
asked Nov 20 '18 at 6:27
Guy SegalGuy Segal
1951520
asked Nov 20 '18 at 6:27
Guy SegalGuy Segal
1951520
1951520
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.
auto.offset.reset
This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.
The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.
While in case of 'earliest' consumer starts from first available message on topic.
answered Nov 20 '18 at 10:23
bittubittu
342110
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53387406%2ftrying-to-setup-consumer-and-producer-in-kafka%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.
auto.offset.reset
This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.
The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.
While in case of 'earliest' consumer starts from first available message on topic.
answered Nov 20 '18 at 10:23
bittubittu
342110
add a comment |
You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.
auto.offset.reset
This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.
The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.
While in case of 'earliest' consumer starts from first available message on topic.
answered Nov 20 '18 at 10:23
bittubittu
342110
add a comment |
You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.
auto.offset.reset
This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.
The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.
While in case of 'earliest' consumer starts from first available message on topic.
answered Nov 20 '18 at 10:23
bittubittu
342110
You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.
auto.offset.reset
This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.
The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.
While in case of 'earliest' consumer starts from first available message on topic.
answered Nov 20 '18 at 10:23
bittubittu
342110
edited Nov 20 '18 at 10:33
answered Nov 20 '18 at 10:23
bittubittu
342110
answered Nov 20 '18 at 10:23
bittubittu
342110
answered Nov 20 '18 at 10:23
bittubittu
342110
342110
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53387406%2ftrying-to-setup-consumer-and-producer-in-kafka%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
