Mixing
Java Kafka adminClient topic configuration. Configuration values are overwritten
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
0
While trying to configure a newly created kafka topic, using java kafka adminClient, values are overwritten.
I have tried to set the same topic configuration using console commands and it works. Unfortunately when I try through Java code some values collide and are overwritten.
ConfigResource resource = new ConfigResource(ConfigResource.Type.TOPIC, topicName);
Map<ConfigResource, Config> updateConfig = new HashMap<>();
// update retention Bytes for this topic
ConfigEntry retentionBytesEntry = new ConfigEntry(TopicConfig.RETENTION_BYTES_CONFIG, String.valueOf(retentionBytes));
updateConfig.put(resource, new Config(Collections.singleton(retentionBytesEntry)));
// update retention ms for this topic
ConfigEntry retentionMsEntry = new ConfigEntry(TopicConfig.RETENTION_MS_CONFIG, String.valueOf(retentionMs));
updateConfig.put(resource, new Config(Collections.singleton(retentionMsEntry)));
// update segment Bytes for this topic
ConfigEntry segmentBytesEntry = new ConfigEntry(TopicConfig.SEGMENT_BYTES_CONFIG, String.valueOf(segmentbytes));
updateConfig.put(resource, new Config(Collections.singleton(segmentBytesEntry)));
// update segment ms for this topic
ConfigEntry segmentMsEntry = new ConfigEntry(TopicConfig.SEGMENT_MS_CONFIG, String.valueOf(segmentMs));
updateConfig.put(resource, new Config(Collections.singleton(segmentMsEntry)));
// Update the configuration
client.alterConfigs(updateConfig);
I expect the topic to have all given configuration values correctly.
java apache-kafka
edited Jan 3 at 15:20
Nishu Tayal
13k73584
asked Jan 3 at 15:02
Athanassios MparekasAthanassios Mparekas
31
add a comment |
0
While trying to configure a newly created kafka topic, using java kafka adminClient, values are overwritten.
I have tried to set the same topic configuration using console commands and it works. Unfortunately when I try through Java code some values collide and are overwritten.
ConfigResource resource = new ConfigResource(ConfigResource.Type.TOPIC, topicName);
Map<ConfigResource, Config> updateConfig = new HashMap<>();
// update retention Bytes for this topic
ConfigEntry retentionBytesEntry = new ConfigEntry(TopicConfig.RETENTION_BYTES_CONFIG, String.valueOf(retentionBytes));
updateConfig.put(resource, new Config(Collections.singleton(retentionBytesEntry)));
// update retention ms for this topic
ConfigEntry retentionMsEntry = new ConfigEntry(TopicConfig.RETENTION_MS_CONFIG, String.valueOf(retentionMs));
updateConfig.put(resource, new Config(Collections.singleton(retentionMsEntry)));
// update segment Bytes for this topic
ConfigEntry segmentBytesEntry = new ConfigEntry(TopicConfig.SEGMENT_BYTES_CONFIG, String.valueOf(segmentbytes));
updateConfig.put(resource, new Config(Collections.singleton(segmentBytesEntry)));
// update segment ms for this topic
ConfigEntry segmentMsEntry = new ConfigEntry(TopicConfig.SEGMENT_MS_CONFIG, String.valueOf(segmentMs));
updateConfig.put(resource, new Config(Collections.singleton(segmentMsEntry)));
// Update the configuration
client.alterConfigs(updateConfig);
I expect the topic to have all given configuration values correctly.
java apache-kafka
edited Jan 3 at 15:20
Nishu Tayal
13k73584
asked Jan 3 at 15:02
Athanassios MparekasAthanassios Mparekas
31
which configurations are overwritten?
– Nishu Tayal
Jan 3 at 15:30
what do you mean by "configurations are overwritten"?
– ppatierno
Jan 3 at 15:34
I need retention.ms and segment.ms to have a specific value (same one). But my code sets the proper value only to the one that is set at the end of the map and overwrittes the one that is set first.
– Athanassios Mparekas
Jan 3 at 15:36
both retentionMs and segmentMs have the same value (60000) but only segment.ms is configured a 60000. Retentionms gets a default value. If I swap those setting and the reverse is happening.
– Athanassios Mparekas
Jan 3 at 15:39
add a comment |
0
0
0
While trying to configure a newly created kafka topic, using java kafka adminClient, values are overwritten.
I have tried to set the same topic configuration using console commands and it works. Unfortunately when I try through Java code some values collide and are overwritten.
ConfigResource resource = new ConfigResource(ConfigResource.Type.TOPIC, topicName);
Map<ConfigResource, Config> updateConfig = new HashMap<>();
// update retention Bytes for this topic
ConfigEntry retentionBytesEntry = new ConfigEntry(TopicConfig.RETENTION_BYTES_CONFIG, String.valueOf(retentionBytes));
updateConfig.put(resource, new Config(Collections.singleton(retentionBytesEntry)));
// update retention ms for this topic
ConfigEntry retentionMsEntry = new ConfigEntry(TopicConfig.RETENTION_MS_CONFIG, String.valueOf(retentionMs));
updateConfig.put(resource, new Config(Collections.singleton(retentionMsEntry)));
// update segment Bytes for this topic
ConfigEntry segmentBytesEntry = new ConfigEntry(TopicConfig.SEGMENT_BYTES_CONFIG, String.valueOf(segmentbytes));
updateConfig.put(resource, new Config(Collections.singleton(segmentBytesEntry)));
// update segment ms for this topic
ConfigEntry segmentMsEntry = new ConfigEntry(TopicConfig.SEGMENT_MS_CONFIG, String.valueOf(segmentMs));
updateConfig.put(resource, new Config(Collections.singleton(segmentMsEntry)));
// Update the configuration
client.alterConfigs(updateConfig);
I expect the topic to have all given configuration values correctly.
java apache-kafka
edited Jan 3 at 15:20
Nishu Tayal
13k73584
asked Jan 3 at 15:02
Athanassios MparekasAthanassios Mparekas
31
While trying to configure a newly created kafka topic, using java kafka adminClient, values are overwritten.
I have tried to set the same topic configuration using console commands and it works. Unfortunately when I try through Java code some values collide and are overwritten.
ConfigResource resource = new ConfigResource(ConfigResource.Type.TOPIC, topicName);
Map<ConfigResource, Config> updateConfig = new HashMap<>();
// update retention Bytes for this topic
ConfigEntry retentionBytesEntry = new ConfigEntry(TopicConfig.RETENTION_BYTES_CONFIG, String.valueOf(retentionBytes));
updateConfig.put(resource, new Config(Collections.singleton(retentionBytesEntry)));
// update retention ms for this topic
ConfigEntry retentionMsEntry = new ConfigEntry(TopicConfig.RETENTION_MS_CONFIG, String.valueOf(retentionMs));
updateConfig.put(resource, new Config(Collections.singleton(retentionMsEntry)));
// update segment Bytes for this topic
ConfigEntry segmentBytesEntry = new ConfigEntry(TopicConfig.SEGMENT_BYTES_CONFIG, String.valueOf(segmentbytes));
updateConfig.put(resource, new Config(Collections.singleton(segmentBytesEntry)));
// update segment ms for this topic
ConfigEntry segmentMsEntry = new ConfigEntry(TopicConfig.SEGMENT_MS_CONFIG, String.valueOf(segmentMs));
updateConfig.put(resource, new Config(Collections.singleton(segmentMsEntry)));
// Update the configuration
client.alterConfigs(updateConfig);
I expect the topic to have all given configuration values correctly.
java apache-kafka
java apache-kafka
edited Jan 3 at 15:20
Nishu Tayal
13k73584
asked Jan 3 at 15:02
Athanassios MparekasAthanassios Mparekas
31
edited Jan 3 at 15:20
Nishu Tayal
13k73584
asked Jan 3 at 15:02
Athanassios MparekasAthanassios Mparekas
31
edited Jan 3 at 15:20
Nishu Tayal
13k73584
edited Jan 3 at 15:20
Nishu Tayal
13k73584
edited Jan 3 at 15:20
Nishu Tayal
13k73584
13k73584
asked Jan 3 at 15:02
Athanassios MparekasAthanassios Mparekas
31
asked Jan 3 at 15:02
Athanassios MparekasAthanassios Mparekas
31
asked Jan 3 at 15:02
Athanassios MparekasAthanassios Mparekas
31
31
which configurations are overwritten?
– Nishu Tayal
Jan 3 at 15:30
what do you mean by "configurations are overwritten"?
– ppatierno
Jan 3 at 15:34
I need retention.ms and segment.ms to have a specific value (same one). But my code sets the proper value only to the one that is set at the end of the map and overwrittes the one that is set first.
– Athanassios Mparekas
Jan 3 at 15:36
both retentionMs and segmentMs have the same value (60000) but only segment.ms is configured a 60000. Retentionms gets a default value. If I swap those setting and the reverse is happening.
– Athanassios Mparekas
Jan 3 at 15:39
add a comment |
which configurations are overwritten?
– Nishu Tayal
Jan 3 at 15:30
what do you mean by "configurations are overwritten"?
– ppatierno
Jan 3 at 15:34
I need retention.ms and segment.ms to have a specific value (same one). But my code sets the proper value only to the one that is set at the end of the map and overwrittes the one that is set first.
– Athanassios Mparekas
Jan 3 at 15:36
both retentionMs and segmentMs have the same value (60000) but only segment.ms is configured a 60000. Retentionms gets a default value. If I swap those setting and the reverse is happening.
– Athanassios Mparekas
Jan 3 at 15:39
which configurations are overwritten?
– Nishu Tayal
Jan 3 at 15:30
which configurations are overwritten?
– Nishu Tayal
Jan 3 at 15:30
what do you mean by "configurations are overwritten"?
– ppatierno
Jan 3 at 15:34
what do you mean by "configurations are overwritten"?
– ppatierno
Jan 3 at 15:34
I need retention.ms and segment.ms to have a specific value (same one). But my code sets the proper value only to the one that is set at the end of the map and overwrittes the one that is set first.
– Athanassios Mparekas
Jan 3 at 15:36
I need retention.ms and segment.ms to have a specific value (same one). But my code sets the proper value only to the one that is set at the end of the map and overwrittes the one that is set first.
– Athanassios Mparekas
Jan 3 at 15:36
both retentionMs and segmentMs have the same value (60000) but only segment.ms is configured a 60000. Retentionms gets a default value. If I swap those setting and the reverse is happening.
– Athanassios Mparekas
Jan 3 at 15:39
both retentionMs and segmentMs have the same value (60000) but only segment.ms is configured a 60000. Retentionms gets a default value. If I swap those setting and the reverse is happening.
– Athanassios Mparekas
Jan 3 at 15:39
add a comment |
1 Answer
1
active
oldest
votes
0
Your logic is not working correctly because you call Map.put() several times with the same key. Hence only the last entry is kept.
The correct way to specify multiple topic configurations is to add them in the ConfigEntry object. Only after add the ConfigEntry to the Map.
For example:
// Your Topic Resource
ConfigResource cr = new ConfigResource(Type.TOPIC, "mytopic");
// Create all your configurations
Collection<ConfigEntry> entries = new ArrayList<>();
entries.add(new ConfigEntry(TopicConfig.SEGMENT_BYTES_CONFIG, String.valueOf(segmentbytes)));
entries.add(new ConfigEntry(TopicConfig.RETENTION_BYTES_CONFIG, String.valueOf(retentionBytes)));
...
// Create the Map
Config config = new Config(entries);
Map<ConfigResource, Config> configs = new HashMap<>();
configs.put(cr, config);
// Call alterConfigs()
admin.alterConfigs(configs);
answered Jan 3 at 16:45
Mickael MaisonMickael Maison
8,72842831
Yes you are correct :) thank you!
– Athanassios Mparekas
Jan 3 at 18:23
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54024829%2fjava-kafka-adminclient-topic-configuration-configuration-values-are-overwritten%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
Your logic is not working correctly because you call Map.put() several times with the same key. Hence only the last entry is kept.
The correct way to specify multiple topic configurations is to add them in the ConfigEntry object. Only after add the ConfigEntry to the Map.
For example:
// Your Topic Resource
ConfigResource cr = new ConfigResource(Type.TOPIC, "mytopic");
// Create all your configurations
Collection<ConfigEntry> entries = new ArrayList<>();
entries.add(new ConfigEntry(TopicConfig.SEGMENT_BYTES_CONFIG, String.valueOf(segmentbytes)));
entries.add(new ConfigEntry(TopicConfig.RETENTION_BYTES_CONFIG, String.valueOf(retentionBytes)));
...
// Create the Map
Config config = new Config(entries);
Map<ConfigResource, Config> configs = new HashMap<>();
configs.put(cr, config);
// Call alterConfigs()
admin.alterConfigs(configs);
answered Jan 3 at 16:45
Mickael MaisonMickael Maison
8,72842831
Yes you are correct :) thank you!
– Athanassios Mparekas
Jan 3 at 18:23
add a comment |
0
Your logic is not working correctly because you call Map.put() several times with the same key. Hence only the last entry is kept.
The correct way to specify multiple topic configurations is to add them in the ConfigEntry object. Only after add the ConfigEntry to the Map.
For example:
// Your Topic Resource
ConfigResource cr = new ConfigResource(Type.TOPIC, "mytopic");
// Create all your configurations
Collection<ConfigEntry> entries = new ArrayList<>();
entries.add(new ConfigEntry(TopicConfig.SEGMENT_BYTES_CONFIG, String.valueOf(segmentbytes)));
entries.add(new ConfigEntry(TopicConfig.RETENTION_BYTES_CONFIG, String.valueOf(retentionBytes)));
...
// Create the Map
Config config = new Config(entries);
Map<ConfigResource, Config> configs = new HashMap<>();
configs.put(cr, config);
// Call alterConfigs()
admin.alterConfigs(configs);
answered Jan 3 at 16:45
Mickael MaisonMickael Maison
8,72842831
Yes you are correct :) thank you!
– Athanassios Mparekas
Jan 3 at 18:23
add a comment |
0
0
0
Your logic is not working correctly because you call Map.put() several times with the same key. Hence only the last entry is kept.
The correct way to specify multiple topic configurations is to add them in the ConfigEntry object. Only after add the ConfigEntry to the Map.
For example:
// Your Topic Resource
ConfigResource cr = new ConfigResource(Type.TOPIC, "mytopic");
// Create all your configurations
Collection<ConfigEntry> entries = new ArrayList<>();
entries.add(new ConfigEntry(TopicConfig.SEGMENT_BYTES_CONFIG, String.valueOf(segmentbytes)));
entries.add(new ConfigEntry(TopicConfig.RETENTION_BYTES_CONFIG, String.valueOf(retentionBytes)));
...
// Create the Map
Config config = new Config(entries);
Map<ConfigResource, Config> configs = new HashMap<>();
configs.put(cr, config);
// Call alterConfigs()
admin.alterConfigs(configs);
answered Jan 3 at 16:45
Mickael MaisonMickael Maison
8,72842831
Your logic is not working correctly because you call Map.put() several times with the same key. Hence only the last entry is kept.
The correct way to specify multiple topic configurations is to add them in the ConfigEntry object. Only after add the ConfigEntry to the Map.
For example:
// Your Topic Resource
ConfigResource cr = new ConfigResource(Type.TOPIC, "mytopic");
// Create all your configurations
Collection<ConfigEntry> entries = new ArrayList<>();
entries.add(new ConfigEntry(TopicConfig.SEGMENT_BYTES_CONFIG, String.valueOf(segmentbytes)));
entries.add(new ConfigEntry(TopicConfig.RETENTION_BYTES_CONFIG, String.valueOf(retentionBytes)));
...
// Create the Map
Config config = new Config(entries);
Map<ConfigResource, Config> configs = new HashMap<>();
configs.put(cr, config);
// Call alterConfigs()
admin.alterConfigs(configs);
answered Jan 3 at 16:45
Mickael MaisonMickael Maison
8,72842831
answered Jan 3 at 16:45
Mickael MaisonMickael Maison
8,72842831
answered Jan 3 at 16:45
Mickael MaisonMickael Maison
8,72842831
answered Jan 3 at 16:45
Mickael MaisonMickael Maison
8,72842831
8,72842831
Yes you are correct :) thank you!
– Athanassios Mparekas
Jan 3 at 18:23
add a comment |
Yes you are correct :) thank you!
– Athanassios Mparekas
Jan 3 at 18:23
Yes you are correct :) thank you!
– Athanassios Mparekas
Jan 3 at 18:23
Yes you are correct :) thank you!
– Athanassios Mparekas
Jan 3 at 18:23
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54024829%2fjava-kafka-adminclient-topic-configuration-configuration-values-are-overwritten%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
which configurations are overwritten?
– Nishu Tayal
Jan 3 at 15:30
what do you mean by "configurations are overwritten"?
– ppatierno
Jan 3 at 15:34
I need retention.ms and segment.ms to have a specific value (same one). But my code sets the proper value only to the one that is set at the end of the map and overwrittes the one that is set first.
– Athanassios Mparekas
Jan 3 at 15:36
both retentionMs and segmentMs have the same value (60000) but only segment.ms is configured a 60000. Retentionms gets a default value. If I swap those setting and the reverse is happening.
– Athanassios Mparekas
Jan 3 at 15:39