Mixing
Is there a way to connect to multiple databases in multiple hosts using Kafka Connect?
I have a need to get data from Informix database using Kafka Connect. The scenario is this - I have 50 Informix Databases residing in 50 hosts. What I have understood by reading from Kafka connect is that we need to install the Kafka connect in each hosts to get the data from the database residing in that host. My question is this - Is there a way in which I can create the connectors centrally for these 50 hosts instead of installing into each of them and pull data from the databases?
apache-kafka informix apache-kafka-connect
edited Nov 21 '18 at 22:57
cricket_007
82.3k1143111
asked Nov 21 '18 at 18:35
Anish KurianAnish Kurian
53117
add a comment |
I have a need to get data from Informix database using Kafka Connect. The scenario is this - I have 50 Informix Databases residing in 50 hosts. What I have understood by reading from Kafka connect is that we need to install the Kafka connect in each hosts to get the data from the database residing in that host. My question is this - Is there a way in which I can create the connectors centrally for these 50 hosts instead of installing into each of them and pull data from the databases?
apache-kafka informix apache-kafka-connect
edited Nov 21 '18 at 22:57
cricket_007
82.3k1143111
asked Nov 21 '18 at 18:35
Anish KurianAnish Kurian
53117
1
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 '18 at 21:09
add a comment |
I have a need to get data from Informix database using Kafka Connect. The scenario is this - I have 50 Informix Databases residing in 50 hosts. What I have understood by reading from Kafka connect is that we need to install the Kafka connect in each hosts to get the data from the database residing in that host. My question is this - Is there a way in which I can create the connectors centrally for these 50 hosts instead of installing into each of them and pull data from the databases?
apache-kafka informix apache-kafka-connect
edited Nov 21 '18 at 22:57
cricket_007
82.3k1143111
asked Nov 21 '18 at 18:35
Anish KurianAnish Kurian
53117
I have a need to get data from Informix database using Kafka Connect. The scenario is this - I have 50 Informix Databases residing in 50 hosts. What I have understood by reading from Kafka connect is that we need to install the Kafka connect in each hosts to get the data from the database residing in that host. My question is this - Is there a way in which I can create the connectors centrally for these 50 hosts instead of installing into each of them and pull data from the databases?
apache-kafka informix apache-kafka-connect
apache-kafka informix apache-kafka-connect
edited Nov 21 '18 at 22:57
cricket_007
82.3k1143111
asked Nov 21 '18 at 18:35
Anish KurianAnish Kurian
53117
edited Nov 21 '18 at 22:57
cricket_007
82.3k1143111
asked Nov 21 '18 at 18:35
Anish KurianAnish Kurian
53117
edited Nov 21 '18 at 22:57
cricket_007
82.3k1143111
edited Nov 21 '18 at 22:57
cricket_007
82.3k1143111
edited Nov 21 '18 at 22:57
cricket_007
82.3k1143111
82.3k1143111
asked Nov 21 '18 at 18:35
Anish KurianAnish Kurian
53117
asked Nov 21 '18 at 18:35
Anish KurianAnish Kurian
53117
asked Nov 21 '18 at 18:35
Anish KurianAnish Kurian
53117
53117
1
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 '18 at 21:09
add a comment |
1
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 '18 at 21:09
1
1
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 '18 at 21:09
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 '18 at 21:09
add a comment |
2 Answers
2
active
oldest
votes
Kafka Connect JDBC does not have to run on the database, just as other JDBC clients don't, so you can a have a Kafka Connect cluster be larger or smaller than your database pool.
Informix seems to have a thing called "CDC Replication Engine for Kafka", however, which might be something worth looking into, as CDC overall causes less load on the database
answered Nov 21 '18 at 22:56
cricket_007cricket_007
82.3k1143111
add a comment |
You don’t need any additional software installation on the system where Informix server is running.I am not fully clear about the question or the type of operation you are plan to do. If you are planning to setup a real time replication type of scenario, then you may have to invoke CDC API. Then one-time setup of CDC API at server is needed, then this APIs can be invoked using any Informix database driver API. If you are plan to read existing data from table(s) and pump into Kafka topic, then no need of any additional setup at server side. You could connect to all 50 database server(s) from a single program (remotely) and then pump those records to the Kafka topic(s). Base on the program language you are using you may choose Informix database driver.
answered Nov 27 '18 at 6:06
SatyanSatyan
27114
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53418543%2fis-there-a-way-to-connect-to-multiple-databases-in-multiple-hosts-using-kafka-co%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Kafka Connect JDBC does not have to run on the database, just as other JDBC clients don't, so you can a have a Kafka Connect cluster be larger or smaller than your database pool.
Informix seems to have a thing called "CDC Replication Engine for Kafka", however, which might be something worth looking into, as CDC overall causes less load on the database
answered Nov 21 '18 at 22:56
cricket_007cricket_007
82.3k1143111
add a comment |
Kafka Connect JDBC does not have to run on the database, just as other JDBC clients don't, so you can a have a Kafka Connect cluster be larger or smaller than your database pool.
Informix seems to have a thing called "CDC Replication Engine for Kafka", however, which might be something worth looking into, as CDC overall causes less load on the database
answered Nov 21 '18 at 22:56
cricket_007cricket_007
82.3k1143111
add a comment |
Kafka Connect JDBC does not have to run on the database, just as other JDBC clients don't, so you can a have a Kafka Connect cluster be larger or smaller than your database pool.
Informix seems to have a thing called "CDC Replication Engine for Kafka", however, which might be something worth looking into, as CDC overall causes less load on the database
answered Nov 21 '18 at 22:56
cricket_007cricket_007
82.3k1143111
Kafka Connect JDBC does not have to run on the database, just as other JDBC clients don't, so you can a have a Kafka Connect cluster be larger or smaller than your database pool.
Informix seems to have a thing called "CDC Replication Engine for Kafka", however, which might be something worth looking into, as CDC overall causes less load on the database
answered Nov 21 '18 at 22:56
cricket_007cricket_007
82.3k1143111
answered Nov 21 '18 at 22:56
cricket_007cricket_007
82.3k1143111
answered Nov 21 '18 at 22:56
cricket_007cricket_007
82.3k1143111
answered Nov 21 '18 at 22:56
cricket_007cricket_007
82.3k1143111
82.3k1143111
add a comment |
add a comment |
You don’t need any additional software installation on the system where Informix server is running.I am not fully clear about the question or the type of operation you are plan to do. If you are planning to setup a real time replication type of scenario, then you may have to invoke CDC API. Then one-time setup of CDC API at server is needed, then this APIs can be invoked using any Informix database driver API. If you are plan to read existing data from table(s) and pump into Kafka topic, then no need of any additional setup at server side. You could connect to all 50 database server(s) from a single program (remotely) and then pump those records to the Kafka topic(s). Base on the program language you are using you may choose Informix database driver.
answered Nov 27 '18 at 6:06
SatyanSatyan
27114
add a comment |
You don’t need any additional software installation on the system where Informix server is running.I am not fully clear about the question or the type of operation you are plan to do. If you are planning to setup a real time replication type of scenario, then you may have to invoke CDC API. Then one-time setup of CDC API at server is needed, then this APIs can be invoked using any Informix database driver API. If you are plan to read existing data from table(s) and pump into Kafka topic, then no need of any additional setup at server side. You could connect to all 50 database server(s) from a single program (remotely) and then pump those records to the Kafka topic(s). Base on the program language you are using you may choose Informix database driver.
answered Nov 27 '18 at 6:06
SatyanSatyan
27114
add a comment |
You don’t need any additional software installation on the system where Informix server is running.I am not fully clear about the question or the type of operation you are plan to do. If you are planning to setup a real time replication type of scenario, then you may have to invoke CDC API. Then one-time setup of CDC API at server is needed, then this APIs can be invoked using any Informix database driver API. If you are plan to read existing data from table(s) and pump into Kafka topic, then no need of any additional setup at server side. You could connect to all 50 database server(s) from a single program (remotely) and then pump those records to the Kafka topic(s). Base on the program language you are using you may choose Informix database driver.
answered Nov 27 '18 at 6:06
SatyanSatyan
27114
You don’t need any additional software installation on the system where Informix server is running.I am not fully clear about the question or the type of operation you are plan to do. If you are planning to setup a real time replication type of scenario, then you may have to invoke CDC API. Then one-time setup of CDC API at server is needed, then this APIs can be invoked using any Informix database driver API. If you are plan to read existing data from table(s) and pump into Kafka topic, then no need of any additional setup at server side. You could connect to all 50 database server(s) from a single program (remotely) and then pump those records to the Kafka topic(s). Base on the program language you are using you may choose Informix database driver.
answered Nov 27 '18 at 6:06
SatyanSatyan
27114
answered Nov 27 '18 at 6:06
SatyanSatyan
27114
answered Nov 27 '18 at 6:06
SatyanSatyan
27114
answered Nov 27 '18 at 6:06
SatyanSatyan
27114
27114
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53418543%2fis-there-a-way-to-connect-to-multiple-databases-in-multiple-hosts-using-kafka-co%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 '18 at 21:09