In Kafka Connector, how do I get the bootstrap-server address My Kafka Connect is currently using?
up vote
1
down vote
favorite
I'm developing a Kafka Sink connector on my own. My deserializer is JSONConverter. However, when someone send a wrong JSON data into my connector's topic, I want to omit this record and send this record to a specific topic of my company.
My confuse is: I can't find any API for me to get my Connect's bootstrap.servers.(I know it's in the confluent's etc directory but it's not a good idea to write hard code of the directory of "connect-distributed.properties" to get the bootstrap.servers)
So question, is there another way for me to get the value of bootstrap.servers conveniently in my connector program?
apache-kafka apache-kafka-connect
edited Nov 20 at 15:31
cricket_007
78.4k1142109
asked Nov 20 at 11:44
SkyOne
85
|
show 1 more comment
up vote
1
down vote
favorite
I'm developing a Kafka Sink connector on my own. My deserializer is JSONConverter. However, when someone send a wrong JSON data into my connector's topic, I want to omit this record and send this record to a specific topic of my company.
My confuse is: I can't find any API for me to get my Connect's bootstrap.servers.(I know it's in the confluent's etc directory but it's not a good idea to write hard code of the directory of "connect-distributed.properties" to get the bootstrap.servers)
So question, is there another way for me to get the value of bootstrap.servers conveniently in my connector program?
apache-kafka apache-kafka-connect
edited Nov 20 at 15:31
cricket_007
78.4k1142109
asked Nov 20 at 11:44
SkyOne
85
What do you mean by Connect'sbootstrap.servers? Do you want the host of Kafka Connect or Kafka brokers?
– Giorgos Myrianthous
Nov 20 at 12:57
@GiorgosMyrianthous the bootstrap.servers for me to build a Kafka producer... I think that should be the Kafka brokers
– SkyOne
Nov 20 at 13:08
I don't think you can make a call to REST Proxy API in order to getbootstrap.servers. You can only get Brokers' IDs. I would suggest to create a.propertiesfile that contains yourbootstrap.serversand load that file on producer's startup.
– Giorgos Myrianthous
Nov 20 at 13:10
I don't see anything wrong with reading the file from disk, assuming all Connect workers are consistently installed... In fact, that's how the new password hiding feature works
– cricket_007
Nov 20 at 15:34
@cricket_007 What if the file name is various against different environments? i.e. etc/dev/dev-connect-distributed.properties and etc/prd/prd-connect-distributed.properties
– SkyOne
Nov 20 at 22:50
|
show 1 more comment
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I'm developing a Kafka Sink connector on my own. My deserializer is JSONConverter. However, when someone send a wrong JSON data into my connector's topic, I want to omit this record and send this record to a specific topic of my company.
My confuse is: I can't find any API for me to get my Connect's bootstrap.servers.(I know it's in the confluent's etc directory but it's not a good idea to write hard code of the directory of "connect-distributed.properties" to get the bootstrap.servers)
So question, is there another way for me to get the value of bootstrap.servers conveniently in my connector program?
apache-kafka apache-kafka-connect
edited Nov 20 at 15:31
cricket_007
78.4k1142109
asked Nov 20 at 11:44
SkyOne
85
I'm developing a Kafka Sink connector on my own. My deserializer is JSONConverter. However, when someone send a wrong JSON data into my connector's topic, I want to omit this record and send this record to a specific topic of my company.
My confuse is: I can't find any API for me to get my Connect's bootstrap.servers.(I know it's in the confluent's etc directory but it's not a good idea to write hard code of the directory of "connect-distributed.properties" to get the bootstrap.servers)
So question, is there another way for me to get the value of bootstrap.servers conveniently in my connector program?
apache-kafka apache-kafka-connect
apache-kafka apache-kafka-connect
edited Nov 20 at 15:31
cricket_007
78.4k1142109
asked Nov 20 at 11:44
SkyOne
85
edited Nov 20 at 15:31
cricket_007
78.4k1142109
asked Nov 20 at 11:44
SkyOne
85
edited Nov 20 at 15:31
cricket_007
78.4k1142109
edited Nov 20 at 15:31
cricket_007
78.4k1142109
edited Nov 20 at 15:31
cricket_007
78.4k1142109
78.4k1142109
asked Nov 20 at 11:44
SkyOne
85
asked Nov 20 at 11:44
SkyOne
85
asked Nov 20 at 11:44
SkyOne
85
85
What do you mean by Connect'sbootstrap.servers? Do you want the host of Kafka Connect or Kafka brokers?
– Giorgos Myrianthous
Nov 20 at 12:57
@GiorgosMyrianthous the bootstrap.servers for me to build a Kafka producer... I think that should be the Kafka brokers
– SkyOne
Nov 20 at 13:08
I don't think you can make a call to REST Proxy API in order to getbootstrap.servers. You can only get Brokers' IDs. I would suggest to create a.propertiesfile that contains yourbootstrap.serversand load that file on producer's startup.
– Giorgos Myrianthous
Nov 20 at 13:10
I don't see anything wrong with reading the file from disk, assuming all Connect workers are consistently installed... In fact, that's how the new password hiding feature works
– cricket_007
Nov 20 at 15:34
@cricket_007 What if the file name is various against different environments? i.e. etc/dev/dev-connect-distributed.properties and etc/prd/prd-connect-distributed.properties
– SkyOne
Nov 20 at 22:50
|
show 1 more comment
What do you mean by Connect'sbootstrap.servers? Do you want the host of Kafka Connect or Kafka brokers?
– Giorgos Myrianthous
Nov 20 at 12:57
@GiorgosMyrianthous the bootstrap.servers for me to build a Kafka producer... I think that should be the Kafka brokers
– SkyOne
Nov 20 at 13:08
I don't think you can make a call to REST Proxy API in order to getbootstrap.servers. You can only get Brokers' IDs. I would suggest to create a.propertiesfile that contains yourbootstrap.serversand load that file on producer's startup.
– Giorgos Myrianthous
Nov 20 at 13:10
I don't see anything wrong with reading the file from disk, assuming all Connect workers are consistently installed... In fact, that's how the new password hiding feature works
– cricket_007
Nov 20 at 15:34
@cricket_007 What if the file name is various against different environments? i.e. etc/dev/dev-connect-distributed.properties and etc/prd/prd-connect-distributed.properties
– SkyOne
Nov 20 at 22:50
What do you mean by Connect's
bootstrap.servers ? Do you want the host of Kafka Connect or Kafka brokers?– Giorgos Myrianthous
Nov 20 at 12:57
What do you mean by Connect's
bootstrap.servers ? Do you want the host of Kafka Connect or Kafka brokers?– Giorgos Myrianthous
Nov 20 at 12:57
@GiorgosMyrianthous the bootstrap.servers for me to build a Kafka producer... I think that should be the Kafka brokers
– SkyOne
Nov 20 at 13:08
@GiorgosMyrianthous the bootstrap.servers for me to build a Kafka producer... I think that should be the Kafka brokers
– SkyOne
Nov 20 at 13:08
I don't think you can make a call to REST Proxy API in order to get
bootstrap.servers. You can only get Brokers' IDs. I would suggest to create a .properties file that contains your bootstrap.servers and load that file on producer's startup.– Giorgos Myrianthous
Nov 20 at 13:10
I don't think you can make a call to REST Proxy API in order to get
bootstrap.servers. You can only get Brokers' IDs. I would suggest to create a .properties file that contains your bootstrap.servers and load that file on producer's startup.– Giorgos Myrianthous
Nov 20 at 13:10
I don't see anything wrong with reading the file from disk, assuming all Connect workers are consistently installed... In fact, that's how the new password hiding feature works
– cricket_007
Nov 20 at 15:34
I don't see anything wrong with reading the file from disk, assuming all Connect workers are consistently installed... In fact, that's how the new password hiding feature works
– cricket_007
Nov 20 at 15:34
@cricket_007 What if the file name is various against different environments? i.e. etc/dev/dev-connect-distributed.properties and etc/prd/prd-connect-distributed.properties
– SkyOne
Nov 20 at 22:50
@cricket_007 What if the file name is various against different environments? i.e. etc/dev/dev-connect-distributed.properties and etc/prd/prd-connect-distributed.properties
– SkyOne
Nov 20 at 22:50
|
show 1 more comment
1 Answer
1
active
oldest
votes
up vote
2
down vote
accepted
Instead of trying to send the "bad" records from a SinkTask to Kafka, you should instead try to use the dead letter queue feature that was added in Kafka Connect 2.0.
You can configure the Connect runtime to automatically dump records that failed to be processed to a configured topic acting as a DLQ.
For more details, see the KIP that added this feature.
answered Nov 20 at 13:31
Mickael Maison
6,84532529
Worth pointing out - Connect can be upgraded independently of the brokers
– cricket_007
Nov 20 at 15:32
@cricket_007 Our company now use Confluent4.0.1 with Kafka 1.0.1, can we just upgrade Confluent4 to Confluent5 while still using the Kafka 1.0.1 cluster?
– SkyOne
Nov 21 at 1:58
@Sky You should be able to yes. Brokers since Confluent 3.0 support newer Confluent versions for Connect workers
– cricket_007
Nov 21 at 4:38
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53392298%2fin-kafka-connector-how-do-i-get-the-bootstrap-server-address-my-kafka-connect-i%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
accepted
Instead of trying to send the "bad" records from a SinkTask to Kafka, you should instead try to use the dead letter queue feature that was added in Kafka Connect 2.0.
You can configure the Connect runtime to automatically dump records that failed to be processed to a configured topic acting as a DLQ.
For more details, see the KIP that added this feature.
answered Nov 20 at 13:31
Mickael Maison
6,84532529
Worth pointing out - Connect can be upgraded independently of the brokers
– cricket_007
Nov 20 at 15:32
@cricket_007 Our company now use Confluent4.0.1 with Kafka 1.0.1, can we just upgrade Confluent4 to Confluent5 while still using the Kafka 1.0.1 cluster?
– SkyOne
Nov 21 at 1:58
@Sky You should be able to yes. Brokers since Confluent 3.0 support newer Confluent versions for Connect workers
– cricket_007
Nov 21 at 4:38
add a comment |
up vote
2
down vote
accepted
Instead of trying to send the "bad" records from a SinkTask to Kafka, you should instead try to use the dead letter queue feature that was added in Kafka Connect 2.0.
You can configure the Connect runtime to automatically dump records that failed to be processed to a configured topic acting as a DLQ.
For more details, see the KIP that added this feature.
answered Nov 20 at 13:31
Mickael Maison
6,84532529
Worth pointing out - Connect can be upgraded independently of the brokers
– cricket_007
Nov 20 at 15:32
@cricket_007 Our company now use Confluent4.0.1 with Kafka 1.0.1, can we just upgrade Confluent4 to Confluent5 while still using the Kafka 1.0.1 cluster?
– SkyOne
Nov 21 at 1:58
@Sky You should be able to yes. Brokers since Confluent 3.0 support newer Confluent versions for Connect workers
– cricket_007
Nov 21 at 4:38
add a comment |
up vote
2
down vote
accepted
up vote
2
down vote
accepted
Instead of trying to send the "bad" records from a SinkTask to Kafka, you should instead try to use the dead letter queue feature that was added in Kafka Connect 2.0.
You can configure the Connect runtime to automatically dump records that failed to be processed to a configured topic acting as a DLQ.
For more details, see the KIP that added this feature.
answered Nov 20 at 13:31
Mickael Maison
6,84532529
Instead of trying to send the "bad" records from a SinkTask to Kafka, you should instead try to use the dead letter queue feature that was added in Kafka Connect 2.0.
You can configure the Connect runtime to automatically dump records that failed to be processed to a configured topic acting as a DLQ.
For more details, see the KIP that added this feature.
answered Nov 20 at 13:31
Mickael Maison
6,84532529
answered Nov 20 at 13:31
Mickael Maison
6,84532529
answered Nov 20 at 13:31
Mickael Maison
6,84532529
answered Nov 20 at 13:31
Mickael Maison
6,84532529
6,84532529
Worth pointing out - Connect can be upgraded independently of the brokers
– cricket_007
Nov 20 at 15:32
@cricket_007 Our company now use Confluent4.0.1 with Kafka 1.0.1, can we just upgrade Confluent4 to Confluent5 while still using the Kafka 1.0.1 cluster?
– SkyOne
Nov 21 at 1:58
@Sky You should be able to yes. Brokers since Confluent 3.0 support newer Confluent versions for Connect workers
– cricket_007
Nov 21 at 4:38
add a comment |
Worth pointing out - Connect can be upgraded independently of the brokers
– cricket_007
Nov 20 at 15:32
@cricket_007 Our company now use Confluent4.0.1 with Kafka 1.0.1, can we just upgrade Confluent4 to Confluent5 while still using the Kafka 1.0.1 cluster?
– SkyOne
Nov 21 at 1:58
@Sky You should be able to yes. Brokers since Confluent 3.0 support newer Confluent versions for Connect workers
– cricket_007
Nov 21 at 4:38
Worth pointing out - Connect can be upgraded independently of the brokers
– cricket_007
Nov 20 at 15:32
Worth pointing out - Connect can be upgraded independently of the brokers
– cricket_007
Nov 20 at 15:32
@cricket_007 Our company now use Confluent4.0.1 with Kafka 1.0.1, can we just upgrade Confluent4 to Confluent5 while still using the Kafka 1.0.1 cluster?
– SkyOne
Nov 21 at 1:58
@cricket_007 Our company now use Confluent4.0.1 with Kafka 1.0.1, can we just upgrade Confluent4 to Confluent5 while still using the Kafka 1.0.1 cluster?
– SkyOne
Nov 21 at 1:58
@Sky You should be able to yes. Brokers since Confluent 3.0 support newer Confluent versions for Connect workers
– cricket_007
Nov 21 at 4:38
@Sky You should be able to yes. Brokers since Confluent 3.0 support newer Confluent versions for Connect workers
– cricket_007
Nov 21 at 4:38
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53392298%2fin-kafka-connector-how-do-i-get-the-bootstrap-server-address-my-kafka-connect-i%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What do you mean by Connect's
bootstrap.servers? Do you want the host of Kafka Connect or Kafka brokers?– Giorgos Myrianthous
Nov 20 at 12:57
@GiorgosMyrianthous the bootstrap.servers for me to build a Kafka producer... I think that should be the Kafka brokers
– SkyOne
Nov 20 at 13:08
I don't think you can make a call to REST Proxy API in order to get
bootstrap.servers. You can only get Brokers' IDs. I would suggest to create a.propertiesfile that contains yourbootstrap.serversand load that file on producer's startup.– Giorgos Myrianthous
Nov 20 at 13:10
I don't see anything wrong with reading the file from disk, assuming all Connect workers are consistently installed... In fact, that's how the new password hiding feature works
– cricket_007
Nov 20 at 15:34
@cricket_007 What if the file name is various against different environments? i.e. etc/dev/dev-connect-distributed.properties and etc/prd/prd-connect-distributed.properties
– SkyOne
Nov 20 at 22:50