Apache Beam RetryTransientErrors/neverRetry does not respect table not found error
up vote
0
down vote
favorite
Below is the code that I'm using to write data to BigQuery
WriteResult result = formattedData.get(successRows).setCoder(TableRowJsonCoder.of())
.apply("BQ SteamingInserts",BigQueryIO.writeTableRows()
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withFormatFunction(new TableRowFormatFn())
.to(new DestinationMapper())
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
.withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
.withoutValidation()
.withExtendedErrorInfo());
The code is handling all the schema related issues but when table does not exist in BigQuery, it keeps retrying the inserts indefinitely resulting in pipeline getting stalled.
Below is the error obtained in Dataflow
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"reason" : "notFound"
} ],
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"status" : "NOT_FOUND"
}
Can someone help?
google-cloud-dataflow apache-beam asked Nov 20 at 11:44
Tameem
1229
add a comment |
up vote
0
down vote
favorite
Below is the code that I'm using to write data to BigQuery
WriteResult result = formattedData.get(successRows).setCoder(TableRowJsonCoder.of())
.apply("BQ SteamingInserts",BigQueryIO.writeTableRows()
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withFormatFunction(new TableRowFormatFn())
.to(new DestinationMapper())
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
.withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
.withoutValidation()
.withExtendedErrorInfo());
The code is handling all the schema related issues but when table does not exist in BigQuery, it keeps retrying the inserts indefinitely resulting in pipeline getting stalled.
Below is the error obtained in Dataflow
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"reason" : "notFound"
} ],
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"status" : "NOT_FOUND"
}
Can someone help?
google-cloud-dataflow apache-beam asked Nov 20 at 11:44
Tameem
1229
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Below is the code that I'm using to write data to BigQuery
WriteResult result = formattedData.get(successRows).setCoder(TableRowJsonCoder.of())
.apply("BQ SteamingInserts",BigQueryIO.writeTableRows()
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withFormatFunction(new TableRowFormatFn())
.to(new DestinationMapper())
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
.withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
.withoutValidation()
.withExtendedErrorInfo());
The code is handling all the schema related issues but when table does not exist in BigQuery, it keeps retrying the inserts indefinitely resulting in pipeline getting stalled.
Below is the error obtained in Dataflow
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"reason" : "notFound"
} ],
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"status" : "NOT_FOUND"
}
Can someone help?
google-cloud-dataflow apache-beam asked Nov 20 at 11:44
Tameem
1229
Below is the code that I'm using to write data to BigQuery
WriteResult result = formattedData.get(successRows).setCoder(TableRowJsonCoder.of())
.apply("BQ SteamingInserts",BigQueryIO.writeTableRows()
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withFormatFunction(new TableRowFormatFn())
.to(new DestinationMapper())
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
.withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
.withoutValidation()
.withExtendedErrorInfo());
The code is handling all the schema related issues but when table does not exist in BigQuery, it keeps retrying the inserts indefinitely resulting in pipeline getting stalled.
Below is the error obtained in Dataflow
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"reason" : "notFound"
} ],
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"status" : "NOT_FOUND"
}
Can someone help?
google-cloud-dataflow apache-beam google-cloud-dataflow apache-beam asked Nov 20 at 11:44
Tameem
1229
asked Nov 20 at 11:44
Tameem
1229
edited Nov 20 at 16:21
asked Nov 20 at 11:44
Tameem
1229
asked Nov 20 at 11:44
Tameem
1229
asked Nov 20 at 11:44
Tameem
1229
1229
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
answered Nov 21 at 22:32
MonicaPC
926
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53392311%2fapache-beam-retrytransienterrors-neverretry-does-not-respect-table-not-found-err%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
answered Nov 21 at 22:32
MonicaPC
926
add a comment |
up vote
0
down vote
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
answered Nov 21 at 22:32
MonicaPC
926
add a comment |
up vote
0
down vote
up vote
0
down vote
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
answered Nov 21 at 22:32
MonicaPC
926
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
answered Nov 21 at 22:32
MonicaPC
926
answered Nov 21 at 22:32
MonicaPC
926
answered Nov 21 at 22:32
MonicaPC
926
answered Nov 21 at 22:32
MonicaPC
926
926
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53392311%2fapache-beam-retrytransienterrors-neverretry-does-not-respect-table-not-found-err%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
