Beam Dataflow only writes to temp in GCS











up vote
0
down vote

favorite












I have a very basic Python Dataflow job that reads some data from Pub/Sub, applies a FixedWindow and writes to Google Cloud Storage.



transformed = ...

transformed | beam.io.WriteToText(known_args.output)


The output is written to the location specific in --output, but only the temporary stage, i.e.



gs://MY_BUCKET/MY_DIR/beam-temp-2a5c0e1eec1c11e8b98342010a800004/...some_UUID...


The file never gets placed into the correctly named location with the sharding template.



Tested on local and DataFlow runner.




When testing further, I have noticed that the streaming_wordcount example has the same issues, however the standard wordcount example writes fine. Perhaps the issues is to with windowing, or reading from pubsub?




It appears WriteToText is not compatible with the streaming source of PubSub. There are likely workarounds, or the Java version may be compatible, but I have opted to use a different solution altogether.






google-cloud-storage google-cloud-dataflow apache-beam google-cloud-pubsub







share|improve this question
























  • can you please post the code?
    – Tanveer Uddin
    Nov 19 at 21:41















up vote
0
down vote

favorite












I have a very basic Python Dataflow job that reads some data from Pub/Sub, applies a FixedWindow and writes to Google Cloud Storage.



transformed = ...

transformed | beam.io.WriteToText(known_args.output)


The output is written to the location specific in --output, but only the temporary stage, i.e.



gs://MY_BUCKET/MY_DIR/beam-temp-2a5c0e1eec1c11e8b98342010a800004/...some_UUID...


The file never gets placed into the correctly named location with the sharding template.



Tested on local and DataFlow runner.




When testing further, I have noticed that the streaming_wordcount example has the same issues, however the standard wordcount example writes fine. Perhaps the issues is to with windowing, or reading from pubsub?




It appears WriteToText is not compatible with the streaming source of PubSub. There are likely workarounds, or the Java version may be compatible, but I have opted to use a different solution altogether.






google-cloud-storage google-cloud-dataflow apache-beam google-cloud-pubsub







share|improve this question
























  • can you please post the code?
    – Tanveer Uddin
    Nov 19 at 21:41













up vote
0
down vote

favorite









up vote
0
down vote

favorite











I have a very basic Python Dataflow job that reads some data from Pub/Sub, applies a FixedWindow and writes to Google Cloud Storage.



transformed = ...

transformed | beam.io.WriteToText(known_args.output)


The output is written to the location specific in --output, but only the temporary stage, i.e.



gs://MY_BUCKET/MY_DIR/beam-temp-2a5c0e1eec1c11e8b98342010a800004/...some_UUID...


The file never gets placed into the correctly named location with the sharding template.



Tested on local and DataFlow runner.




When testing further, I have noticed that the streaming_wordcount example has the same issues, however the standard wordcount example writes fine. Perhaps the issues is to with windowing, or reading from pubsub?




It appears WriteToText is not compatible with the streaming source of PubSub. There are likely workarounds, or the Java version may be compatible, but I have opted to use a different solution altogether.






google-cloud-storage google-cloud-dataflow apache-beam google-cloud-pubsub







share|improve this question















I have a very basic Python Dataflow job that reads some data from Pub/Sub, applies a FixedWindow and writes to Google Cloud Storage.



transformed = ...

transformed | beam.io.WriteToText(known_args.output)


The output is written to the location specific in --output, but only the temporary stage, i.e.



gs://MY_BUCKET/MY_DIR/beam-temp-2a5c0e1eec1c11e8b98342010a800004/...some_UUID...


The file never gets placed into the correctly named location with the sharding template.



Tested on local and DataFlow runner.




When testing further, I have noticed that the streaming_wordcount example has the same issues, however the standard wordcount example writes fine. Perhaps the issues is to with windowing, or reading from pubsub?




It appears WriteToText is not compatible with the streaming source of PubSub. There are likely workarounds, or the Java version may be compatible, but I have opted to use a different solution altogether.






google-cloud-storage google-cloud-dataflow apache-beam google-cloud-pubsub




google-cloud-storage google-cloud-dataflow apache-beam google-cloud-pubsub






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 20 at 10:41

























asked Nov 19 at 17:10









Daniel Messias

6151818




6151818












  • can you please post the code?
    – Tanveer Uddin
    Nov 19 at 21:41


















  • can you please post the code?
    – Tanveer Uddin
    Nov 19 at 21:41
















can you please post the code?
– Tanveer Uddin
Nov 19 at 21:41




can you please post the code?
– Tanveer Uddin
Nov 19 at 21:41

















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














 

draft saved


draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53379585%2fbeam-dataflow-only-writes-to-temp-in-gcs%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown






























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes
















 

draft saved


draft discarded



















































 


draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53379585%2fbeam-dataflow-only-writes-to-temp-in-gcs%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

To store a contact into the json file from server.js file using a class in NodeJS

Image
0 I have created three files -contactPresentation.js, contactService.js, contacts.json. I will run the contactService.js file. And it will navigate to contactPresentation.js file that will only choose an option to either add or display a file. If add(), then add the data into add of service file.And the data will be added to the contacts.json file . And if display(), then display the display() of service file. And the display will take the data from contacts.json file. However, i know my code is flawed. Please correct it . I am also getting an error - this.contacts.push(new ContactService(userName, contactNumber, emailId)); ^ TypeError: Cannot read property 'push' of undefined at ContactService.add (e:NodeJSProjectcontactManager2contactService.js:57:23) at new C
Read more

Redirect URL with Chrome Remote Debugging Android Devices

Image
1 How can i redirect an URL with Chrome Remote Debugging Android Devices? like Requestly Chrome extension on desktop. google-chrome debugging url redirect testing share | improve this question edited Nov 24 '18 at 8:16 sachinjain024 14.7k 23 73 129 asked Nov 23 '18 at 16:09 Momen Momen 91 10
Read more

Dieringhausen

Image
Dieringhausen Stadt Gummersbach 50.982777777778 7.5316666666667 165 Koordinaten: 50° 58′ 58″  N , 7° 31′ 54″  O Höhe: 165  (160–245)  m Einwohner: 5055  (30. Jun. 2016) Postleitzahl: 51645 Vorwahl: 02261 Lage von Dieringhausen in Gummersbach Im Hohl ca. 1927–1930 Ansicht von Südwesten bei Bünghausen Dieringhausen (im örtlichen Dialekt Dierkesen ) ist ein Ortsteil von Gummersbach im Oberbergischen Kreis, Regierungsbezirk Köln, Nordrhein-Westfalen (Deutschland). Inhaltsverzeichnis 1 Geographie 2 Geschichte 3 Kultur 3.1 Sehenswürdigkeiten 3.2 Regelmäßige Veranstaltungen 4 Infrastruktur und Wirtschaft 4.1 Verkehr 4.1.1 Schienen- und Busverkehr 4.1.2 Straßen 4.2 Öffentliche Einrichtungen 4.2.1 Schulen und Bildungseinrichtungen 4.2.2 Kirchliche Einrichtungen 5 Persönlichkeiten 5.1 In Dieringhausen geboren 5.2 In Dieringhausen gelebt 6 Einzelnachweise 7
Read more