replace files with move when using pig on HDFS
I have a process which moves files to a processing folder with a pig script as MapReduce2 task in part of a hadoop workflow.
I have seen copies fail recently resulting in a partial move of the files.
When the job then next re-runs there is an error as the pig script will try to move the file file again but as there is a partial of the file at the target location it will fail. There is no option in pig to move with replace.
I could do a copy and delete but the risk with this is that while the copy is in progress another file could be uploaded to HDFS what wasn't included in the origional copy operation and then when I run the delete all I also delete a file that has yet to be moved to the processing directory.
I know there is no force replace on move but is there a way to create a list of all the files that I'm about to move from the source location and then check for them at the target location, and then if they exist in the target delete them before moving?
Many thanks for your help in advance.
hdfs apache-pig
asked Nov 20 at 17:07
Hubington
414
add a comment |
I have a process which moves files to a processing folder with a pig script as MapReduce2 task in part of a hadoop workflow.
I have seen copies fail recently resulting in a partial move of the files.
When the job then next re-runs there is an error as the pig script will try to move the file file again but as there is a partial of the file at the target location it will fail. There is no option in pig to move with replace.
I could do a copy and delete but the risk with this is that while the copy is in progress another file could be uploaded to HDFS what wasn't included in the origional copy operation and then when I run the delete all I also delete a file that has yet to be moved to the processing directory.
I know there is no force replace on move but is there a way to create a list of all the files that I'm about to move from the source location and then check for them at the target location, and then if they exist in the target delete them before moving?
Many thanks for your help in advance.
hdfs apache-pig
asked Nov 20 at 17:07
Hubington
414
add a comment |
I have a process which moves files to a processing folder with a pig script as MapReduce2 task in part of a hadoop workflow.
I have seen copies fail recently resulting in a partial move of the files.
When the job then next re-runs there is an error as the pig script will try to move the file file again but as there is a partial of the file at the target location it will fail. There is no option in pig to move with replace.
I could do a copy and delete but the risk with this is that while the copy is in progress another file could be uploaded to HDFS what wasn't included in the origional copy operation and then when I run the delete all I also delete a file that has yet to be moved to the processing directory.
I know there is no force replace on move but is there a way to create a list of all the files that I'm about to move from the source location and then check for them at the target location, and then if they exist in the target delete them before moving?
Many thanks for your help in advance.
hdfs apache-pig
asked Nov 20 at 17:07
Hubington
414
I have a process which moves files to a processing folder with a pig script as MapReduce2 task in part of a hadoop workflow.
I have seen copies fail recently resulting in a partial move of the files.
When the job then next re-runs there is an error as the pig script will try to move the file file again but as there is a partial of the file at the target location it will fail. There is no option in pig to move with replace.
I could do a copy and delete but the risk with this is that while the copy is in progress another file could be uploaded to HDFS what wasn't included in the origional copy operation and then when I run the delete all I also delete a file that has yet to be moved to the processing directory.
I know there is no force replace on move but is there a way to create a list of all the files that I'm about to move from the source location and then check for them at the target location, and then if they exist in the target delete them before moving?
Many thanks for your help in advance.
hdfs apache-pig
hdfs apache-pig
asked Nov 20 at 17:07
Hubington
414
asked Nov 20 at 17:07
Hubington
414
asked Nov 20 at 17:07
Hubington
414
asked Nov 20 at 17:07
Hubington
414
asked Nov 20 at 17:07
Hubington
414
414
add a comment |
add a comment |
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398051%2freplace-files-with-move-when-using-pig-on-hdfs%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398051%2freplace-files-with-move-when-using-pig-on-hdfs%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
