Nifi 1.8.0 - Thread termination in nifi while using fetchHDFS processor
I'm trying to pull some data from HDFS. I'm running the listHDFS and fetchHDFS processor for this.
When I stopped the fetchHDFS processor, there were a number of active threads even after stopping the processor. To kill these threads, I used the "terminate" option.
Just wanted to know the working of the terminate option.
- Does it gracefully shut all the connections with the FS?
- Since all the threads are killed, do I lose out on the data that was consumed by these threads?
- Is it advised the terminate option only when the threads are stuck or the flow enters a frozen state?
hadoop apache-nifi hortonworks-data-platform
asked Nov 26 '18 at 7:21
devnongdevnong
295
add a comment |
I'm trying to pull some data from HDFS. I'm running the listHDFS and fetchHDFS processor for this.
When I stopped the fetchHDFS processor, there were a number of active threads even after stopping the processor. To kill these threads, I used the "terminate" option.
Just wanted to know the working of the terminate option.
- Does it gracefully shut all the connections with the FS?
- Since all the threads are killed, do I lose out on the data that was consumed by these threads?
- Is it advised the terminate option only when the threads are stuck or the flow enters a frozen state?
hadoop apache-nifi hortonworks-data-platform
asked Nov 26 '18 at 7:21
devnongdevnong
295
add a comment |
I'm trying to pull some data from HDFS. I'm running the listHDFS and fetchHDFS processor for this.
When I stopped the fetchHDFS processor, there were a number of active threads even after stopping the processor. To kill these threads, I used the "terminate" option.
Just wanted to know the working of the terminate option.
- Does it gracefully shut all the connections with the FS?
- Since all the threads are killed, do I lose out on the data that was consumed by these threads?
- Is it advised the terminate option only when the threads are stuck or the flow enters a frozen state?
hadoop apache-nifi hortonworks-data-platform
asked Nov 26 '18 at 7:21
devnongdevnong
295
I'm trying to pull some data from HDFS. I'm running the listHDFS and fetchHDFS processor for this.
When I stopped the fetchHDFS processor, there were a number of active threads even after stopping the processor. To kill these threads, I used the "terminate" option.
Just wanted to know the working of the terminate option.
- Does it gracefully shut all the connections with the FS?
- Since all the threads are killed, do I lose out on the data that was consumed by these threads?
- Is it advised the terminate option only when the threads are stuck or the flow enters a frozen state?
hadoop apache-nifi hortonworks-data-platform
hadoop apache-nifi hortonworks-data-platform
asked Nov 26 '18 at 7:21
devnongdevnong
295
asked Nov 26 '18 at 7:21
devnongdevnong
295
asked Nov 26 '18 at 7:21
devnongdevnong
295
asked Nov 26 '18 at 7:21
devnongdevnong
295
asked Nov 26 '18 at 7:21
devnongdevnong
295
295
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
When you stop a processor it tells the NiFi framework to no longer schedule/execute the processor, but there may already be threads executing which need to finish what they were doing. Usually these threads should complete and you will see the active threads go away, but sometimes a thread is blocked (typically when trying to make a network connection somewhere without having proper timeouts set) and this thread may never complete, and therefore needs to be terminated.
The terminate option will issue an interrupt to the thread and then quarantine it, which takes it out of the pool for further execution. The thread may then complete in the background, or if it did not respond to the interrupt and is blocked then it may stay stuck in the background until the next restart of NiFi.
In the FetchHDFS case, assuming it was successfully fetching data, it was most likely in the middle of reading a file from HDFS and just needs a few minutes to complete and shouldn't need to use terminate. If it was never fetching data and was stuck connecting to HDFS then you would use terminate.
answered Nov 27 '18 at 14:00
Bryan BendeBryan Bende
11.8k919
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53476351%2fnifi-1-8-0-thread-termination-in-nifi-while-using-fetchhdfs-processor%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
When you stop a processor it tells the NiFi framework to no longer schedule/execute the processor, but there may already be threads executing which need to finish what they were doing. Usually these threads should complete and you will see the active threads go away, but sometimes a thread is blocked (typically when trying to make a network connection somewhere without having proper timeouts set) and this thread may never complete, and therefore needs to be terminated.
The terminate option will issue an interrupt to the thread and then quarantine it, which takes it out of the pool for further execution. The thread may then complete in the background, or if it did not respond to the interrupt and is blocked then it may stay stuck in the background until the next restart of NiFi.
In the FetchHDFS case, assuming it was successfully fetching data, it was most likely in the middle of reading a file from HDFS and just needs a few minutes to complete and shouldn't need to use terminate. If it was never fetching data and was stuck connecting to HDFS then you would use terminate.
answered Nov 27 '18 at 14:00
Bryan BendeBryan Bende
11.8k919
add a comment |
When you stop a processor it tells the NiFi framework to no longer schedule/execute the processor, but there may already be threads executing which need to finish what they were doing. Usually these threads should complete and you will see the active threads go away, but sometimes a thread is blocked (typically when trying to make a network connection somewhere without having proper timeouts set) and this thread may never complete, and therefore needs to be terminated.
The terminate option will issue an interrupt to the thread and then quarantine it, which takes it out of the pool for further execution. The thread may then complete in the background, or if it did not respond to the interrupt and is blocked then it may stay stuck in the background until the next restart of NiFi.
In the FetchHDFS case, assuming it was successfully fetching data, it was most likely in the middle of reading a file from HDFS and just needs a few minutes to complete and shouldn't need to use terminate. If it was never fetching data and was stuck connecting to HDFS then you would use terminate.
answered Nov 27 '18 at 14:00
Bryan BendeBryan Bende
11.8k919
add a comment |
When you stop a processor it tells the NiFi framework to no longer schedule/execute the processor, but there may already be threads executing which need to finish what they were doing. Usually these threads should complete and you will see the active threads go away, but sometimes a thread is blocked (typically when trying to make a network connection somewhere without having proper timeouts set) and this thread may never complete, and therefore needs to be terminated.
The terminate option will issue an interrupt to the thread and then quarantine it, which takes it out of the pool for further execution. The thread may then complete in the background, or if it did not respond to the interrupt and is blocked then it may stay stuck in the background until the next restart of NiFi.
In the FetchHDFS case, assuming it was successfully fetching data, it was most likely in the middle of reading a file from HDFS and just needs a few minutes to complete and shouldn't need to use terminate. If it was never fetching data and was stuck connecting to HDFS then you would use terminate.
answered Nov 27 '18 at 14:00
Bryan BendeBryan Bende
11.8k919
When you stop a processor it tells the NiFi framework to no longer schedule/execute the processor, but there may already be threads executing which need to finish what they were doing. Usually these threads should complete and you will see the active threads go away, but sometimes a thread is blocked (typically when trying to make a network connection somewhere without having proper timeouts set) and this thread may never complete, and therefore needs to be terminated.
The terminate option will issue an interrupt to the thread and then quarantine it, which takes it out of the pool for further execution. The thread may then complete in the background, or if it did not respond to the interrupt and is blocked then it may stay stuck in the background until the next restart of NiFi.
In the FetchHDFS case, assuming it was successfully fetching data, it was most likely in the middle of reading a file from HDFS and just needs a few minutes to complete and shouldn't need to use terminate. If it was never fetching data and was stuck connecting to HDFS then you would use terminate.
answered Nov 27 '18 at 14:00
Bryan BendeBryan Bende
11.8k919
answered Nov 27 '18 at 14:00
Bryan BendeBryan Bende
11.8k919
answered Nov 27 '18 at 14:00
Bryan BendeBryan Bende
11.8k919
answered Nov 27 '18 at 14:00
Bryan BendeBryan Bende
11.8k919
11.8k919
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53476351%2fnifi-1-8-0-thread-termination-in-nifi-while-using-fetchhdfs-processor%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
