how to debug evicted pod on k8s
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
Team, I searched some but could not get much as am new to k8s. my pods are evicted and I get message as
"Status: Failed
Reason: Evicted
Message: The node was low on resource: nodefs."
any helm how can I figure out what is going on?
kubernetes kubectl kubelet asked Nov 26 '18 at 19:35
AhmFMAhmFM
798
add a comment |
Team, I searched some but could not get much as am new to k8s. my pods are evicted and I get message as
"Status: Failed
Reason: Evicted
Message: The node was low on resource: nodefs."
any helm how can I figure out what is going on?
kubernetes kubectl kubelet asked Nov 26 '18 at 19:35
AhmFMAhmFM
798
add a comment |
Team, I searched some but could not get much as am new to k8s. my pods are evicted and I get message as
"Status: Failed
Reason: Evicted
Message: The node was low on resource: nodefs."
any helm how can I figure out what is going on?
kubernetes kubectl kubelet asked Nov 26 '18 at 19:35
AhmFMAhmFM
798
Team, I searched some but could not get much as am new to k8s. my pods are evicted and I get message as
"Status: Failed
Reason: Evicted
Message: The node was low on resource: nodefs."
any helm how can I figure out what is going on?
kubernetes kubectl kubelet kubernetes kubectl kubelet asked Nov 26 '18 at 19:35
AhmFMAhmFM
798
asked Nov 26 '18 at 19:35
AhmFMAhmFM
798
asked Nov 26 '18 at 19:35
AhmFMAhmFM
798
asked Nov 26 '18 at 19:35
AhmFMAhmFM
798
asked Nov 26 '18 at 19:35
AhmFMAhmFM
798
798
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Run kubectl describe pod <pod name> and look for the node name of this pod. Followed by kubectl describe node <node-name> that will show what type of resource cap the node is hitting under Conditions: section.
From my experience this happens when the host node runs out of disk space.
answered Nov 26 '18 at 19:44
ZPrimeZPrime
464
They look ok. Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- OutOfDisk False Mon, 26 Nov 2018 11:49:32 -0800 Mon, 05 Nov 2018 16:09:45 -0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
– AhmFM
Nov 26 '18 at 19:57
but you suggestion did help me get to cause. Thanks!. however not sure what else could I do to find the reason? will it be just to delete pods now as last resort?
– AhmFM
Nov 26 '18 at 21:12
It's hard to say then. You can try starting the pod again and monitor the describe commandwatch kubectl describe node <>and see if the condition values change. Also after starting the pod you should runkubectl logs <pod-name> -fand see if there are anything helpful on the log output.
– ZPrime
Nov 26 '18 at 21:18
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53487855%2fhow-to-debug-evicted-pod-on-k8s%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Run kubectl describe pod <pod name> and look for the node name of this pod. Followed by kubectl describe node <node-name> that will show what type of resource cap the node is hitting under Conditions: section.
From my experience this happens when the host node runs out of disk space.
answered Nov 26 '18 at 19:44
ZPrimeZPrime
464
They look ok. Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- OutOfDisk False Mon, 26 Nov 2018 11:49:32 -0800 Mon, 05 Nov 2018 16:09:45 -0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
– AhmFM
Nov 26 '18 at 19:57
but you suggestion did help me get to cause. Thanks!. however not sure what else could I do to find the reason? will it be just to delete pods now as last resort?
– AhmFM
Nov 26 '18 at 21:12
It's hard to say then. You can try starting the pod again and monitor the describe commandwatch kubectl describe node <>and see if the condition values change. Also after starting the pod you should runkubectl logs <pod-name> -fand see if there are anything helpful on the log output.
– ZPrime
Nov 26 '18 at 21:18
add a comment |
Run kubectl describe pod <pod name> and look for the node name of this pod. Followed by kubectl describe node <node-name> that will show what type of resource cap the node is hitting under Conditions: section.
From my experience this happens when the host node runs out of disk space.
answered Nov 26 '18 at 19:44
ZPrimeZPrime
464
They look ok. Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- OutOfDisk False Mon, 26 Nov 2018 11:49:32 -0800 Mon, 05 Nov 2018 16:09:45 -0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
– AhmFM
Nov 26 '18 at 19:57
but you suggestion did help me get to cause. Thanks!. however not sure what else could I do to find the reason? will it be just to delete pods now as last resort?
– AhmFM
Nov 26 '18 at 21:12
It's hard to say then. You can try starting the pod again and monitor the describe commandwatch kubectl describe node <>and see if the condition values change. Also after starting the pod you should runkubectl logs <pod-name> -fand see if there are anything helpful on the log output.
– ZPrime
Nov 26 '18 at 21:18
add a comment |
Run kubectl describe pod <pod name> and look for the node name of this pod. Followed by kubectl describe node <node-name> that will show what type of resource cap the node is hitting under Conditions: section.
From my experience this happens when the host node runs out of disk space.
answered Nov 26 '18 at 19:44
ZPrimeZPrime
464
Run kubectl describe pod <pod name> and look for the node name of this pod. Followed by kubectl describe node <node-name> that will show what type of resource cap the node is hitting under Conditions: section.
From my experience this happens when the host node runs out of disk space.
answered Nov 26 '18 at 19:44
ZPrimeZPrime
464
answered Nov 26 '18 at 19:44
ZPrimeZPrime
464
answered Nov 26 '18 at 19:44
ZPrimeZPrime
464
answered Nov 26 '18 at 19:44
ZPrimeZPrime
464
464
They look ok. Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- OutOfDisk False Mon, 26 Nov 2018 11:49:32 -0800 Mon, 05 Nov 2018 16:09:45 -0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
– AhmFM
Nov 26 '18 at 19:57
but you suggestion did help me get to cause. Thanks!. however not sure what else could I do to find the reason? will it be just to delete pods now as last resort?
– AhmFM
Nov 26 '18 at 21:12
It's hard to say then. You can try starting the pod again and monitor the describe commandwatch kubectl describe node <>and see if the condition values change. Also after starting the pod you should runkubectl logs <pod-name> -fand see if there are anything helpful on the log output.
– ZPrime
Nov 26 '18 at 21:18
add a comment |
They look ok. Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- OutOfDisk False Mon, 26 Nov 2018 11:49:32 -0800 Mon, 05 Nov 2018 16:09:45 -0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
– AhmFM
Nov 26 '18 at 19:57
but you suggestion did help me get to cause. Thanks!. however not sure what else could I do to find the reason? will it be just to delete pods now as last resort?
– AhmFM
Nov 26 '18 at 21:12
It's hard to say then. You can try starting the pod again and monitor the describe commandwatch kubectl describe node <>and see if the condition values change. Also after starting the pod you should runkubectl logs <pod-name> -fand see if there are anything helpful on the log output.
– ZPrime
Nov 26 '18 at 21:18
They look ok. Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- OutOfDisk False Mon, 26 Nov 2018 11:49:32 -0800 Mon, 05 Nov 2018 16:09:45 -0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
– AhmFM
Nov 26 '18 at 19:57
They look ok. Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- OutOfDisk False Mon, 26 Nov 2018 11:49:32 -0800 Mon, 05 Nov 2018 16:09:45 -0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
– AhmFM
Nov 26 '18 at 19:57
but you suggestion did help me get to cause. Thanks!. however not sure what else could I do to find the reason? will it be just to delete pods now as last resort?
– AhmFM
Nov 26 '18 at 21:12
but you suggestion did help me get to cause. Thanks!. however not sure what else could I do to find the reason? will it be just to delete pods now as last resort?
– AhmFM
Nov 26 '18 at 21:12
It's hard to say then. You can try starting the pod again and monitor the describe command
watch kubectl describe node <> and see if the condition values change. Also after starting the pod you should run kubectl logs <pod-name> -f and see if there are anything helpful on the log output.– ZPrime
Nov 26 '18 at 21:18
It's hard to say then. You can try starting the pod again and monitor the describe command
watch kubectl describe node <> and see if the condition values change. Also after starting the pod you should run kubectl logs <pod-name> -f and see if there are anything helpful on the log output.– ZPrime
Nov 26 '18 at 21:18
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53487855%2fhow-to-debug-evicted-pod-on-k8s%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
