Linux · critical · Containers & Kubernetes

A Kubernetes node goes NotReady

The kubelet has stopped reporting, or is reporting a condition that makes the node unschedulable.

What you see

kubectl get nodes shows NotReady. Pods are evicted and rescheduled elsewhere, or stay Terminating indefinitely.

What is actually wrong

The kubelet has stopped or cannot reach the API server, the container runtime is unresponsive, or the node is under disk or memory pressure.

Codes and articles

NotReadykubelet stopped posting node statusDiskPressureMemoryPressurePLEG is not healthycontext deadline exceeded

Fixes (3)

Relieve the pressure the node is reporting

Root shell on the node40 minutesmedium riskreversible

DiskPressure or MemoryPressure.

Read the exact condition and its message.

Shell

kubectl describe node node01 | sed -n '/Conditions:/,/Addresses:/p'

For DiskPressure, find what filled up — it is usually images or logs.

Shell

df -h /var/lib/containerd /var/log /sudo du -sh /var/log/pods /var/lib/containerd 2>/dev/null

Let the kubelet's own garbage collection do the work, then prune what remains.
Shell
```
sudo crictl imagessudo crictl rmi --prune
```
crictl rmi --prune removes images not used by any container, which is what the kubelet would eventually do itself. Deleting files under /var/lib/containerd by hand corrupts the runtime's metadata.
Check the eviction thresholds are sensible for the disk size — the defaults are percentage-based and can be far too aggressive on a large disk.
Shell
```
sudo grep -E 'evictionHard|imageGC' /var/lib/kubelet/config.yaml
```

Cap the log sizes so this does not recur.

Shell

sudo grep -E 'containerLogMaxSize|containerLogMaxFiles' /var/lib/kubelet/config.yaml

Confirm it workedThe node returns to Ready and the condition clears.

Shell

kubectl get nodes; kubectl describe node node01 | grep -A6 Conditions

If you need to undo itConfiguration changes to the kubelet can be reverted and the service restarted.

Get the kubelet talking to the API server again

Root shell on the node40 minutesmedium riskreversible

The kubelet has stopped posting status.

Check the service and read its log.

Shell

systemctl status kubelet --no-pager -lsudo journalctl -u kubelet -n 80 --no-pager

Check the client certificate has not expired — this is the most common cause on a cluster that has been running for about a year.
Shell
```
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates -subject
```
Kubelet certificates are typically valid for a year and rotate automatically only if rotation is enabled. A cluster built without it goes NotReady, one node at a time, exactly twelve months later.

Check connectivity to the API server.

Shell

sudo grep server /etc/kubernetes/kubelet.confcurl -sk https://10.0.0.1:6443/healthz

Check the clock, which breaks certificate validation in both directions.
Shell
```
timedatectl status
```
If the certificate has expired, approve a new signing request or rotate it.
Shell
```
kubectl get csrkubectl certificate approve <csr-name>
```

Confirm it workedThe node reports Ready and the kubelet log is quiet.

Shell

kubectl get nodes -o wide

If you need to undo itKubelet configuration is in /var/lib/kubelet/config.yaml; keep a copy before editing.

Deal with an unresponsive container runtime

Root shell on the node40 minuteshigh riskreversible

PLEG is not healthy, or container operations time out.

PLEG is the kubelet's pod lifecycle event generator. It reports unhealthy when the runtime takes too long to list containers, which usually means the runtime is overloaded or wedged.
Shell
```
sudo crictl info | head -20sudo crictl ps 2>&1 | head
```
Check how many containers the node is holding — an accumulation of exited containers slows every list operation.
Shell
```
sudo crictl ps -a | wc -lsudo crictl pods | wc -l
```
PLEG lists every container on every cycle. A node holding thousands of dead containers cannot complete that within the timeout, and the node flaps between Ready and NotReady.

Remove exited containers.

Shell

sudo crictl rm --all --force 2>/dev/null || true

Check I/O wait, which is the other common cause.
Shell
```
iostat -x 2 5vmstat 2 5
```

Cordon and drain before restarting the runtime, so workloads move rather than being killed.

Shell

kubectl cordon node01kubectl drain node01 --ignore-daemonsets --delete-emptydir-datasudo systemctl restart containerd kubeletkubectl uncordon node01

Confirm it workedThe node stays Ready and PLEG warnings stop appearing.

Shell

sudo journalctl -u kubelet --since '10 min ago' | grep -ci pleg

If you need to undo itkubectl uncordon returns the node to service; nothing persistent was changed.

Where this stops. This write-up was written and checked by hand. It says what each step changes, how to confirm it worked and how to reverse it, and anything destructive is flagged before you reach it. If it does not match what your machine is doing, search the Support Centre for the exact code or message — and when something needs a person, get in touch.