Node Debugging — NotReady, Resource Pressure, kubelet

Tại sao quan trọng ở Production

Pods tidak bisa schedule kalau nodes tidak healthy. Ketika node problem:

Pods pending atau evicted
Existing pods bisa tiba-tiba terminate
Entire cluster capacity reduced
Hard to spot karena node problem vs pod problem berbeda

Node di GKE adalah Compute Engine VM. Problem bisa dari:

Kubernetes layer (kubelet, containerd)
OS layer (kernel, systemd, disk)
Hardware (network, storage, CPU)
GCP layer (VM issue, disk performance)

Anda perlu methodical approach untuk narrow down layer mana yang broken.

Internal Model: Node Health State Machine

Node Status dan Conditions

Node punya status yang kubelet manage:

yaml

status:
  phase: Running  # deprecated
  conditions:
  - type: Ready
    status: "True"
    reason: "KubeletReady"
    message: "kubelet posting ready status"
  - type: MemoryPressure
    status: "False"
  - type: DiskPressure
    status: "False"
  - type: PIDPressure
    status: "False"
  - type: NetworkUnavailable
    status: "False"

Ready=True: node ready to accept pods. Scheduling work normally.

Ready=False atau Unknown: node not ready. Kubelet stop accepting new pods. Existing pods may be evicted.

Node NotReady Root Causes

Kubelet report NotReady kalau:

Kubelet process down (cannot connect to node for 5 min)
Kubelet crash loop (keep restarting)
Container runtime fail (containerd down, cannot launch containers)
Cgroup system fail (cannot manage resources)
Memory/Disk pressure extreme (cannot allocate resources)
Network interface down (kubelet cannot reach API server)

Kubelet Architecture for Node Health

Kubelet pada node background task:

1. Heart Beat: every 10 seconds, send NodeStatus to API server
   - Report Ready condition
   - Report resource available (allocatable)
   - Report conditions (MemoryPressure, DiskPressure, etc.)

2. Watch kubelet itself:
   - Monitor PID count (if too high → PIDPressure=True)
   - Monitor disk usage (if >85% → DiskPressure=True)
   - Monitor memory available (if <100Mi free → MemoryPressure=True)
   - Monitor network interface (if down → NetworkUnavailable=True)

3. If heartbeat fail (cannot reach API server):
   - kubelet keep trying every 5 seconds
   - After 5 min without successful heartbeat → node status Unknown
   - Controller manager mark node NotReady (after additional delay)

Timing important: Node don't go NotReady immediately. Delay terjadi untuk prevent flapping (quick up-down-up).

Container Runtime Interface

Kubelet not directly manage containers. Kubelet call container runtime API:

kubelet ←→ CRI (Container Runtime Interface)
         ├─ CreateContainer
         ├─ StartContainer
         ├─ StopContainer
         ├─ RemoveContainer
         └─ InspectContainer

Container Runtime (containerd/docker):
- Allocate resources
- Mount volumes
- Setup networking
- Run process

Jika container runtime unreachable atau broken → kubelet cannot manage pods → Node go NotReady.

Debugging Node NotReady

Step 1: Verify Node is Actually NotReady

bash

kubectl get nodes
# Look untuk STATUS column
# "NotReady" or "Ready"

# If NotReady, check when it happened
kubectl describe node <node-name>
# Look untuk "Conditions" section
# Check "LastHeartbeatTime", "LastTransitionTime"

# Example output:
# Conditions:
#   Ready: False
#   LastHeartbeatTime: Wed, 24 Jun 2025 10:30:45 +0000
#   LastTransitionTime: Wed, 24 Jun 2025 10:30:15 +0000

Step 2: SSH to Node, Check kubelet

bash

ssh <node-ip>

# Check kubelet process
sudo systemctl status kubelet
# Look untuk:
# - Is service running?
# - Exit code (if stopped)?
# - Recent restart?

# Check kubelet logs
sudo journalctl -u kubelet -n 50 --no-pager
# Look untuk error messages dari last 50 lines

# Specific error patterns:
# - "cannot reach API server" → network problem
# - "cannot connect container runtime" → containerd down
# - "cgroup error" → resource problem
# - "disk full" → storage issue

Step 3: Check Container Runtime (containerd)

GKE use containerd as container runtime:

bash

ssh <node-ip>

# Check containerd service
sudo systemctl status containerd
# Is it running?

# Check containerd logs
sudo journalctl -u containerd -n 50 --no-pager

# Try connect to containerd
sudo crictl ps
# If work: containerd responsive
# If error "cannot connect": containerd socket problem

# Check containerd socket
ls -la /run/containerd/containerd.sock
# File should exist and be writable by kubelet user

Step 4: Check Node Resources

Memory Pressure

bash

ssh <node-ip>

# Check free memory
free -h
# Total, Used, Free, Available
# If Available < 100Mi → likely MemoryPressure=True

# Check page cache
grep Cached /proc/meminfo

# Check swap
swapon --show
# If swap active and high usage → might be memory pressure

# Find what's using memory
top -b -n 1 -o %MEM | head -20
# Top 20 processes by memory usage

If memory pressure:

Check if pods consuming too much:

bash

kubectl top pods --all-namespaces | sort -k3 -rn | head -20

Check if system pods (kubelet, containerd) leak memory:

bash

ps aux | grep kubelet
# VSZ column: virtual memory
# RSS column: resident memory

ps aux | grep containerd

Solution:
- Evict some pods (scale down deployments)
- Increase node memory (GCP: change machine type → need drain node first)
- Find memory leak in application (heap dump, check for OOMKilled)

Disk Pressure

bash

ssh <node-ip>

# Check disk usage
df -h
# Look untuk "/" (root filesystem)
# If Used > 85% → likely DiskPressure=True

# Check what's using disk
du -sh /*
# Largest directories at root level

# Specific problem areas:
du -sh /var/lib/kubelet/pods
# Pod volumes mounted here, could be large

du -sh /var/lib/containerd
# Container images and storage

# Check inode usage (if inode exhausted → "disk full" error)
df -i
# Look untuk "IUsed%" > 85%

If disk pressure:

Clean up old images:

bash

# Via kubelet configuration: imageGCLowThresholdPercent=50, imageGCHighThresholdPercent=80
# kubelet auto-clean old images when usage high

# Manual cleanup
sudo crictl rmi --all

Clean up pods/containers:

bash

sudo crictl ps -a --quiet | xargs -I {} sudo crictl rm {}

Check for large log files:

bash

find /var/log -type f -size +100M
# Rotate or delete old logs

CPU / Process Issues

bash

ssh <node-ip>

# Check CPU usage
top -b -n 1 | head -20
# Top processes

# Check CPU throttling (if using cgroups v2)
grep cpu.stat /proc/cgroups

# Check PID usage
cat /proc/sys/kernel/pid_max
ps aux | wc -l
# If close to limit → PIDPressure=True

Step 5: Check Network Connectivity

bash

ssh <node-ip>

# Check network interface up
ip link show
# eth0 should be UP

# Check routes
ip route
# Should have route to API server

# Test connectivity to API server
curl -v https://<api-server-ip>:443/healthz
# If timeout → network unreachable

# Check DNS
cat /etc/resolv.conf
nslookup kubernetes.default.svc.cluster.local

# Check firewall
sudo iptables -L -n | head -30
# Look untuk DROP rules yang might block API traffic

Step 6: Check Kubelet Configuration

bash

ssh <node-ip>

# Kubelet config usually at:
cat /etc/kubernetes/kubelet/kubelet-config.yaml

# or from docker inspect (if using kubeadm/GKE):
sudo ps aux | grep kubelet
# See flag values

# Key config to check:
# - --max-pods: max pods per node (default 110)
# - --cgroup-driver: cgroup driver match with containerd
# - --kubelet-cgroups: cgroup path

Specific Node Failure Modes

Failure Mode 1: Containerd Crash

bash

kubectl describe node
# Conditions: Ready=False, reason="NodeStatusUnknown"

ssh <node>
sudo systemctl status containerd
# Status: inactive

Root cause:

OOMKilled by kernel
Crash during image pull
Configuration wrong
Kernel upgrade break containerd

Debugging:

bash

# Check why it crashed
sudo journalctl -u containerd -n 100 --no-pager

# Restart containerd
sudo systemctl restart containerd

# Verify it come up
sudo systemctl status containerd
sudo crictl ps

If containerd keep crashing:

Check containerd config:
bash
```
cat /etc/containerd/config.toml
```

Check for corrupted state:

bash

sudo rm -rf /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
sudo systemctl restart containerd

If all fail, recreate node:

bash

gcloud container node-pools update <pool> \
  --node-taints="" \
  --num-nodes 0  # drain first
# Then increase back

Failure Mode 2: Kubelet Disk Space Exhausted

bash

ssh <node>
df -h /
# /dev/sda1     50G   49G  1G  98%  /

# kubelet cannot create new pod (cgroup allocation fail)

Quick fix:

bash

# Clean up container runtime
sudo crictl rmi --all

# Or clean specific pod volumes
sudo rm -rf /var/lib/kubelet/pods/*
# This delete all pod data! Use carefully

# Restart kubelet
sudo systemctl restart kubelet

Long term fix:

Increase node disk size (GCP: increase boot disk size)
Implement storage cleanup strategy (image GC, log rotation)

Failure Mode 3: Network Interface Down

bash

ssh <node>
ip link show
# eth0: DOWN (not UP)

# Or check kubelet logs
sudo journalctl -u kubelet | grep -i network
# "error: failed to setup network: ..."

Root cause:

NIC physically unplugged (rare in cloud)
Network driver issue
GCP network configuration broken

Debug:

bash

# Check NIC status
ethtool eth0
# Check carrier (on/off)

# Check GCP side
gcloud compute instances describe <instance-name>
# Check network interface status

# Reboot node (might help with driver issue)
# But this will cause downtime! Do carefully
sudo reboot

# Or recreate node via GKE API
gcloud container nodes create ...

Debugging Tools

`kubectl top` for Node Resource Usage

bash

kubectl top nodes
# Shows CPU and memory usage per node

# Compare with node capacity
kubectl describe nodes | grep "Allocatable" -A 10

Cloud Logging for kubelet/containerd Logs

bash

# Query kubelet logs
gcloud logging read \
  'resource.type="k8s_node" AND component="kubelet"' \
  --limit 50

# Query for specific errors
gcloud logging read \
  'resource.type="k8s_node" AND jsonPayload.level="ERROR"' \
  --limit 50

# Query containerd logs
gcloud logging read \
  'resource.type="k8s_node" AND component="containerd"' \
  --limit 50

Node Auto-Recovery

GKE bisa auto-repair nodes if configured:

bash

gcloud container node-pools describe <pool>
# Look untuk "Management" → "autoRepair: true"

# If enabled, GKE detect unhealthy nodes and recreate
# Takes ~5-10 minutes

GCP-Specific Node Issues

GCP Compute Engine Instance Issues

Node di GKE adalah Compute Engine VM. Problem bisa from GCP side:

bash

# Check instance health
gcloud compute instances describe <instance-name>
# Look untuk "cpuPlatform", "machineType", "status"

# Check if instance had recent restart
gcloud compute operations list --filter="targetResource:<instance-name>"

# Check instance serial port output (early boot logs)
gcloud compute instances get-serial-port-output <instance-name> --port 1

GCP Disk Performance Issues

If node slow (not NotReady, just sluggish):

bash

ssh <node>

# Check disk I/O
iostat -x 1 5
# Look untuk %util, await time

# Check if IOPS limit reached
# Standard persistent disk: ~3 IOPS per GB
# SSD persistent disk: ~30 IOPS per GB

# Solution: increase disk size (auto-increase disk IOPS)
# Or use SSD disk (faster I/O)

GCP Zone Maintenance / Disruption

Sometimes node NotReady because of GCP zone maintenance:

bash

# Check maintenance schedule
gcloud compute zones list --filter="name:<zone>" --format="value(name, scheduledMaintenance)"

# If scheduled maintenance → node might restart soon
# GKE might auto-reschedule pods

# To avoid maintenance:
# 1. Use different zone (geo-redundancy)
# 2. Schedule maintenance window when traffic low
# 3. Use node affinity to spread replicas

Operational Practices

1. Node Taint & Drain for Maintenance

Before reboot node (upgrade OS, etc):

bash

# Cordon node (stop new pods from scheduling)
kubectl cordon <node>

# Drain existing pods (move to other nodes)
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# After maintenance, uncordon
kubectl uncordon <node>

2. Monitor Node Health

bash

# Check via Prometheus / Cloud Monitoring
# Alert on:
# - node.status.conditions[Ready] != True
# - node.status.allocatable[memory] < threshold
# - kubelet restarts in last hour > 5

3. Set Resource Requests Properly

If every node OOMKilled:

bash

# Check if pods have resource requests
kubectl get pods -o yaml | grep -A3 "resources:"

# Pods without request → kubelet cannot enforce limits
# Set resource requests to prevent overcommit

Node Debugging — NotReady, Resource Pressure, kubelet ​

Tại sao quan trọng ở Production ​

Internal Model: Node Health State Machine ​

Node Status dan Conditions ​

Node NotReady Root Causes ​

Kubelet Architecture for Node Health ​

Container Runtime Interface ​

Debugging Node NotReady ​

Step 1: Verify Node is Actually NotReady ​

Step 2: SSH to Node, Check kubelet ​

Step 3: Check Container Runtime (containerd) ​

Step 4: Check Node Resources ​

Memory Pressure ​

Disk Pressure ​

CPU / Process Issues ​

Step 5: Check Network Connectivity ​

Step 6: Check Kubelet Configuration ​

Specific Node Failure Modes ​

Failure Mode 1: Containerd Crash ​

Failure Mode 2: Kubelet Disk Space Exhausted ​

Failure Mode 3: Network Interface Down ​

Debugging Tools ​

kubectl top for Node Resource Usage ​

Cloud Logging for kubelet/containerd Logs ​

Node Auto-Recovery ​

GCP-Specific Node Issues ​

GCP Compute Engine Instance Issues ​

GCP Disk Performance Issues ​

GCP Zone Maintenance / Disruption ​

Operational Practices ​

1. Node Taint & Drain for Maintenance ​

2. Monitor Node Health ​

3. Set Resource Requests Properly ​

References ​

Node Debugging — NotReady, Resource Pressure, kubelet

Tại sao quan trọng ở Production

Internal Model: Node Health State Machine

Node Status dan Conditions

Node NotReady Root Causes

Kubelet Architecture for Node Health

Container Runtime Interface

Debugging Node NotReady

Step 1: Verify Node is Actually NotReady

Step 2: SSH to Node, Check kubelet

Step 3: Check Container Runtime (containerd)

Step 4: Check Node Resources

Memory Pressure

Disk Pressure

CPU / Process Issues

Step 5: Check Network Connectivity

Step 6: Check Kubelet Configuration

Specific Node Failure Modes

Failure Mode 1: Containerd Crash

Failure Mode 2: Kubelet Disk Space Exhausted

Failure Mode 3: Network Interface Down

Debugging Tools

`kubectl top` for Node Resource Usage

Cloud Logging for kubelet/containerd Logs

Node Auto-Recovery

GCP-Specific Node Issues

GCP Compute Engine Instance Issues

GCP Disk Performance Issues

GCP Zone Maintenance / Disruption

Operational Practices

1. Node Taint & Drain for Maintenance

2. Monitor Node Health

3. Set Resource Requests Properly

References