Node Issues: NotReady Diagnosis

Symptoms Recognition

Node NotReady xuất hiện khi:

kubectl get nodes show node with NotReady status
Pod cannot schedule (nodeName point to NotReady node)
Workloads disappear from node (pod eviction)
No new Pods can start on node (capacity zero)
Node stuck NotReady for > 10 minutes

Why This Matters

NotReady nodes reduce cluster effective capacity — Kubernetes won't schedule Pods on them. Single NotReady node ở small cluster = 10-50% capacity loss. Trong production, cascading pod evictions → outages. Understanding node health diagnosis là critical để SRE.

Information Gathering — Quick Diagnostics

Step 1: Describe Node & Check Conditions

bash

# View node status
kubectl get nodes -o wide

# Detailed node info
kubectl describe node <node-name>
# Cari section "Conditions" và "Node Status"

# View node events
kubectl get events --field-selector involvedObject.name=<node-name> --all-namespaces --sort-by='.lastTimestamp'

Interpretasi Conditions:

Ready=False = Node NotReady (kubelet không reporting)
Ready=Unknown = kubelet timeout (health check hanging)
DiskPressure=True = disk usage high (> 85%)
MemoryPressure=True = memory usage high (> 85%)
PIDPressure=True = process count high (near limit)
NetworkUnavailable=True = networking not working

Step 2: Check Kubelet Status & Logs

bash

# SSH into node (hoặc use node-shell)
# First, get node internal IP
NODE_IP=$(kubectl get node <node-name> -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')

# Option 1: Use node-shell pod (easier, no SSH)
kubectl node-shell <node-name>

# Inside node shell:
# Check kubelet service status
systemctl status kubelet

# Check kubelet logs
journalctl -u kubelet -n 100 --no-pager

# Check container runtime
systemctl status docker  # hoặc containerd

# Check kubelet PID
ps aux | grep kubelet

Daftar để dipahami:

kubelet status = running vs failed?
Recent log entries: errors related đến networking, pod management?
Restart count: high restart = underlying issue?

Step 3: Check Node Resource Pressure

bash

# Describe node show resource metrics
kubectl describe node <node-name> | grep -A 20 "Allocated resources"

# Actual usage
kubectl top node <node-name>

# Check disk space
# Via node-shell:
df -h

# Check memory
free -h
cat /proc/meminfo

# Check processes
ps aux --sort=-%cpu | head -10
ps aux --sort=-%mem | head -10

Step 4: Check Kubernetes Components Health

bash

# Inside node-shell:
# Check CNI plugin
ps aux | grep -i cni
ls -la /opt/cni/bin/

# Check container runtime
ps aux | grep docker/containerd
# For containerd:
systemctl status containerd

# Check kubelet config
cat /etc/kubernetes/kubelet.conf | head -20

# Check kubelet health check endpoint
curl http://localhost:10248/healthz

Step 5: Cluster Autoscaler & Auto-Repair Status

bash

# Check CA logs
kubectl logs -n kube-system deployment/cluster-autoscaler | tail -50

# Check if node pool has auto-repair
gcloud container node-pools describe <pool-name> \
  --cluster=<cluster> --zone=<zone> | grep autoRepair

# Check GCP VM status
gcloud compute instances describe <instance-name> --zone=<zone>
# Status should be "RUNNING"

Diagnostic Decision Tree

Node NotReady
│
├─ Node actually running?
│  ├─ NO (GCP VM stopped) → start instance
│  │
│  └─ YES (VM RUNNING) → continue
│
├─ Kubelet running?
│  ├─ NO (process not found) → restart kubelet
│  │  └─ systemctl restart kubelet
│  │
│  └─ YES (running) → continue
│
├─ Kubelet logs show errors?
│  ├─ Connection refused → container runtime not running
│  │  └─ Restart docker/containerd
│  │
│  ├─ "SIGKILL" → OOM killed
│  │  └─ Scale down workloads hoặc increase node memory
│  │
│  ├─ Network timeouts → network issue
│  │  └─ Check network connectivity, firewall
│  │
│  └─ Other errors → analyze specific message
│
├─ Disk pressure (> 85%)?
│  ├─ YES → cleanup disk, scale node, or add disk
│  │
│  └─ NO → continue
│
├─ Memory pressure (> 85%)?
│  ├─ YES → pod eviction happening
│  │  └─ Check memory usage per pod
│  │
│  └─ NO → continue
│
├─ Network connectivity?
│  ├─ NO (curl google.com timeout) → network issue
│  │  └─ Check firewall, route, physical interface
│  │
│  └─ YES → continue
│
├─ Container runtime health?
│  ├─ Docker/Containerd not running → restart
│  ├─ Daemon socket missing → reinstall
│  │
│  └─ Running → continue
│
└─ Auto-repair enabled?
   ├─ YES → wait (should auto-repair in 10-15 min)
   ├─ NO → manual fix needed

Common Root Causes & Fixes

Root Cause 1: Kubelet Crash / Resource Starvation

Dấu hiệu:

Kubelet process not running
Recent restart visible trong logs
Node conditions show high memory/CPU pressure
Message: "kubelet: OOMKilled" trong journalctl

Nguyên nhân: Kubelet OOMKilled vì insufficient memory, hoặc crash due to bug.

Diagnostic:

bash

# Check kubelet resource limits
cat /etc/kubernetes/kubelet.conf | grep -i memory
# On GKE nodes, usually: --kubeReserved=memory=500Mi

# Check actual pod count consuming memory
kubectl get pods --field-selector spec.nodeName=<node-name> --all-namespaces

Immediate Fix:

bash

# Option 1: Restart kubelet
systemctl restart kubelet

# Option 2: Drain & replace node (safest)
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
# GKE will replace node (if auto-repair enabled)

# Option 3: Scale node to larger machine type (increase memory)
# Create new node pool, migrate workloads

Permanent Fix:

Increase kubeReserved memory nếu consistently OOMKilled
Implement pod resource limits (LimitRange)
Use Vertical Pod Autoscaler để right-sizing pods

Root Cause 2: Container Runtime (Docker/Containerd) Not Running

Dấu hiệu:

Kubelet logs: "Cannot get docker client" hoặc "containerd connection refused"
systemctl status docker = inactive
Kubelet cannot create containers

Nguyên nhân: Container runtime crashed, service disabled, socket file missing.

Diagnostic:

bash

# Check docker/containerd status
systemctl status docker
systemctl status containerd

# Check socket file
ls -la /var/run/docker.sock  # for docker
ls -la /run/containerd/containerd.sock  # for containerd

# Check for crash loops
journalctl -u docker -n 50
journalctl -u containerd -n 50

Immediate Fix:

bash

# Option 1: Restart container runtime
systemctl restart docker  # or containerd

# Option 2: If socket missing, reinstall
# (GKE managed, unlikely to happen)
# Manual reinstall if needed:
# apt-get install docker.io  # or containerd package

# Option 3: Drain and replace node
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
gcloud compute instances delete <instance-name> --zone=<zone>
# New node will be created by autoscaler

Permanent Fix:

Ensure container runtime auto-starts: systemctl enable docker
Monitor container runtime health (watchdog)
GKE nodes should have auto-repair enabled

Root Cause 3: Disk Pressure (> 85% Full)

Dấu hiệu:

Condition: DiskPressure=True
Event: "node has insufficient disk space"
Pods cannot start (cannot write logs)
kubelet cleanup (kill pods) attempting to free space

Nguyên nhân: Node disk filled up:

Log files accumulating (container logs, kubelet logs)
Unused container images
Pod ephemeral storage exhausted

Diagnostic:

bash

# Check disk usage
df -h /

# Disk usage per directory
du -sh /* | sort -h

# Container logs size
du -sh /var/log/containers/

# Docker/containerd image storage
du -sh /var/lib/docker  # for docker
du -sh /var/lib/containerd  # for containerd

Immediate Fix:

bash

# Option 1: Cleanup old logs
journalctl --vacuum=time:7d  # Keep last 7 days

# Option 2: Cleanup container logs
docker system prune  # Remove dangling images/containers
# or for containerd:
ctr images ls | head -20  # List images
ctr images rm <image-ref>  # Remove unused

# Option 3: Increase disk size (GCP persistent disk)
gcloud compute disks resize <disk-name> --size=100GB

# Option 4: Drain & replace node
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
# Auto-repair or manual create new node

Permanent Fix:

Kubelet garbage collection:

bash

# Kubelet automatically cleanup old pods/images
# Configure --image-gc-high-threshold (default 85%)
# Configure --image-gc-low-threshold (default 80%)

Log rotation & cleanup:

bash

# Configure logrotate for kubelet logs
# Pod logs auto-deleted by container runtime

Node disk size planning:
- GKE default 100GB root disk
- For busy nodes: increase đến 200GB
- For data-heavy: separate data disk
Monitoring:
- Alert if disk usage > 80%
- Proactive drain before full

Root Cause 4: Memory Pressure (> 85% Used)

Dấu hiệu:

Condition: MemoryPressure=True
Event: "node has insufficient memory"
Pod evictions starting (QoS-based)
Pods with low priority killed first

Nguyên nhân: Node memory exhausted:

Pods without memory limits consuming unchecked
Workload spike unexpected
Memory leak trong application

Diagnostic:

bash

# Check memory usage
free -h
cat /proc/meminfo

# Memory by process
ps aux --sort=-%mem | head -10

# Memory by pod
kubectl top pods --all-namespaces --sort-by=memory

# Pod memory requests vs actual
kubectl get pods --all-namespaces -o json | \
  jq '.items[] | {name: .metadata.name, namespace: .metadata.namespace, request: .spec.containers[].resources.requests.memory, limit: .spec.containers[].resources.limits.memory}'

Immediate Fix:

bash

# Option 1: Scale down/delete unnecessary pods
kubectl scale deployment <dep> --replicas=0 -n <namespace>

# Option 2: Evict lowest priority pods manually
# Kubernetes will do this automatically, but can force:
kubectl delete pod <low-priority-pod> -n <namespace>

# Option 3: Increase node memory (if possible)
# Scale node to larger machine type

# Option 4: Drain & replace node
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Permanent Fix:

Enforce resource requests/limits:

yaml

# LimitRange per namespace
apiVersion: v1
kind: LimitRange
metadata:
  name: pod-limits
spec:
  limits:
  - max:
      memory: "2Gi"
    min:
      memory: "128Mi"
    defaultRequest:
      memory: "256Mi"
    type: Container

QoS class management:
- Guaranteed: requests == limits
- Burstable: request < limit
- BestEffort: no requests/limits (evicted first)
- For production: avoid BestEffort
Vertical Pod Autoscaler:
- Recommendations để right-sizing memory

Root Cause 5: Network Connectivity Issue

Dấu hiệu:

Condition: NetworkUnavailable=True
Kubelet cannot reach API server (timeout)
Node never becomes Ready
Network interface missing

Nguyên nhân: Network issue:

Virtual network interface down
Firewall blocking cluster network
IP misconfiguration
GCP VPC peering broken

Diagnostic:

bash

# Check network interfaces
ip addr show
# Should see eth0 (primary) with IP address

# Check routing
ip route show
# Should have default route

# Test connectivity to API server
curl https://<api-server-ip>:443 -k

# Check firewall rules
# GCP Console → VPC network → Firewall rules
gcloud compute firewall-rules list

# Check network interfaces
gcloud compute instances describe <instance-name> --zone=<zone> | grep networkInterface

Immediate Fix:

bash

# Option 1: Reset network interface
ip link set eth0 down
ip link set eth0 up

# Option 2: Restart networking service
systemctl restart networking  # Debian-based
systemctl restart network  # RHEL-based

# Option 3: Restart GCP VM
gcloud compute instances reset <instance-name> --zone=<zone>

# Option 4: Drain & replace node
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
# Create new node

Permanent Fix:

Network configuration managed by GCP
Auto-repair should handle recovery
Monitor network connectivity proactively

Prevention & Monitoring

Node Health Monitoring

bash

# Cloud Monitoring dashboards:

# Dashboard: Node Health Status
# Metrics:
#   - kubernetes.io/node/condition (broken down by condition type)
#   - compute.googleapis.com/instance/status (VM level)
#   - kubelet uptime
#   - container runtime health

# Alert Rules:
#   - Node NotReady > 5 min → Page
#   - Node condition change (DiskPressure/MemoryPressure) → Alert
#   - High pod eviction rate → Alert

Auto-Repair Configuration

bash

# Ensure auto-repair enabled (GKE default: yes)
gcloud container node-pools describe <pool> \
  --cluster=<cluster> --zone=<zone> | grep -A5 autoRepair

# If disabled, enable:
gcloud container node-pools update <pool> \
  --cluster=<cluster> --zone=<zone> \
  --enable-autorepair

Node Problem Detector

yaml

# Optional: Deploy Node Problem Detector
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-problem-detector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: node-problem-detector
  template:
    metadata:
      labels:
        app: node-problem-detector
    spec:
      containers:
      - name: detector
        image: gke.gcr.io/node-problem-detector:v0.8.14-gke.0
        # Detects kernel deadlock, file system corruption, etc

Escalation Criteria

Escalate if:

Node remains NotReady > 30 minutes despite auto-repair
Multiple nodes NotReady concurrently (infrastructure issue)
Disk/network issues persist after node replacement
Suspected hardware failure (repeated crashes)

Pod Creation Failures — Nếu pod không schedule đến node
Scheduling Failures — Node capacity zero
Storage Issues — Nếu disk pressure related

Node Issues: NotReady Diagnosis ​

Symptoms Recognition ​

Why This Matters ​

Information Gathering — Quick Diagnostics ​

Step 1: Describe Node & Check Conditions ​

Step 2: Check Kubelet Status & Logs ​

Step 3: Check Node Resource Pressure ​

Step 4: Check Kubernetes Components Health ​

Step 5: Cluster Autoscaler & Auto-Repair Status ​

Diagnostic Decision Tree ​

Common Root Causes & Fixes ​

Root Cause 1: Kubelet Crash / Resource Starvation ​

Root Cause 2: Container Runtime (Docker/Containerd) Not Running ​

Root Cause 3: Disk Pressure (> 85% Full) ​

Root Cause 4: Memory Pressure (> 85% Used) ​

Root Cause 5: Network Connectivity Issue ​

Prevention & Monitoring ​

Node Health Monitoring ​

Auto-Repair Configuration ​

Node Problem Detector ​

Escalation Criteria ​

Related Sections ​

References ​

Node Issues: NotReady Diagnosis

Symptoms Recognition

Why This Matters

Information Gathering — Quick Diagnostics

Step 1: Describe Node & Check Conditions

Step 2: Check Kubelet Status & Logs

Step 3: Check Node Resource Pressure

Step 4: Check Kubernetes Components Health

Step 5: Cluster Autoscaler & Auto-Repair Status

Diagnostic Decision Tree

Common Root Causes & Fixes

Root Cause 1: Kubelet Crash / Resource Starvation

Root Cause 2: Container Runtime (Docker/Containerd) Not Running

Root Cause 3: Disk Pressure (> 85% Full)

Root Cause 4: Memory Pressure (> 85% Used)

Root Cause 5: Network Connectivity Issue

Prevention & Monitoring

Node Health Monitoring

Auto-Repair Configuration

Node Problem Detector

Escalation Criteria

Related Sections

References