My Awesome Project

Appearance

Pod Lifecycle Debugging — Pending, CrashLoopBackOff, OOMKilled ​

Tại sao quan trọng ở Production ​

Hầu hết GKE outages bắt đầu từ Pod không start hoặc crash liên tục. Nếu bạn không hiểu cách kubelet quản lý Pod lifecycle, debugging sẽ là blind guessing:

  • Pod stuck Pending: bạn không tìm hiểu tại sao scheduler không place nó, thay vào đó bạn random add resources hoặc node
  • Pod CrashLoopBackOff: bạn chỉ đọc logs mà không hiểu kubelet đã try restart bao nhiêu lần, restart policy là gì
  • OOMKilled: bạn không biết cgroup limit so với actual process memory usage

Kết quả: bạn tốn hàng giờ fireighting khi có structured approach có thể resolve trong 10 phút.

Internal Model: Pod Lifecycle State Machine ​

Pod lifecycle không phải random. Nó là explicit state machine được kubelet điều khiển:

Pod Lifecycle Stages ​

1. Pending
   ├─ Phase=Pending, Conditions=[Initialized=False, Ready=False, ContainersReady=False]
   └─ Meaning: kubelet chưa start container. Có thể:
      - Waiting for image pull
      - Waiting for storage mount
      - Scheduler chưa place pod (Pending → reason=Unschedulable)
      - Kubelet received pod object nhưng chưa start it

2. Running
   ├─ Phase=Running, Conditions=[Initialized=True, Ready=True, ContainersReady=True]
   └─ Meaning: container process running, passed readiness probe

3. Succeeded / Failed
   ├─ Phase=Succeeded: tất cả container exit code 0
   ├─ Phase=Failed: tối thiểu 1 container exit code != 0
   └─ Meaning: pod lifecycle complete, not restart

4. Unknown
   └─ Meaning: kubelet lost contact with pod (rare, usually node NotReady)

Kubelet Pod Status Transitions ​

Kubelet View pod object từ API server. Nó sẽ:

  1. Validate: Pod spec có valid không? Resource request có malformed không?

    • Nếu invalid → conditions = [...]Initialized=False" with "reason=UnexpectedAdmissionError"

  2. Create CGroup: Setup cgroup namespace, memory limits, CPU shares

    • Nếu fail (kernel old, memory unavailable) → Pod stuck Pending

  3. Setup Volumes: Mount PVC, ConfigMap, Secret

    • Nếu PVC không bind → Pod stuck Pending with "reason=PodInitializing"
    • Nếu mount fail (permission denied, disk error) → Pod stuck Pending

  4. Pull Image: Fetch container image từ registry

    • Nếu image tidak exist → Pod stuck Pending with "reason=ImagePullBackOff"
    • Nếu registry auth fail → Pod stuck Pending with "reason=ErrImagePull"

  5. Create Container: Call containerd/docker API untuk create container

    • Container runtime allocate resources, setup network interface
    • Nếu fail (OOMKilled during allocation) → Pod stuck Pending

  6. Run Container: Kubelet start container process

    • Container process PID 1 mulai, run entrypoint command
    • Nếu process exit immediately → Phase=Failed or CrashLoopBackOff

  7. Execute Probes: Run startup, readiness, liveness probes

    • Startup probe: check apakah app started
    • Readiness probe: check apakah app ready for traffic
    • Liveness probe: check apakah app still alive
    • Probe fail → Pod unhealthy, kubelet trigger action per probe config

Pod Conditions: Granular Health Status ​

Pod .status.conditions adalah structured data tentang Pod health:

yaml
status:
  conditions:
  - type: Initialized
    status: "True"
    reason: "PodInitialized"
  - type: Ready
    status: "False"
    reason: "ContainersNotReady"
  - type: ContainersReady
    status: "False"
    reason: "Init:0/1"  # Init container 0 of 1 completed
  - type: PodScheduled
    status: "True"
    reason: "Successfully assigned"

Ini sangat penting: Jangan hanya View Pod.status.phase. View conditions untuk tahu state mana yang fail.

RestartPolicy: Kubelet Retry Logic ​

Pod spec memiliki .spec.restartPolicy:

  • Always (default): Container exit → kubelet restart

    • Exponential backoff: wait 1s, 2s, 4s, 8s, ..., max 5 min
    • Setelah multiple failures dalam 5 min → Pod status CrashLoopBackOff

  • OnFailure: Container exit code != 0 → kubelet restart, exit code 0 → no restart

  • Never: Container exit → no restart, Pod status Failed

CrashLoopBackOff bukan status field, itu wait reason: Pod phase masih Running, tapi kubelet sedang wait sebelum restart. View .status.containerStatuses[0].state.waiting.reason = "CrashLoopBackOff".

Container State: Waiting vs Running vs Terminated ​

Setiap container dalam Pod memiliki state:

yaml
containerStatuses:
- name: app
  state:
    waiting:
      reason: "CrashLoopBackOff"  # atau "ImagePullBackOff", "CreateContainerConfigError"
      message: "back-off 5m0s restarting failed container=app pod=..."
    # atau
    running:
      startedAt: "2025-06-24T10:23:45Z"
    # atau
    terminated:
      reason: "OOMKilled"  # atau "Error", "Completed"
      exitCode: 137  # 128 + 9 = OOMKilled
      signal: 9
      startedAt: "2025-06-24T10:23:40Z"
      finishedAt: "2025-06-24T10:23:45Z"

Jangan View Pod.status.phase saja. View .containerStatuses[].state untuk tahu container actually doing what.

Debugging Each Pod Failure State ​

State 1: Pod Stuck Pending ​

Visible behavior: kubectl get pod → STATUS=Pending (untuk >10 menit)

Diagnostic checklist:

bash
# Step 1: Check pod description untuk events
kubectl describe pod <name> -n <ns>
# Look untuk events yang paling recent:
# - "0/3 nodes are available" → insufficient resources
# - "PodInitializing" → volume/secret mount pending
# - "ImagePullBackOff" → image tidak ada
# - "Unschedulable" → tapi ada reason: "node selector does not match"

Case A: Scheduler tidak place pod (Unschedulable)

Conditions:
  PodScheduled: False
  Reason: "Unschedulable"

Events:
  Warning  Unschedulable    2m    default-scheduler
    0/3 nodes are available: 3 Insufficient memory

Artinya: Pod request > available resources di semua nodes. Diagnostics:

bash
# Check pod request
kubectl get pod <name> -o yaml | grep -A2 "resources:"

# Check node capacity
kubectl describe nodes
# Look untuk "Allocated resources" section
# Compare: Pod request > Node allocatable → Pod cannot fit

# Root causes:
# 1. Pod requests terlalu besar
# 2. Node pool sudah full dengan pods lain
# 3. Node autoscaler gagal scale up

Jika scheduler tidak bisa place pod, itu bukan kubelet problem. Itu cluster capacity problem.

Case B: Pod init blocked (PodInitializing)

Waiting:
  Reason: "PodInitializing"
  Message: "containers with incomplete status: [app]"

Events:
  Normal  Pulling    3m    kubelet
    Pulling image "my-app:latest"

Artinya: Kubelet sedang pull image atau mount volume. Tunggu saja jika image kecil. Tapi kalau stuck >5 menit:

bash
# Check kubelet logs untuk image pull errors
gcloud logging read "resource.type=k8s_node AND jsonPayload.message=~'ImagePull|image'" \
  --limit 50

# Check apakah docker/containerd bisa pull image
ssh <node>
sudo crictl pull <image>

# Kalau gagal, debug registry connectivity
sudo crictl pull --debug <image>

Case C: Volume mount blocked

Conditions:
  PodScheduled: True
  Initialized: False
  Reason: "PodInitializing"

Events:
  Warning  FailedAttachVolume  2m  attachdetach-controller
    Failed to attach volume "pvc-xxx" to node "node-1":
    GCE Error: RESOURCE_EXHAUSTED, no more storage quota

Kubelet sudah place pod, tapi volume belum attach. Debug:

bash
# Check PVC status
kubectl get pvc
kubectl describe pvc <name>

# Check apakah backing GCP disk exist
gcloud compute disks list --filter="name:pvc-xxx"

# Check quota
gcloud compute project-info describe --format='value(quotas)'

State 2: Pod CrashLoopBackOff ​

Visible behavior: Pod restart berkali-kali. Check:

bash
kubectl get pod <name> -o yaml | grep -A20 "containerStatuses"

Output:

yaml
containerStatuses:
- name: app
  restartCount: 15
  state:
    waiting:
      reason: "CrashLoopBackOff"
  lastState:
    terminated:
      exitCode: 1
      reason: "Error"

Meaning: Container exited dengan exit code 1, kubelet tried restart 15 kali, now waiting before retry (exponential backoff).

Root causes (by frequency):

  1. Application error: Entrypoint command crash

    bash
    kubectl logs <pod> --previous
# Check last output sebelum crash
# Kalau empty → application silent exit, check stderr too

# Kalau logs cut off, check kubelet logs
gcloud logging read "resource.type=k8s_node AND jsonPayload.reason='Error'" --limit 10

  2. Missing dependencies: Application expect file/socket yang tidak ada

    bash
    kubectl exec <pod> -it -- /bin/sh
# Apakah file ada?
ls -la /config
env | grep DB_HOST
# Jalankan application command manually

  3. Resource constraints: Process killed by kernel OOM

    bash
    # Check exit code
exit_code = container.state.terminated.exitCode
if exit_code == 137:  # 128 + 9 (SIGKILL)
  # Possible OOMKilled
  # Tapi juga bisa liveness probe failed and kubelet kill
  kubectl describe pod | grep "Last State" → reason

# If "OOMKilled" → jump to OOMKilled section

  4. Liveness probe too aggressive

    bash
    kubectl get pod <name> -o yaml | grep -A10 "livenessProbe"

# Check kalau:
# - initialDelaySeconds < time to start app
# - failureThreshold too low
# - periodSeconds too frequent

Debugging approach:

bash
# 1. Get last exit reason
kubectl describe pod <name>
# Look untuk "Last State" → terminated.reason

# 2. Get last logs
kubectl logs <pod> --previous

# 3. Exec into pod sekarang (kalau running state)
kubectl exec <pod> -it -- /bin/sh
# Check dependencies, env vars, file mounts

# 4. Check kubelet logs untuk signal info
gcloud logging read \
  "resource.type=k8s_node AND jsonPayload.reason=~'Signal|OOM|Killed'" \
  --limit 20

State 3: OOMKilled ​

Visible behavior:

yaml
containerStatuses:
- name: app
  state:
    terminated:
      exitCode: 137  # 128 + 9 = SIGKILL
      signal: 9
      reason: "OOMKilled"

Meaning: Kernel OOM killer terminate process karena memory exhausted.

Case A: Container Limit Exceeded ​

Container spec:

yaml
resources:
  limits:
    memory: "512Mi"

Application menggunakan >512Mi → kernel kill process.

Debugging:

bash
# 1. Check memory metrics
kubectl top pod <name>
# MEMORY akan menunjukkan actual usage

# 2. Check container limit vs usage
kubectl get pod <name> -o yaml | grep -A3 "memory"

# 3. Check kernel OOM logs
gcloud logging read \
  "resource.type=k8s_node AND severity=ERROR AND jsonPayload=~'OOM|memory'" \
  --limit 20

# 4. SSH node dan check OOM killer logs
ssh <node>
dmesg | grep -i "killed process"
# Output: "[7684] 1000 7684 567481 123456  5  0   0  python Out of memory"

Root causes:

  1. Memory leak: Application hold memory ke-unakan. Debug:

    bash
    kubectl exec <pod> -- top -b -n 1
# Check process VSZ, RSS terus-menerus naik?

# Kalau Java:
kubectl exec <pod> -- jps -l
# Dump heap untuk analisis

  2. Limit terlalu kecil: Set based on SLA, bukan actual test

    bash
    # Increase limit
kubectl set resources deployment <name> \
  --limits=memory=2Gi

  3. Spike traffic: Sudden load cause memory spike

    bash
    # Check traffic metrics
kubectl top pods --all-namespaces | head -20
# See if traffic spike correlate dengan OOMKilled time

Case B: Node-Level OOM (Rare, More Serious) ​

Entire node memory exhausted → kubelet itself cannot function → Pod evicted.

status:
  reason: "Evicted"
  message: "The node had condition: memory.pressure=True"

Kubelet monitor node resources. Kalau memory pressure tinggi:

  1. Kubelet start evict pods (berdasarkan QoS, priority)

    • Burstable pods (punya requests tapi tidak limit) → evict first
    • Best-effort pods → evict next
    • Guaranteed pods (request == limit) → evict last

  2. Kubelet set node condition MemoryPressure=True

  3. New pods tidak bisa schedule ke node

Debugging:

bash
# Check node conditions
kubectl describe node <name> | grep -A20 "Conditions"
# Look untuk "MemoryPressure: True"

# Check evicted pods
kubectl get pods -A --field-selector=status.reason=Evicted

# SSH node untuk meView OS-level memory
ssh <node>
free -h
# Check kalau memang memory exhausted

# Check apa yang menggunakan memory
top -b -n 1 | head -15
ps aux --sort=-%mem | head -15

State 4: Init Container Failures ​

Pod spec bisa punya init containers (run before app containers):

yaml
spec:
  initContainers:
  - name: wait-db
    image: busybox
    command: ['sh', '-c', 'until nc -z db:5432; do sleep 1; done']
  containers:
  - name: app
    image: my-app

Kalau init container fail → entire Pod fail (tidak jalan app container).

Visible behavior:

Conditions:
  Initialized: False
  
Events:
  reason: Init:0/1  # Init container 0 completed, total 1
bash
kubectl logs <pod> -c wait-db --previous

Root causes:

  1. Init container exited with error:

    bash
    kubectl logs <pod> -c wait-db
# e.g., "nc: not found" → busybox image missing netcat

  2. Init container hanging (timeout waiting):

    bash
    # Check logs:
kubectl logs <pod> -c wait-db
# See kalau ada loop running

# Describe pod untuk tahu berapa lama waiting
kubectl describe pod <name>
# "waiting" condition untuk tahu berapa lama

  3. Dependency tidak siap (kalau init container wait for service):

    bash
    kubectl logs <pod> -c wait-db
# Error: "Connection refused" atau "Timeout"

# Check target service exist
kubectl get service db
# Kalau tidak exist atau Pod not ready → dependency issue

Constraints & Trade-offs ​

Container Limits vs Requests ​

yaml
resources:
  requests:
    memory: "256Mi"  # Kubelet guarantee ini amount
  limits:
    memory: "512Mi"  # Kernel OOM kill kalau exceed

Mental model:

  • Request = reservation. Scheduler hanya place pod kalau node ada request available
  • Limit = hard cap. Process tidak bisa exceed; kernel kill jika exceed

Trade-off:

  • High limit, low request = pod bisa overcommit → OOM risk ketika traffic spike
  • High request = pod guaranteed memory tapi cluster less flexible
  • Best practice: request = limit untuk predictable workloads, tapi 20-30% buffer untuk bursty workloads

Liveness vs Readiness vs Startup Probes ​

yaml
startupProbe:  # Check kalau app startup selesai (contoh: database migration)
  httpGet:
    path: /health
    port: 8080
  failureThreshold: 30  # Retry 30x sebelum fail pod
  periodSeconds: 10
  
readinessProbe:  # Check kalau app ready untuk traffic
  httpGet:
    path: /ready
    port: 8080
  failureThreshold: 3
  
livenessProbe:  # Check kalau app still alive
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 3
  periodSeconds: 10

Trade-off:

  • Aggressive probes (low failureThreshold, low periodSeconds) = quick detect failure tapi false positive risk
  • Lenient probes = tolerant untuk slow start/GC pause tapi slow detect real failure

Best practice:

  • startupProbe untuk lenient initial check
  • readinessProbe untuk strict traffic check
  • Liveness untuk detect hung process (jarang dibutuhkan kalau readiness sufficient)

Common Anti-patterns & Solutioning ​

Anti-pattern 1: Pod Pending But No Clear Reason ​

bash
kubectl describe pod
# Output: "0/3 nodes available" tapi logs tidak jelas kenapa

Mistake: Hanya View Pod description, tidak View scheduler logs.

Fix:

bash
# Check scheduler logs
gcloud logging read "resource.type=k8s_node AND component=scheduler" --limit 20

# Search untuk pod name
gcloud logging read "resource.type=k8s_node AND jsonPayload=~'pod-name' AND component=scheduler" --limit 10

Anti-pattern 2: Immediately Increase Resource Limits After OOMKilled ​

bash
# Mistake: kubectl set resources deployment app --limits=memory=10Gi
# Without debugging actual usage

Why wrong: Jika 10Gi juga tidak cukup, anda akan OOM lagi. Jika actual leak (memory leak), increasing limit hanya menunda problem.

Fix:

  1. Debug actual memory usage: kubectl top pod
  2. Kalau stable tapi high: increase limit 30-40%
  3. Kalau increasing → memory leak: restart pod dengan debug container untuk heap dump

Anti-pattern 3: CrashLoopBackOff Tapi Just Restart Service ​

bash
# Mistake: kubectl rollout restart deployment
# Tapi Pod akan restart lagi karena root cause tidak fixed

Fix: Debug first before restart. Root cause paling sering:

  • Missing config file atau env var
  • Dependency (database, cache) tidak siap
  • Permission error pada volume mount

Kalau root cause tidak fixed, pod akan crash lagi.

Anti-pattern 4: Init Container Hang Forever ​

yaml
initContainers:
- name: wait-db
  image: busybox
  command: ['sh', '-c', 'until nc -z db:5432; do sleep 1; done']
  # No timeout! Jika db never ready, init container hang forever

Fix: Add timeout:

yaml
initContainers:
- name: wait-db
  image: busybox
  command: ['sh', '-c', 'timeout 60 sh -c "until nc -z db:5432; do sleep 1; done"']

Atau use startupProbe instead:

yaml
startupProbe:
  exec:
    command: ['nc', '-z', 'db', '5432']
  failureThreshold: 30
  periodSeconds: 10

GCP-Specific Considerations ​

GKE Node Image & Container Runtime ​

GKE use containerd (not Docker) as container runtime. Container state message berbeda:

CrashLoopBackOff dari containerd vs Docker memiliki message sedikit berbeda:
- containerd: "back-off 5m0s restarting failed container"
- Docker: "Back-off restarting failed container"

Debugging dengan crictl (containerd CLI):

bash
ssh <node>
# List containers
sudo crictl ps -a

# Get container logs
sudo crictl logs <container-id>

# Get container status
sudo crictl inspect <container-id>

GKE Workload Identity & Permission ​

Kalau pod crash dengan "permission denied" error, bisa karena:

  1. Pod running dengan wrong service account

    bash
    kubectl get pod <name> -o yaml | grep serviceAccountName

  2. Service account tidak bind ke Google service account

    bash
    kubectl describe serviceaccount <sa-name>
# Check annotations: iam.gke.io/gcp-service-account

  3. Google service account tidak punya permission

    bash
    gcloud iam service-accounts get-iam-policy <gsa-email>

GKE Dataplane V2 & eBPF ​

GKE bisa enable Dataplane V2 yang menggunakan eBPF. Impact pada debugging:

  • Network policy enforcement via eBPF, bukan iptables
  • Container network interface setup berbeda
  • Kalau network error, bisa related ke eBPF verifier (compile time) atau eBPF runtime error

Debug:

bash
ssh <node>
# Check BPF programs
ip link show
# Look untuk "bpf(1)" di xdp section

# Check BPF errors
sudo dmesg | grep -i "verifier"

Operational Practices ​

1. Resource Request Tuning ​

Jangan hardcode resource request. Measure dari metrics:

bash
# Get 95th percentile CPU/memory over 1 week
kubectl get hpa <name> -o yaml
# Look untuk metrics dalam status

# Or use Cloud Monitoring
gcloud monitoring timeseries list \
  --filter='metric.type="kubernetes.io/pod/memory/used_bytes"' \
  --format='table(points[0].value.double_value)'

2. Probe Configuration ​

Startup probe untuk slow-start apps:

yaml
startupProbe:
  httpGet:
    path: /health
    port: 8080
  failureThreshold: 30  # 30 * 10s = 5 menit max startup
  periodSeconds: 10

Readiness probe untuk traffic routing:

yaml
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  failureThreshold: 2  # Fail after 2 consecutive failures
  periodSeconds: 5

3. Logging Strategy ​

Capture stderr juga untuk debugging:

Dockerfile:

dockerfile
# Redirect stderr to stdout sehingga kubectl logs dapat semua output
RUN exec 2>&1
CMD ["./app"]

References ​