Pod Creation Failures: Troubleshooting Checklist

Symptoms Recognition

Pod creation failures xuất hiện khi:

Pod manifest được submit nhưng không bao giờ được created (không xuất hiện trong kubectl get pods)
Pod ngay lập tức bị rejected với error message trong events
Pod có status khác pending (ví dụ: ImageInspectError, Failed, Unknown)
Pod stuck trong Init:0/N hoặc PodInitializing state vô thời hạn

Why This Matters

Pod creation failures là blocking issues — nếu Pod không được tạo, workload không thể chạy. Khác với Pending pods (có thể chờ resources), Pod creation failures thường chỉ ra configuration errors hoặc infrastructure problems cần fix ngay lập tức.

Information Gathering — Quick Diagnostics

Step 1: Check Pod Status & Events

bash

# Xem danh sách Pod, tìm những cái không ở Pending/Running
kubectl get pods -A --sort-by=.status.phase

# Xem chi tiết Pod, đặc biệt phần "Events"
kubectl describe pod <pod-name> -n <namespace>

# Xem events trong namespace
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Làm gì với output:

Nếu Pod không xuất hiện, vấn đề là pre-creation (API validation, admission webhooks)
Nếu Pod xuất hiện nhưng stuck Init:0/N, vấn đề là init container hoặc CNI
Nếu events show Failed thì đọc event message để xác định nguyên nhân

Step 2: Check Namespace Resource Quota

bash

# Xem Resource Quota của namespace
kubectl describe resourcequota -n <namespace>

# Xem current resource usage
kubectl top pods -n <namespace>
kubectl top nodes

Dấu hiệu quota problem:

Event message chứa "exceeded quota"
ResourceQuota status show "used > hard"
Ngay cả simple Pod (ví dụ: nginx) cũng bị reject

Step 3: Check Admission Webhook Status

bash

# Liệt kê validating webhooks
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# Xem webhook details
kubectl describe validatingwebhookconfigurations <webhook-name>

# Check webhook logs nếu là custom webhook
kubectl logs -n <webhook-namespace> -l app=<webhook-app>

Red flags:

Webhook có failurePolicy: Fail (nếu webhook down, Pods bị reject)
Webhook selector/namespace selector quá rộng (match mọi Pod)

Step 4: Check Node & Cluster Status

bash

# Xem tất cả nodes
kubectl get nodes -o wide
kubectl top nodes

# Kiểm tra allocatable resources
kubectl describe node <node-name>

# Xem cluster autoscaler status (nếu enabled)
kubectl logs -n kube-system -l app=cluster-autoscaler | tail -100

Step 5: Check Storage Status (nếu Pod dùng PVC)

bash

# Liệt kê PersistentVolumeClaims
kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

# Liệt kê PersistentVolumes
kubectl get pv
kubectl describe pv <pv-name>

Dấu hiệu storage problem:

PVC status Pending hoặc Failed
PV status không Bound
Event show "Unable to provision volume" hoặc "FailedBinding"

Diagnostic Decision Tree

text

Pod Creation Fails
│
├─ Pod không xuất hiện trong kubectl get pods? (YES)
│  ├─ API Server error (validation failed)?
│  │  └─ Kiểm tra kubectl apply output
│  │  └─ Kiểm tra webhook validating (nếu có)
│  │  └─ Kiểm tra API server logs: kubectl logs -n kube-system kube-apiserver
│  │
│  └─ Admission webhook reject?
│     └─ Kiểm tra admission webhook logs
│     └─ Verify webhook selector/failurePolicy
│
├─ Pod xuất hiện nhưng status khác Pending/Running?
│  ├─ Status: Init:0/N hoặc PodInitializing (> 5 phút)?
│  │  ├─ CNI plugin chưa assign IP → check CNI pod logs
│  │  ├─ Init container failed → check init container logs
│  │  └─ Webhook gây issue → kiểm tra admission webhook
│  │
│  ├─ Status: Failed, ImageInspectError, hoặc lỗi khác?
│  │  └─ Đọc event message để xác định root cause
│  │
│  └─ Status: CrashLoopBackOff?
│     └─ Không phải creation failure (Pod được tạo, app lỗi)
│     └─ Skip section này, xem Container Logs
│
├─ ResourceQuota exceeded?
│  ├─ Kiểm tra quota: kubectl describe resourcequota -n [namespace]
│  ├─ Reduce pod requests hoặc increase quota
│  └─ Check system pods không consume quota
│
├─ Không đủ node resources?
│  ├─ Pod requests > available node capacity
│  ├─ Trigger cluster autoscaler nếu enable
│  ├─ Hoặc create node pool mới với size phù hợp
│  └─ Nếu CA stuck → xem Autoscaling Issues section
│
├─ Webhook quá lâu hoặc timeout?
│  ├─ Kiểm tra webhook latency trong logs
│  ├─ Tối ưu webhook logic
│  ├─ Tăng timeout nếu cần
│  └─ Xây dựng webhook resilience (add retry logic)
│
└─ Storage issue (PVC không bind)?
   ├─ Kiểm tra PVC status
   ├─ Kiểm tra PV status
   ├─ Nếu provisioning fails → check storage class
   └─ Nếu attach fails → check disk quota, zone constraints

Common Root Causes & Fixes

Root Cause 1: Resource Quota Exceeded

Dấu hiệu:

Event: Pod <name> failed to fit in any node
Event: "exceeded quota: requests.cpu"
kubectl describe resourcequota show used == hard

Nguyên nhân: GKE không thể tạo Pod vì namespace đã sử dụng hết CPU/memory quota. Điều này thường xảy ra khi:

Daemonsets, system pods consume quota trong shared namespace
Quota được set quá thấp
Previous Pods chưa bị xóa

Immediate Fix (Workaround):

bash

# Xem quota chi tiết
kubectl describe resourcequota -n <namespace>

# Option 1: Xóa Pods không cần thiết
kubectl delete pod <old-pod> -n <namespace>

# Option 2: Giảm request của Pod mới
# Edit pod spec, giảm resources.requests.cpu/memory

# Option 3: Tăng quota (tạm thời để debug)
kubectl edit resourcequota <quota-name> -n <namespace>
# Tăng hard limits

Permanent Fix:

Tính toán quota chính xác: sum of all pods (apps + daemonsets) + margin
Đặt separate ResourceQuotas cho user apps vs system components
Sử dụng LimitRange để set default requests (tránh Pods chạy không request)

Prevention:

Luôn set requests/limits cho Pods
Monitor quota usage qua Cloud Monitoring: kubernetes.io/namespace/used
Alert nếu usage > 80% quota

Root Cause 2: Insufficient Node Resources (Across All Nodes)

Dấu hiệu:

Pod requests muốn 4 CPU nhưng largest node chỉ có 2 CPU available
GKE không trigger cluster autoscaler (không thể scale up)
Pod pending vô thời hạn

Nguyên nhân: Pod resource request lớn hơn bất kỳ node nào có thể cung cấp. Cluster autoscaler cũng không thể giúp vì sẽ scale up node nhưng Pod vẫn không fit.

Immediate Fix:

bash

# Xem cluster autoscaler logs
kubectl logs -n kube-system -l app=cluster-autoscaler | grep -i "can not scale"

# Option 1: Giảm Pod requests
kubectl set resources pod <pod-name> -n <namespace> \
  --requests=cpu=2,memory=1Gi

# Option 2: Xóa Pod constraints (node selector, affinity)
kubectl edit pod <pod-name> -n <namespace>
# Xóa nodeSelector, affinity sections

Permanent Fix:

Kiểm tra machine type availability:

bash

gcloud compute machine-types list --filter="zone:us-central1-a"

Nếu muốn Pod lớn hơn 4 CPU, create node pool với N1/N2/C2 machines

Cấu hình cluster autoscaler để scale up pool đó:

bash

gcloud container node-pools update <pool-name> \
  --cluster=<cluster-name> --enable-autoscaling \
  --min-nodes=1 --max-nodes=10

Prevention:

Document machine types available trong cluster
Enforce Pod resource limits qua LimitRange admission controller
Educate developers: "Max single Pod phải <= largest node size"

Root Cause 3: Network Plugin (CNI) Failure

Dấu hiệu:

Pod stuck Init:0/N (Pod tạo nhưng init container không run)
Pod stuck PodInitializing vô thời hạn
Event: "Unable to attach or mount volumes"
Pod logs show network unreachable

Nguyên nhân: CNI plugin (GKE uses Google VPC CNI) chịu trách nhiệm assign IP, setup iptables routing. Nếu CNI pod bị crash hoặc slow, new Pods không được initialized.

Diagnostic:

bash

# Xem CNI pods (Google managed)
kubectl get pods -n kube-system | grep gke-

# Nếu GKE managed, CNI pods nằm trong gke-managed namespace
kubectl get pods -n gke-managed
kubectl describe pod <cni-pod> -n gke-managed
kubectl logs <cni-pod> -n gke-managed

# Xem daemonset status
kubectl get daemonset -n kube-system -o wide
kubectl get daemonset -n gke-managed -o wide

# Xem node allocatable IPs
kubectl describe node <node-name> | grep -A5 "Allocatable"
# Nếu Pod IPs hampir habis (pods: 0/110) → Pod không fit

Immediate Fix:

bash

# Restart CNI pod trên node cụ thể
kubectl rollout restart daemonset/<cni-daemonset> -n gke-managed

# Hoặc nếu system-wide issue, restart node
gcloud compute instances reset <instance-name> --zone=<zone>

# Chờ node restart, CNI pods redeploy
kubectl wait --for=condition=Ready node/<node-name> --timeout=300s

Permanent Fix:

Monitor CNI pod health:

bash

kubectl top pod -n gke-managed  # Check CPU/memory

Nếu CNI pod OOMKilled → node sở hữu terlalu nhiều IP allocations
Gunakan larger node machine type hoặc reduce pod density

Prevention:

Monitoring rule: Alert nếu CNI pod CrashLoop
Cluster autoscaler phải scale up nếu tất cả nodes Pod IPs hampir habis
Graceful node draining để maintenance (jangan hard shutdown)

Root Cause 4: Admission Webhook Rejection

Dấu hiệu:

Pod API request rejected trước creation
Error: "admission webhook denied the request"
kubectl apply output show error
Event không muncul (Pod không pernah terbuat)

Nguyên nhân: ValidatingAdmissionWebhook hoặc MutatingAdmissionWebhook (custom hoặc GCP-managed) reject Pod spec.

Diagnostic:

bash

# View webhook mà installed
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# View webhook config
kubectl describe validatingwebhookconfigurations <webhook-name>

# Webhook failure policy?
kubectl get validatingwebhookconfigurations -o yaml | grep failurePolicy

# Test webhook manually (dry-run)
kubectl apply -f <pod-yaml> --dry-run=server --v=10
# Check output để webhook-related logs

Immediate Fix:

bash

# Option 1: Nonton webhook pod logs
kubectl logs -n <webhook-namespace> -l app=<webhook-app> -f

# Option 2: Disable webhook temporarily (nếu failurePolicy=Fail)
kubectl delete validatingwebhookconfigurations <webhook-name>
# (HATI-HATI: có thể menghilangkan security enforcement)

# Option 3: Update webhook selector để exclude namespace
kubectl edit validatingwebhookconfigurations <webhook-name>
# Add: namespaceSelector: matchExpressions: [{key: "skip-webhook", operator: "In", values: ["true"]}]
# Label namespace: kubectl label namespace <ns> skip-webhook=true

Permanent Fix:

Audit webhook rules: cek apakah selector terlalu broad
Tambah error handling di webhook logic
Set failurePolicy: Ignore nếu webhook non-critical (nhưng kurang ideal để security)
Implement webhook resilience: timeout, retry logic

Prevention:

Test webhook với various Pod specs trước deploy đến production
Implement webhook logging để debug
Monitoring rule: Alert nếu webhook response time > threshold
Webhook phải sở hữu SLA commitment (99.9% availability)

Root Cause 5: Persistent Volume Provisioning Failure

Dấu hiệu:

Pod terbuat nhưng stuck Pending
Event: "FailedScheduling: 0/N nodes are available"
PVC status Pending hoặc Failed
Pod event: "Unable to attach volume"

Nguyên nhân: Pod sử dụng PVC, nhưng PVC không bind đến PV. Kubernetes không akan schedule Pod đến PVC bound.

Diagnostic:

bash

# View PVC status
kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

# View PV status
kubectl get pv
kubectl describe pv <pv-name>

# Cek storage class
kubectl get storageclass
kubectl describe storageclass <sc-name>

# View storage provisioner logs
kubectl logs -n kube-system -l app=gce-pd-csi-driver

Common Issues:

Issue	Indicator	Fix
Storage class không exists	PVC event: "storageclass does not exist"	kubectl get storageclass, create hoặc use existing
PV quota exceeded	Cloud error: "quota exceeded"	Increase PD quota via Cloud Console
Zone mismatch	PVC pending, PV unbound	Node và PV phải same zone
Disk attach limit	"Max of 128 disks per instance"	Use Local SSD hoặc decrease attachments

Immediate Fix:

bash

# Nếu zone mismatch
kubectl describe node <node> | grep topology.kubernetes.io/zone
kubectl get pv -o wide  # View zone

# Nếu zone không match, create PV di correct zone
# Edit PVC spec, hoặc delete and recreate với correct zone selector

# Nếu quota issue
# Increase quota di GCP Console → Quotas & System Limits

Permanent Fix:

Default storage class phải tersedia
Educate developers về zone constraints để PVs
Gunakan volume expansion để grow storage tanpa recreate
Implement PVC monitoring để alert unbound PVCs

Prevention & Monitoring

Enable Audit Logging

bash

# Pastikan GKE logging enabled
gcloud container clusters update <cluster-name> \
  --enable-cloud-logging \
  --logging=SYSTEM,WORKLOAD,API_SERVER

Monitor Key Metrics

bash

# Setup alert để Pod creation failure rate
# Cloud Monitoring → Create Alert Policy
# Metric: kubernetes.io/pod/error_count
# Condition: rate > 0.1 per minute (adjust threshold)

# Monitor PVC binding time
# Metric: custom metric để PVC provisioning latency

Preventive Configuration

yaml

# Recommended: DefaultRuns để all namespaces
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
spec:
  hard:
    requests.cpu: "100"
    requests.memory: "200Gi"
    pods: "200"
  scopeSelector:
    matchExpressions:
    - operator: NotIn
      scopeName: PriorityClass
      values: ["system"]
---
# LimitRange để ensure reasonable defaults
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
  - max:
      cpu: "4"
      memory: "8Gi"
    min:
      cpu: "100m"
      memory: "128Mi"
    default:
      cpu: "500m"
      memory: "512Mi"
    type: Container

Escalation Criteria

Escalate to Platform Team / Google Support if:

CNI plugin repeatedly crashes → suggest GKE upgrade hoặc investigate node-level issues
Multiple webhooks rejecting Pods → webhook misconfiguration, need architecture review
Quota constantly exceeded → need cluster expansion planning
Storage provisioning failing persistently → might be GCP quota issue hoặc infrastructure problem

Information to Include:

bash

# Gather diagnostics bundle
gcloud container clusters describe <cluster-name> --zone <zone>
kubectl cluster-info dump --output-directory=./dump
kubectl logs -n kube-system --tail=1000 > kube-system-logs.txt
kubectl logs -n gke-managed --tail=1000 > gke-managed-logs.txt

# Include trong support ticket

Scheduling Failures: Resolving Pending Pods — Nếu Pod created nhưng stuck Pending
Storage Issues: Volume Attachment Failures — PVC binding deep dive
Control Plane Issues: API Server & etcd Health — Nếu API server hoặc webhook lambat

Pod Creation Failures: Troubleshooting Checklist ​

Symptoms Recognition ​

Why This Matters ​

Information Gathering — Quick Diagnostics ​

Step 1: Check Pod Status & Events ​

Step 2: Check Namespace Resource Quota ​

Step 3: Check Admission Webhook Status ​

Step 4: Check Node & Cluster Status ​

Step 5: Check Storage Status (nếu Pod dùng PVC) ​

Diagnostic Decision Tree ​

Common Root Causes & Fixes ​

Root Cause 1: Resource Quota Exceeded ​

Root Cause 2: Insufficient Node Resources (Across All Nodes) ​

Root Cause 3: Network Plugin (CNI) Failure ​

Root Cause 4: Admission Webhook Rejection ​

Root Cause 5: Persistent Volume Provisioning Failure ​

Prevention & Monitoring ​

Enable Audit Logging ​

Monitor Key Metrics ​

Preventive Configuration ​

Escalation Criteria ​

Related Sections ​

References ​

Pod Creation Failures: Troubleshooting Checklist

Symptoms Recognition

Why This Matters

Information Gathering — Quick Diagnostics

Step 1: Check Pod Status & Events

Step 2: Check Namespace Resource Quota

Step 3: Check Admission Webhook Status

Step 4: Check Node & Cluster Status

Step 5: Check Storage Status (nếu Pod dùng PVC)

Diagnostic Decision Tree

Common Root Causes & Fixes

Root Cause 1: Resource Quota Exceeded

Root Cause 2: Insufficient Node Resources (Across All Nodes)

Root Cause 3: Network Plugin (CNI) Failure

Root Cause 4: Admission Webhook Rejection

Root Cause 5: Persistent Volume Provisioning Failure

Prevention & Monitoring

Enable Audit Logging

Monitor Key Metrics

Preventive Configuration

Escalation Criteria

Related Sections

References