Scheduling Failures: Resolving Pending Pods

Symptoms Recognition

Pods ở status Pending khi:

Pod status = Pending cho > 5 phút
Pod không bao giờ move tới Running
kubectl get pods show pod persistent pending
Events show Unschedulable hoặc FailedScheduling

Điều này khác từ Pod creation failures — Pod được tạo nhưng scheduler không thể tìm node phù hợp để run nó.

Why This Matters

Pending Pods tượng trưng cluster không có capacity (CPU/memory/storage) hoặc constraints conflict (node selector, affinity). Khác từ creation failures (configuration errors), pending pods là resource management problem — cần scale up hoặc rebalance workloads.

Information Gathering — Quick Diagnostics

Step 1: Describe Pod & Examine Events

bash

# Xem pod status
kubectl describe pod <pod-name> -n <namespace>

# Xem pod events chi tiết
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name> --sort-by='.lastTimestamp'

Đọc gì từ output:

Event message chứa "Unschedulable" → scheduling blocked
Message chứa "insufficient" → resource không đủ
Message chứa "node(s) had taint" → affinity conflict
Message chứa "PersistentVolumeClaim not bound" → storage issue

Step 2: Check Node Resources Across Cluster

bash

# Xem tất cả nodes với available resources
kubectl top nodes
kubectl get nodes --sort-by='.status.capacity.memory'

# Xem detailed resource breakdown per node
for node in $(kubectl get nodes -o name); do
  echo "=== $node ==="
  kubectl describe $node | grep -A5 "Allocated resources"
done

# Kiểm hairak actual running pods trên node
kubectl get pods --all-namespaces --field-selector spec.nodeName=<node-name>

Interpretasi:

Nếu tất cả nodes show "Allocated: 90% CPU", then pool is saturated
Nếu chỉ một node sở hữu space, pod stuck (node selector block?)
Nếu no nodes show available resources, trigger cluster autoscaler

Step 3: Check Cluster Autoscaler Status

bash

# View CA deployment
kubectl get deployment -n kube-system cluster-autoscaler -o wide

# View CA logs (terakhir 50 lines)
kubectl logs -n kube-system deployment/cluster-autoscaler -c cluster-autoscaler --tail=50

# Cari error messages
kubectl logs -n kube-system deployment/cluster-autoscaler -c cluster-autoscaler | grep -i "error\|failed\|scale"

# Cek CA events
kubectl get events -n kube-system --field-selector reason=ScaleUp,ScaleDown,FailedScaleUp --sort-by='.lastTimestamp'

Daftar để dipahami:

"Scale up succeeded" = CA provision node mới ✓
Scale up failed: <reason> = CA blocked (quota, constraint, size mismatch)
Can not scale: <reason> = CA không có thể fix situation (pod terlalu lớn)

Step 4: Analyze Pod Constraints

bash

# View pod spec constraints
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A10 "nodeSelector\|affinity\|tolerations"

# View node labels để matching
kubectl get nodes --show-labels

Cek để:

nodeSelector — exact label match required?
affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution — hard constraint?
affinity.podAffinity — inter-pod constraint?
tolerations — pod có thể tolerate node taint?

Diagnostic Decision Tree

text

Pod Pending
│
├─ Unschedulable event present?
│  ├─ Message: "insufficient (cpu|memory|ephemeral-storage)"?
│  │  ├─ All nodes saturated?
│  │  │  ├─ YES: Cluster autoscaler enable? CA logs show error?
│  │  │  │  └─ Nope (CA disabled hoặc stuck) → immediate scale node pool
│  │  │  │
│  │  │  └─ NO: Some nodes punya space?
│  │  │     ├─ Pod nodeSelector/affinity block fit?
│  │  │     │  └─ Check node labels, adjust selector
│  │  │     │
│  │  │     └─ Pod resource request terlalu besar?
│  │  │        └─ Reduce request hoặc create node pool với machine type lebih besar
│  │  │
│  │  └─ Message: "node(s) had taint but not tolerations"?
│  │     └─ Nonton pod tolerations, add nếu diperlukan
│  │
│  └─ Message: "PersistentVolumeClaim is not bound"?
│     └─ Storage issue, xem section Storage Issues
│
├─ No events hoặc events generic?
│  ├─ Pod resource request = 0 (no requests)?
│  │  └─ Add resource requests (scheduler cần info ini)
│  │
│  └─ Pod sempurna nhưng stuck 5+ menit?
│     └─ Scheduler might be hung, check control plane issues section
│
└─ CA scaling up nodes?
   └─ Chờ nodes ready (5-10 menit), pod akan schedule
      └─ Nếu CA stuck, see Autoscaling Issues section

Common Root Causes & Fixes

Root Cause 1: Insufficient CPU/Memory Across All Nodes

Dấu hiệu:

Event: "0/N nodes are available: N Insufficient cpu"
kubectl top nodes show all nodes 80-90%+
Pod requests = 4 CPU, largest available = 1.5 CPU

Nguyên nhân: Cluster không punya aggregate capacity để Pod. Nguyên nhân umum:

Daemonsets consume resources per-node
Previous workloads không di-cleanup
Cluster undersized để actual workload

Immediate Fix:

bash

# Option 1: Trigger Cluster Autoscaler (nếu enable)
# CA phải automatically scale up trong 1-2 menit
# Verify:
kubectl logs -n kube-system deployment/cluster-autoscaler | grep -i "scale up"

# Option 2: Manual scale node pool
POOL_NAME="default-pool"
CURRENT_SIZE=$(gcloud container node-pools describe $POOL_NAME \
  --cluster=<cluster> --zone=<zone> | grep "num_nodes:" | awk '{print $2}')
NEW_SIZE=$((CURRENT_SIZE + 2))

gcloud container node-pools update $POOL_NAME \
  --cluster=<cluster> --zone=<zone> \
  --num-nodes=$NEW_SIZE

# Chờ nodes ready
kubectl wait --for=condition=Ready node/<new-node> --timeout=300s

# Option 3: Reduce Pod resource requests
kubectl patch deployment <dep-name> -n <ns> -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"requests":{"cpu":"1","memory":"1Gi"}}}]}}}}'

Permanent Fix:

Calculate cluster capacity properly:
- Sum all pod requests (apps + daemonsets)
- Add 20% headroom
- Formula: (app_requests + ds_requests) * 1.2 = target_cluster_capacity

Enable cluster autoscaler với proper settings:

bash

gcloud container node-pools update <pool> \
  --cluster=<cluster> \
  --enable-autoscaling --min-nodes=1 --max-nodes=20

Setup monitoring để capacity:

bash

# Alert nếu available capacity < 20%
# Cloud Monitoring → Metric: kubernetes.io/node/available_cpu

Prevention:

Enforce Pod resource requests (LimitRange admission controller)
Continuous capacity monitoring
Regular capacity planning reviews

Root Cause 2: Cluster Autoscaler Cannot Scale Up

Dấu hiệu:

CA logs show "Can not scale up node pool"
Error: "quota exceeded" hoặc "zone capacity reached"
Pod stuck pending meski CA enabled

Nguyên nhân: Cluster autoscaler mau scale up nhưng blocked:

GCP quota exhausted (contoh: max disks per project)
Zone capacity limit reached
Node pool size at maximum
Machine type không available di zone
PreemptibleVM quota limit

Diagnostic:

bash

# View CA logs với detail
kubectl logs -n kube-system deployment/cluster-autoscaler -c cluster-autoscaler -f

# Filter để error
kubectl logs -n kube-system deployment/cluster-autoscaler | \
  grep -E "FailedScaleUp|Can not scale|quota|reached"

# Cek node pool config
gcloud container node-pools describe <pool-name> \
  --cluster=<cluster> --zone=<zone>

# View max-nodes setting
# Check nếu current = max

Immediate Fix:

bash

# Option 1: Increase max-nodes limit
gcloud container node-pools update <pool> \
  --cluster=<cluster> --zone=<zone> \
  --max-nodes=50

# Option 2: Create new node pool (nếu current pool maxed out)
gcloud container node-pools create <new-pool-name> \
  --cluster=<cluster> --zone=<zone> \
  --num-nodes=1 --enable-autoscaling --max-nodes=20 \
  --machine-type=n1-standard-4

# Option 3: Increase GCP quota
# Cloud Console → Quotas and System Limits
# Search để: "In-use Persistent Disks per region", "GPU quota", dll
# Increase quota request

# Option 4: Move workload to different zone
# Edit node pool → enable autoscaling để zone lain

Permanent Fix:

Understand GCP quota limits:

bash

gcloud compute project-info describe --project=<project> | \
  grep -A2 "QUOTA_METRIC\|quota"

Plan capacity by zone (không tất cả zones punya same capacity)
Implement multi-zone node pool configuration
Pre-request quota increase berdasarkan projected growth

Prevention:

Monitoring CA logs continuously
Alert nếu CA logs show "FailedScaleUp"
Periodic audit GCP quota usage

Root Cause 3: Pod Affinity/NodeSelector Conflict

Dấu hiệu:

Event: "node(s) didn't match Pod's node affinity/selector"
Event: "node(s) had taint but not tolerations"
Pod requests fit aber tetap pending

Nguyên nhân: Pod sở hữu scheduling constraint mà không có thể dipenuhi:

nodeSelector: disk=ssd — nhưng không có node với label đó
affinity.nodeAffinity.requiredDuringScheduling — hard constraint không match
Node sở hữu taint, Pod không punya matching toleration

Diagnostic:

bash

# View pod constraints
kubectl get pod <pod-name> -n <namespace> -o yaml | \
  sed -n '/nodeSelector/,/^$/p; /affinity/,/^$/p; /tolerations/,/^$/p'

# View node labels
kubectl get nodes --show-labels

# View node taints
kubectl describe nodes | grep Taints

# Match pod constraints đến nodes
kubectl get nodes -L <your-label-key>

Immediate Fix:

bash

# Option 1: Add label to node (temporary debug)
kubectl label node <node-name> disk=ssd

# Option 2: Remove constraint từ pod
kubectl edit deployment <dep-name> -n <namespace>
# Hapus nodeSelector hoặc affinity section

# Option 3: Add toleration đến pod
kubectl patch pod <pod-name> -n <namespace> --type=json \
  -p='[{"op": "add", "path": "/spec/tolerations", "value":[{"key":"<taint-key>","operator":"Equal","value":"<taint-value>","effect":"NoSchedule"}]}]'

Permanent Fix:

Understand your node pool labeling strategy:

bash

# Expected: node pool = machine type
kubectl get nodes -o wide
# Verify node-role.kubernetes.io/master, etc.

Document pod-to-node mapping:
- Which pods need GPUs? → label accelerator=nvidia
- Which pods need high memory? → label memory=high
- Create node pools accordingly

Use topology spread constraints instead of hard affinity:

yaml

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: DoNotSchedule

Prevention:

Code review: verify affinity constraints trong pod specs
Monitoring: alert nếu pods stuck pending due to affinity

Root Cause 4: PersistentVolumeClaim Not Bound

Dấu hiệu:

Pod event: PersistentVolumeClaim <name> is not bound
Pod pending, cannot schedule
kubectl get pvc show status Pending

Nguyên nhân: Pod sử dụng PVC nhưng PVC không bind đến PV. Scheduler không akan assign pod đến node đến storage ready.

Diagnostic:

bash

# View PVC status
kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

# View PV status
kubectl get pv
kubectl describe pv <pv-name>

# View storage class
kubectl get storageclass
kubectl describe storageclass <sc-name>

# View storage provisioner logs
kubectl logs -n kube-system -l app=gce-pd-csi-driver

Common Issues & Fixes:

Issue	Fix
Storage class not found	Create or use existing: `kubectl get storageclass`
PV provisioning timeout	Check CSI driver logs, increase timeout
Zone mismatch (node zone ≠ PV zone)	Delete PVC, recreate với correct zone
Disk quota exceeded	Increase quota trong Cloud Console

Immediate Fix:

bash

# Option 1: Nếu storage class issue
kubectl get storageclass  # List available
# Update PVC spec với existing class

# Option 2: Nếu zone mismatch
kubectl get pv <pv-name> -o yaml | grep topology
# Versus node zone:
kubectl get node <node-name> -o yaml | grep zone

# Nếu không match, recreate PVC di correct zone

# Option 3: Nếu provisioning stuck
# Force delete stuck PV (HATI-HATI: data loss possible)
kubectl delete pvc <pvc-name> -n <namespace> --force --grace-period=0

Permanent Fix:

See Storage Issues section

Root Cause 5: DaemonSet Blocking Pod Scheduling

Dấu hiệu:

Pod pending meski nodes punya resources
Daemonset pods (CNI, monitoring agents) consume significant CPU/memory
Effective available resources = (node_capacity - daemonsets) < pod_request

Nguyên nhân: DaemonSets run on mỗi node. Nếu daemonset CPU/memory request tinggi, little space for user Pods.

Diagnostic:

bash

# View DaemonSet resource requests
kubectl get daemonsets --all-namespaces -o json | \
  jq '.items[] | {name: .metadata.name, namespace: .metadata.namespace, containers: .spec.template.spec.containers[].resources}'

# Hitung total daemonset resource per node
# Contoh: 10 daemonsets × 500m CPU = 5 CPU per node

Immediate Fix:

bash

# Option 1: Reduce daemonset resource requests
kubectl patch daemonset <ds-name> -n <namespace> \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"requests":{"cpu":"100m","memory":"128Mi"}}}]}}}}'

# Option 2: Node taints để exclude certain node pools từ daemonset
# Create new node pool với taint:
gcloud container node-pools create <pool> \
  --cluster=<cluster> --zone=<zone> \
  --node-taints=daemonset=false:NoSchedule

# Update daemonset toleration
kubectl patch daemonset <ds-name> -n <namespace> -p \
  '{"spec":{"template":{"spec":{"tolerations":[{"key":"daemonset","operator":"Equal","value":"false","effect":"NoSchedule"}]}}}}'

Permanent Fix:

Monitor daemonset resource consumption
Separate node pool để daemonset-only workloads
Educate developers: account for daemonsets khi request node size

Prevention & Monitoring

Setup Proactive Monitoring

bash

# Cloud Monitoring → Custom Alert Policies

# Alert 1: Pods stuck Pending > 10 minutes
# Metric: kubernetes.io/pod/status_by_phase
# Condition: value(pending) > threshold for > 10m

# Alert 2: Node capacity < 20%
# Metric: kubernetes.io/node/available_cpu
# Condition: value < 0.2 * node.capacity

# Alert 3: Cluster Autoscaler FailedScaleUp events
# Metric: custom metric based on CA logs

Recommended Configuration

yaml

# LimitRange để enforce reasonable requests
apiVersion: v1
kind: LimitRange
metadata:
  name: pod-limits
spec:
  limits:
  - max:
      cpu: "4"
      memory: "8Gi"
    min:
      cpu: "100m"
      memory: "128Mi"
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "250m"
      memory: "256Mi"
    type: Container
---
# PodDisruptionBudget để ensure availability
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: myapp

Escalation Criteria

Escalate if:

Pod pending > 30 menit meskipun CA enabled
CA repeatedly fails to scale up (resource quota issue)
Node resources show anomalies (overcommit, leak)
Pattern: pending pods increasing over time (infrastructure issue)

Pod Creation Failures — Nếu Pod không terbuat
Storage Issues — Nếu PVC không bind
Control Plane Issues — Nếu scheduler lambat/hung
Autoscaling Issues — HPA/CA deep dive

Scheduling Failures: Resolving Pending Pods ​

Symptoms Recognition ​

Why This Matters ​

Information Gathering — Quick Diagnostics ​

Step 1: Describe Pod & Examine Events ​

Step 2: Check Node Resources Across Cluster ​

Step 3: Check Cluster Autoscaler Status ​

Step 4: Analyze Pod Constraints ​

Diagnostic Decision Tree ​

Common Root Causes & Fixes ​

Root Cause 1: Insufficient CPU/Memory Across All Nodes ​

Root Cause 2: Cluster Autoscaler Cannot Scale Up ​

Root Cause 3: Pod Affinity/NodeSelector Conflict ​

Root Cause 4: PersistentVolumeClaim Not Bound ​

Root Cause 5: DaemonSet Blocking Pod Scheduling ​

Prevention & Monitoring ​

Setup Proactive Monitoring ​

Recommended Configuration ​

Escalation Criteria ​

Related Sections ​

References ​

Scheduling Failures: Resolving Pending Pods

Symptoms Recognition

Why This Matters

Information Gathering — Quick Diagnostics

Step 1: Describe Pod & Examine Events

Step 2: Check Node Resources Across Cluster

Step 3: Check Cluster Autoscaler Status

Step 4: Analyze Pod Constraints

Diagnostic Decision Tree

Common Root Causes & Fixes

Root Cause 1: Insufficient CPU/Memory Across All Nodes

Root Cause 2: Cluster Autoscaler Cannot Scale Up

Root Cause 3: Pod Affinity/NodeSelector Conflict

Root Cause 4: PersistentVolumeClaim Not Bound

Root Cause 5: DaemonSet Blocking Pod Scheduling

Prevention & Monitoring

Setup Proactive Monitoring

Recommended Configuration

Escalation Criteria

Related Sections

References