Autoscaling Issues: HPA/CA Troubleshooting

Symptoms Recognition

Autoscaling issues xuất hiện khi:

Horizontal Pod Autoscaler (HPA) không scale pods
Cluster Autoscaler (CA) không provision nodes
Pods stuck Pending meski HPA enabled
Metrics unavailable (HPA show "unknown" status)
Scaling stuck (replicas stuck, không change)
Scaling slow (delay 2+ minutes trước trigger)

Why This Matters

Autoscaling là critical để production efficiency:

HPA scales pods theo traffic → ensures responsiveness
CA scales nodes → ensures pod placement
Failure → either pod starvation (overload) hoặc resource waste (underutilization)

Understanding two-tier scaling (HPA → CA) interactions là essential để reliable autoscaling.

Information Gathering — Quick Diagnostics

Step 1: Check HPA Status & Metrics

bash

# List HPAs
kubectl get hpa --all-namespaces

# Detailed HPA status
kubectl describe hpa <hpa-name> -n <namespace>
# Look at "Current/Desired Replicas", "Conditions", "Events"

# Check HPA metrics (what's being monitored)
kubectl get hpa <hpa-name> -n <namespace> -o yaml | \
  sed -n '/metrics:/,/^[a-zA-Z]/p'

# View recent HPA events
kubectl get events -n <namespace> --field-selector involvedObject.name=<hpa-name> \
  --sort-by='.lastTimestamp' | tail -20

Interpretasi:

Current: <X> / Desired: <Y> = scaling decision
"Conditions: Unknown" = metrics unavailable
"Last Scale Time" = khi terakhir HPA adjust replicas

Step 2: Verify Metrics Availability

bash

# Check Metrics Server (để CPU/memory metrics)
kubectl get deployment metrics-server -n kube-system
kubectl get pod -n kube-system -l k8s-app=metrics-server

# Check if pods have metrics
kubectl top pod -n <namespace>
# If pods show <unknown>, metrics not collected yet (wait ~1 min)

# For custom metrics, check adapter
kubectl get deployment -n custom-metrics
# If using Prometheus Adapter

# Check Cloud Monitoring (native custom metrics)
gcloud monitoring metrics-descriptors list | grep kubernetes

Daftar để dipahami:

Metrics Server: CPU/memory (built-in)
Custom Metrics: application metrics via Prometheus/Cloud Monitoring
External Metrics: GCP services metrics

Step 3: Check Cluster Autoscaler Status & Logs

bash

# View CA deployment
kubectl get deployment -n kube-system cluster-autoscaler
kubectl get pod -n kube-system -l app=cluster-autoscaler

# Describe CA pod
kubectl describe pod <ca-pod> -n kube-system

# CA logs (recent activity)
kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100

# Filter để scale events
kubectl logs -n kube-system -l app=cluster-autoscaler | \
  grep -i "scale\|pending\|failed"

# Check CA events in kube-system namespace
kubectl get events -n kube-system --field-selector reason=ScaleUp,reason=FailedScaleUp \
  --sort-by='.lastTimestamp' | tail -20

Step 4: Analyze Pod Resource Requests

bash

# Check pod resource requests (foundation để scaling decisions)
kubectl get pods -n <namespace> -o json | \
  jq '.items[] | {name: .metadata.name, cpu_req: .spec.containers[].resources.requests.cpu, mem_req: .spec.containers[].resources.requests.memory}'

# Check if requests are set (HPA needs this!)
# If requests empty → HPA cannot scale trên CPU/memory
kubectl top pod -n <namespace>  # Actual usage
kubectl get pods -n <namespace> -o wide  # Requests (if set)

Critical: HPA scales on requests, not actual usage. If requests = 0, HPA cannot make scaling decisions.

Step 5: Check Node Pool Capacity & Limits

bash

# List node pools
gcloud container node-pools list --cluster=<cluster> --zone=<zone>

# Check max-nodes setting
for pool in $(gcloud container node-pools list --cluster=<cluster> --zone=<zone> -q); do
  echo "=== $pool ==="
  gcloud container node-pools describe $pool \
    --cluster=<cluster> --zone=<zone> | grep -E "name|maxNodeCount|minNodeCount|numNodes"
done

# Check if any pool at max-nodes limit
gcloud container node-pools describe <pool> \
  --cluster=<cluster> --zone=<zone> | grep numNodes

Diagnostic Decision Tree

Autoscaling Issue
│
├─ HPA or CA (or both) failing?
│  ├─ HPA only → skip CA checks
│  ├─ CA only → skip HPA checks
│  └─ Both → diagnose both
│
├─ HPA Issue Path:
│  │
│  ├─ HPA status = Unknown?
│  │  ├─ Metrics unavailable → check Metrics Server
│  │  │  ├─ Pod memiliki resource requests?
│  │  │  │  ├─ NO → add requests, HPA akan work once calculated
│  │  │  │  └─ YES → wait 1-2 min để metrics appear
│  │  │  │
│  │  │  └─ Metrics Server pod CrashLoop?
│  │  │     └─ Restart hoặc investigate pod logs
│  │  │
│  │  └─ Metrics available but HPA not scaling
│  │     ├─ Current replicas < min? → scale up manually
│  │     ├─ Metric value < threshold? → condition not met
│  │     ├─ ScalingPolicy issue? → check targetAverageUtilization
│  │
│  └─ HPA scaling pods nhưng pods stay Pending
│     └─ CA issue, see CA path below
│
├─ CA Issue Path:
│  │
│  ├─ Pending pods present?
│  │  ├─ YES → CA should trigger scale-up
│  │  ├─ Check CA logs để "Can not scale" message
│  │  │  ├─ Node pool at max-nodes? → increase max
│  │  │  ├─ Quota exceeded? → increase quota
│  │  │  ├─ Zone capacity? → use different zone
│  │  │  └─ Machine type unavailable? → use alternative
│  │  │
│  │  └─ NO pods pending
│  │     └─ CA working correctly
│  │
│  └─ CA pod stuck/CrashLoop?
│     └─ Check cloud provider integration, logs
│
└─ Scaling slow?
   ├─ Metrics lag → normal (5-30 sec delay)
   ├─ CA scale-up latency → normal (2-3 min để node provision)
   └─ If > 5 min, investigate logs

Common Root Causes & Fixes

Root Cause 1: Pod Resource Requests Not Set

Dấu hiệu:

HPA status: "unknown" for CPU/memory
Event: "Failed to get cpu resource metrics"
HPA not scaling despite high actual load

Nguyên nhân: HPA scales based vào resource requests, not actual usage. Nếu pod không punya requests → HPA không có thể calculate percentage.

Diagnostic:

bash

# Check pod requests
kubectl get pod <pod-name> -n <namespace> -o yaml | \
  grep -A5 "resources:"

# If requests empty:
resources:
  limits:
    cpu: "2"
    memory: 2Gi
  # requests missing → HPA cannot work!

Immediate Fix:

bash

# Option 1: Add resource requests đến existing deployment
kubectl set resources deployment <dep> -n <namespace> \
  --requests=cpu=500m,memory=512Mi \
  --limits=cpu=2000m,memory=2Gi

# Option 2: Edit deployment YAML, add requests
kubectl edit deployment <dep> -n <namespace>
# Under spec.template.spec.containers[].resources:
#   requests:
#     cpu: 500m
#     memory: 512Mi

# Option 3: Use LimitRange để auto-set defaults
kubectl create -f - <<EOF
apiVersion: v1
kind: LimitRange
metadata:
  name: auto-requests
spec:
  limits:
  - defaultRequest:
      cpu: 250m
      memory: 256Mi
    type: Container
EOF

Permanent Fix:

Enforce resource requests (Admission Controller):

bash

# LimitRange với defaultRequest ensures all pods have requests

CI/CD validation:
- Reject deployments tanpa resource requests
Vertical Pod Autoscaler:
- Automatically tune requests berdasarkan historical usage

Root Cause 2: Metrics Server Not Running / Delayed

Dấu hiệu:

HPA show "unknown" status constantly
kubectl top pod return no data
Metrics Server pod CrashLoop/NotReady
Delay > 5 minutes trước metrics appear

Nguyên nhân: Metrics Server pod không healthy hoặc không installed.

Diagnostic:

bash

# Check Metrics Server
kubectl get deployment metrics-server -n kube-system
kubectl get pod -n kube-system -l k8s-app=metrics-server

# Check if running and ready
kubectl describe pod <metrics-server-pod> -n kube-system

# Logs
kubectl logs <metrics-server-pod> -n kube-system | tail -50

# Test metrics collection
kubectl top nodes
kubectl top pods --all-namespaces

Immediate Fix:

bash

# Option 1: Restart Metrics Server
kubectl rollout restart deployment metrics-server -n kube-system

# Option 2: Scale up replicas (if stuck at 0)
kubectl scale deployment metrics-server -n kube-system --replicas=2

# Option 3: Check if Metrics Server healthy
kubectl describe service metrics-server -n kube-system
# Should show endpoints

# Option 4: Manual pod metrics collection trigger
# Metrics usually collected every 15 seconds, wait
kubectl top pod --all-namespaces --use-protocol-buffers

Permanent Fix:

Ensure Metrics Server pod has:
- Adequate CPU/memory requests
- Node affinity (avoid NotReady nodes)
Monitor Metrics Server health:
- PodDisruptionBudget: minAvailable=1
- Alert if pod CrashLoop

Root Cause 3: Cluster Autoscaler Scale-Up Blocked

Dấu hiệu:

Pods stuck Pending
CA logs: "Can not scale: insufficient quota"
Node pool at max-nodes
Zone capacity reached

Nguyên nhân: Cluster Autoscaler wants scale up nhưng blocked:

Node pool reached max-nodes limit
GCP quota exhausted (disks, IPs, compute resources)
Zone capacity full
Machine type unavailable

Diagnostic:

bash

# Check CA logs
kubectl logs -n kube-system -l app=cluster-autoscaler | \
  grep -i "failed\|can not\|insufficient\|scale"

# Check node pool limits
for pool in $(gcloud container node-pools list --cluster=<cluster> -q); do
  echo "=== $pool ==="
  gcloud container node-pools describe $pool --cluster=<cluster> | \
    grep -E "maxNodeCount|numNodes|minNodeCount"
done

# Check GCP quotas
gcloud compute project-info describe --project=<project> | grep QUOTA

# Check zone capacity (unofficial, GCP не publish)
# Alternative: try scale-up, CA logs akan show capacity reached

Immediate Fix:

bash

# Option 1: Increase max-nodes để pool
gcloud container node-pools update <pool> --cluster=<cluster> \
  --max-nodes=50

# Option 2: Increase GCP quota
# Cloud Console → Quotas → find relevant quota (Disks, IPs, etc.)
# Click → Edit Quota → Increase

# Option 3: Create new node pool (different machine type / zone)
gcloud container node-pools create <new-pool> --cluster=<cluster> \
  --machine-type=n1-standard-4 --num-nodes=1 --enable-autoscaling \
  --max-nodes=20

# Option 4: Manual scale (temporary)
gcloud container node-pools update <pool> --cluster=<cluster> \
  --num-nodes=20

Permanent Fix:

Capacity planning:
- Calculate max replicas needed
- Plan node pool size accordingly
Multi-pool strategy:
- Different node pools dla different workload types
- Failover capacity
Quota management:
- Pre-request quota increases
- Monitoring quota usage trends

Root Cause 4: HPA Scaling Slow / Delayed

Dấu hiệu:

Load spike 10 min ago, replicas just now increasing
Metrics lag causing delayed scaling
CPU util high but HPA not scaling up

Nguyên nhân: HPA scaling delay causes:

Metrics collection delay (15-30 sec)
HPA evaluation interval (default 15 sec)
Pod startup time (can be 1-2 min)
CA node provisioning (2-3 min)

Total pipeline: 5-10 minutes latency possible.

Diagnostic:

bash

# Check HPA configuration
kubectl get hpa <hpa-name> -n <namespace> -o yaml | \
  grep -i "horizontal-pod-autoscaler"

# Check reconcile period (HPA evaluation interval)
# Default: 15s, aber might be increased
kubectl get hpa <hpa-name> -n <namespace> -o json | \
  jq '.spec.behavior.scaleUpBehavior'

# Measure end-to-end latency
# Kubectl time metric show → HPA scale → pod ready time

Immediate Fix:

bash

# Option 1: Increase HPA evaluation frequency (advanced)
# Edit HPA, set behavior.scaleUpBehavior.stabilizationWindow to lower value
# Default: 5m (prevents thrashing), reduce to 1m if appropriate

# Option 2: Improve pod startup latency
# Reduce image size, optimize initialization

# Option 3: Pre-warm cluster (scaling policy)
kubectl autoscale deployment <dep> \
  --min=3 --max=20 --cpu-percent=70 \
  --horizontal-pod-autoscaler-sync-period=5s  # Aggressive (not recommended)

Permanent Fix:

Accept inherent latency:
- HPA not designed để sub-second response
- 5-10 min latency là normal
- If need faster, need different approach (predictive scaling)
Predictive scaling:
- GKE Workload Metrics feature
- Pre-scale based upon predicted traffic
SLO-based scaling:
- Scale to meet SLO, not just current metrics

Root Cause 5: ComputeClass Blocking CA Scale-Up (Autopilot)

Dấu hiệu:

Autopilot cluster
CA logs: "Can not scale: custom compute class blocking"
Pods pending despite CA enabled
Custom ComputeClass resource stuck

Nguyên nhân: Autopilot custom ComputeClass operations (create/delete) temporarily block CA scale-up.

Diagnostic:

bash

# Check if Autopilot cluster
gcloud container clusters describe <cluster> | grep autopilot

# Check custom ComputeClasses
kubectl get computeclasses

# Check CA logs
kubectl logs -n kube-system -l app=cluster-autoscaler | \
  grep -i "computeclass\|custom.*class"

# Check pending operations
kubectl get computeclasses -o yaml | grep -i pending

Immediate Fix:

bash

# Option 1: Wait để ComputeClass operation complete
# Usually <5 min

# Option 2: Scale down pending operations
# If lots of ComputeClass creating, delay some

# Option 3: Check service level
# Ensure CA not stuck behind ComputeClass creation queue

Permanent Fix:

Minimize ComputeClass churn
Create all needed ComputeClasses upfront
Don't dynamically create/delete classes during production

Prevention & Monitoring

Comprehensive Autoscaling Monitoring

bash

# Cloud Monitoring dashboards

# Dashboard 1: HPA Health
# Metrics:
#   - hpa_current_replicas
#   - hpa_desired_replicas
#   - hpa_max_replicas / hpa_min_replicas
#   - metric_current_value (CPU, memory, custom)

# Dashboard 2: CA Scaling Activity
# Metrics:
#   - ca_nodes_count
#   - ca_scale_up_events
#   - ca_failed_scale_ups
#   - ca_pending_pods_count

# Alert Rules:
#   - CA FailedScaleUp > 1 per 10 min → Alert
#   - Pod pending > 30 min → Page
#   - HPA status Unknown > 5 min → Alert
#   - Metrics lag > 60 sec → Alert

Best Practice Configuration

yaml

# Recommended HPA for production
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 3
  maxReplicas: 100
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleUpBehavior:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 100  # Double replicas every 60s
        periodSeconds: 60
    scaleDownBehavior:
      stabilizationWindowSeconds: 300  # Wait 5 min before scale down
      policies:
      - type: Percent
        value: 50  # Reduce by 50%
        periodSeconds: 60
---
# Enable Cluster Autoscaler (GKE default: enabled)
# Verify via:
# gcloud container node-pools describe <pool> | grep autoScaling

Escalation Criteria

Escalate if:

CA repeatedly FailedScaleUp (infrastructure quota issue)
HPA constantly scaling up/down (thrashing, needs tuning)
Metrics missing for extended period (Metrics Server issue)
ComputeClass operations deadlock CA

Scheduling Failures — Pods stuck pending due к CA not scaling
Control Plane Issues — High API latency affecting HPA/CA

Autoscaling Issues: HPA/CA Troubleshooting ​

Symptoms Recognition ​

Why This Matters ​

Information Gathering — Quick Diagnostics ​

Step 1: Check HPA Status & Metrics ​

Step 2: Verify Metrics Availability ​

Step 3: Check Cluster Autoscaler Status & Logs ​

Step 4: Analyze Pod Resource Requests ​

Step 5: Check Node Pool Capacity & Limits ​

Diagnostic Decision Tree ​

Common Root Causes & Fixes ​

Root Cause 1: Pod Resource Requests Not Set ​

Root Cause 2: Metrics Server Not Running / Delayed ​

Root Cause 3: Cluster Autoscaler Scale-Up Blocked ​

Root Cause 4: HPA Scaling Slow / Delayed ​

Root Cause 5: ComputeClass Blocking CA Scale-Up (Autopilot) ​

Prevention & Monitoring ​

Comprehensive Autoscaling Monitoring ​

Best Practice Configuration ​

Escalation Criteria ​

Related Sections ​

References ​

Autoscaling Issues: HPA/CA Troubleshooting

Symptoms Recognition

Why This Matters

Information Gathering — Quick Diagnostics

Step 1: Check HPA Status & Metrics

Step 2: Verify Metrics Availability

Step 3: Check Cluster Autoscaler Status & Logs

Step 4: Analyze Pod Resource Requests

Step 5: Check Node Pool Capacity & Limits

Diagnostic Decision Tree

Common Root Causes & Fixes

Root Cause 1: Pod Resource Requests Not Set

Root Cause 2: Metrics Server Not Running / Delayed

Root Cause 3: Cluster Autoscaler Scale-Up Blocked

Root Cause 4: HPA Scaling Slow / Delayed

Root Cause 5: ComputeClass Blocking CA Scale-Up (Autopilot)

Prevention & Monitoring

Comprehensive Autoscaling Monitoring

Best Practice Configuration

Escalation Criteria

Related Sections

References