Large-Scale Upgrade: Surge Sizing, Concurrency, Disruption

Upgrade Overview

GKE cluster upgrade involves:

1. Control plane upgrade (API server, etcd, controller manager)
2. Node upgrade (kubelet, container runtime, kernel patches)

Node upgrade is rolling: cordon nodes, drain Pods, upgrade, uncordon. Goal: maintain availability during.

Surge Nodes: Temporary Capacity

During node upgrade, surge nodes temporarily replace cordoned nodes to maintain capacity.

Example (1000-node cluster):

1. Initial: 1000 nodes + 0 surge
2. Cordon 100 nodes (upgrade wave 1)
3. GKE creates 100 surge nodes (temporary)
4. Now: 900 running + 100 cordoned + 100 surge = 1100 total
5. Drain 100 cordoned nodes
6. Pods reschedule to 900 + 100 surge = 1000 capacity
7. Upgrade 100 nodes
8. Uncordon upgraded nodes
9. Delete 100 surge nodes
10. Repeat for next wave

Surge sizing:

• Default: 1 surge node per 50 nodes (2%)
• For 1000 nodes: 20 surge nodes automatically

Cost impact:

• 20 surge nodes × machine cost = additional cost during upgrade
• Example: n2-standard-8 = $200/month = $6.67/day = $0.28/hour
• 20 surge × 2 hours upgrade window = $5.60 extra cost

Configuration:

gcloud container clusters update mycluster \
  --surge-upgrade-max-surge=50 \
  --surge-upgrade-max-unavailable=0

• max-surge=50: max 50 surge nodes (allows faster upgrade, higher cost)
• max-unavailable=0: 0 nodes down at any time (maximum availability)

Upgrade Concurrency

Default: 1 node pool at a time (sequential).

Faster: Multiple node pools upgrade in parallel (if cluster autoscaler can reprovision).

Example:

Parallel upgrade (recommended for 1000+ nodes):
  latency-pool (200 nodes): upgrade 10 waves × 2 minutes = 20 minutes
  batch-pool (600 nodes): upgrade 12 waves × 2 minutes = 24 minutes
  Total: max(20, 24) = 24 minutes
  (vs sequential: 20 + 24 = 44 minutes)

Configuration:

gcloud container clusters update mycluster \
  --parallel-upgrade-max-surge=50 \
  --enable-parallel-node-pool-upgrades

Pod Disruption Budgets (PDB)

PDB defines "how many Pods can be disrupted simultaneously?"

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 5      # at least 5 web Pods available during disruption
  selector:
    matchLabels:
      app: web

During node upgrade:

• Kubelet drains node (evict all Pods)
• For each Pod, check PDB
• If evicting would violate PDB: wait (don't drain Pod immediately)
• Upgrade stalls until other Pods settle

Example (1000-node cluster, web Deployment 100 replicas):

spec:
  minAvailable: 80  # need 80 Pods available always

During upgrade:

1. Drain node A (10 web Pods)
2. Would leave 90 Pods (< 80 required?) No, 90 > 80, ok
3. Wait for 10 Pods to reschedule
4. Once rescheduled on surge node: 100 Pods again
5. Proceed to drain node B

Without PDB:

• 10 Pods get kicked, immediately rescheduled
• No availability guarantee (brief 1-2 second window with 90 Pods)

Cost of conservative PDB:

minAvailable: 99  # only allow 1 Pod evicted at a time

At 100 replicas, 1000 nodes, 1 Pod per wave:

• 100 waves × 2 minutes per wave = 200 minutes (3+ hours)
• Very slow upgrade

Balanced PDB:

minAvailable: 80  # allow up to 20 concurrent evictions

At 100 replicas:

• 5 waves (20 Pods per wave) × 2 minutes = 10 minutes
• Fast, still maintains 80% availability

Upgrade Strategy: Control Plane vs Nodes

Control Plane Upgrade

Control plane (API server, etcd, scheduler) replicas (usually 3-5) upgrade rolling:

1. Drain one replica
2. Upgrade
3. Wait for replica to be ready
4. Repeat

Impact: ~10 seconds downtime per replica (load balancer reroutes)

Total: 3 replicas × 10 seconds = 30 seconds cluster-wide downtime.

Node Upgrade

Nodes upgrade in waves (concurrency tunable). Each wave:

1. Cordon 100 nodes (max-surge=50, so 50-100 at a time)
2. Create surge nodes
3. Drain nodes (respecting PDB)
4. Upgrade OS, kubelet, container runtime
5. Uncordon
6. Delete surge nodes

Total time: (node_count / concurrency) × time_per_node = (1000 / 100) × 2 = 20 minutes.

Upgrade Failure Scenarios

Scenario 1: Drain Timeout

Node drain timeout (default 5 minutes). If Pod won't evict (stuck finalizer, blocking hook):

• Drain force-kills after timeout
• Pod violently terminated
• May lose in-flight data

Mitigation:

terminationGracePeriodSeconds: 30  # let Pod shutdown gracefully

Scenario 2: PDB Violation (Stall)

Upgrade stalls if PDB prevents enough Pods from evicting.

Example:

minAvailable: 100  # all 100 Pods must stay available

Upgrade can't evict any Pod → stall forever.

Mitigation:

• Monitor PDB constraints during upgrade
• Temporarily relax PDB (increase maxUnavailable)
• Or increase replicas to exceed PDB floor

Scenario 3: Out-of-Capacity (Surge Nodes Insufficient)

Surge nodes insufficient, not all evicted Pods can reschedule.

Example:

• 100 nodes being drained (1000 Pods evicted)
• Surge nodes = 10 (capacity for 100 Pods)
• Remaining 900 Pods need to fit in 900 non-cordoned nodes
• But those 900 nodes already full (Pod density 100%)
• Deadlock: can't reschedule

Mitigation:

• Size surge adequately (max-surge=50+ for dense clusters)
• Pre-allocate buffer capacity (90% utilization max)

Real-World Scenario: Upgrade During Peak Load

Case: 1000-node cluster, peak traffic load, upgrade scheduled.

Timeline:

1. 00:00: Upgrade starts
2. 00:02: First wave (100 nodes cordoned, surge created)
3. 00:03: Pods drain, reschedule to remaining capacity
4. 00:05: Some Pods Pending (insufficient capacity, surge not ready)
5. Traffic drops as services degraded (missing Pods)
6. Alerts fire

Better approach:

• Schedule upgrade during off-peak (night)
• Or: over-provision cluster (90% utilization)
• Or: increase surge nodes (max-surge=100, temporary cost)

Upgrade Best Practices at 1000+ Nodes

1. High surge: max-surge=100 (faster upgrade, acceptable cost)
2. Conservative PDB: minAvailable=70-80% (allows upgrade, maintains availability)
3. Gradual rollout: 10% nodes per wave (manage risk)
4. Monitor: track Pod Pending during upgrade
5. Scheduled maintenance windows: communicate downtime

References

• GKE Node Upgrade Strategies
• Pod Disruption Budgets
• GKE Cluster Upgrade