My Awesome Project

Appearance

IP Planning for Large Scale: CIDR Sizing, Pod Density, Exhaustion ​

IP Addressing in GKE ​

GKE clusters use VPC secondary CIDR ranges for Pod networking (in addition to node subnet).

Architecture:

VPC primary CIDR: 10.0.0.0/8 (nodes)
VPC secondary CIDR: 10.4.0.0/12 (Pods)
  • Nodes: allocated from primary CIDR
  • Pods: allocated from secondary CIDR
  • Services: cluster-internal (kube-proxy/eBPF routing, separate from VPC)

Key constraint: VPC-level, not cluster-level. Multiple clusters in same VPC must not overlap secondary CIDRs.

Internal Model: IP Allocation ​

Node IP Allocation ​

Each node gets:

  • 1 primary IP from node subnet (e.g., 10.0.1.5)
  • Multiple secondary IPs from secondary CIDR range (Pod IPs)

Secondary IPs per node: Default /24 = 256 IPs per node (minus gateway, broadcast) = ~250 usable.

Allocation method:

  • GKE allocates contiguous /24 subnet per node
  • 1000 nodes × /24 = 1000 /24s = /14 aggregate

Config:

bash
gcloud container clusters create mycluster \
  --primary-ipv4-cidr=10.0.0.0/17 \
  --cluster-secondary-ipv4-cidr=10.4.0.0/12 \
  --services-ipv4-cidr=10.128.0.0/12

This allocates:

  • Primary: 10.0.0.0/17 (nodes) = 32K IPs
  • Secondary: 10.4.0.0/12 (Pods) = 1M IPs
  • Services: 10.128.0.0/12 (Services, internal only) = 1M IPs

Pod IP Assignment ​

kubelet assigns Pod IPs from node's secondary CIDR range.

Example:

Node gke-node-1 gets /24: 10.4.0.0/24
Pod scheduling on gke-node-1:
  - Pod 1: 10.4.0.2
  - Pod 2: 10.4.0.3
  - ...
  - Pod N: 10.4.0.254

Limit: 256 IPs per /24. If Pod density = 100 Pods/node, need /23 (512 IPs) or larger.

Sizing CIDR for 1000+ Node Cluster ​

Calculation ​

Inputs:

  • Nodes: 1000
  • Pod density target: 60 Pods/node
  • Total Pods: 60,000

IP needed per node:

  • System Pods (kube-proxy, metrics-server, etc): ~10
  • User Pods: 60
  • Reserved (buffer): 10%
  • Total per node: ~80 IPs = /25 (128 IPs)

Total secondary CIDR:

  • 1000 nodes × /25 = 1000 /25s = /15 aggregate
  • 1 /15 = 32K IPs
  • Recommended: allocate 2× = /14 (64K IPs) for future growth

Example allocation:

--cluster-secondary-ipv4-cidr=10.4.0.0/14  # 64K IPs for Pods

Overflow Scenarios ​

Scenario 1: Pod density exceeds estimate

Planned: 60 Pods/node Actual: 100 Pods/node (workload surprise)

Result: Node needs more secondary IPs, can't allocate → kubelet can't assign IPs to new Pods → Pods stuck in "Pending" state.

Diagnosis:

bash
kubectl describe node gke-node-1 | grep "Allocatable\|Pod\|IP"

Fix:

  • Short-term: Delete low-priority Pods to free IPs
  • Medium-term: Add more nodes (spread workload)
  • Long-term: Re-allocate secondary CIDR (cluster restart required, expensive)

Scenario 2: Insufficient subnet per node

Default /24 per node. Cluster at Pod density = 100 Pods/node.

  • /24 = 256 IPs - system = ~240 user Pods possible
  • At 100 Pods/node, no buffer

If Pod creation rate high, node can't allocate new IPs → lag.

Fix:

  • Increase subnet size per node: GKE configuration --secondary-ipv4-range-size (newer GKE versions)
  • Or: reduce Pod density target

Services vs Pods IP Allocation ​

Important distinction:

  • Pod IPs: Allocated from secondary CIDR (10.4.0.0/14), routable within cluster
  • Service IPs: Allocated from cluster service CIDR (10.128.0.0/12), NOT routable in VPC (internal only)

Service routing:

Client Pod → Service IP (10.128.1.5:80)
  ↓ (kube-proxy or eBPF DNAT)
Pod IP (10.4.1.3:80)

Service IP is virtual (never assigned to interface), used only for routing rule lookup.

Limit: Max Services = Service CIDR size / average Service IPs used. Default /12 = 1M services (huge, not practical limit).

Advanced: Multi-Cluster IP Planning ​

If running 10 clusters in same VPC:

Cluster 1 secondary: 10.4.0.0/15 (32K IPs)
Cluster 2 secondary: 10.6.0.0/15
Cluster 3 secondary: 10.8.0.0/15
...

Error to avoid: Two clusters with overlapping secondary CIDR → networking collision, traffic misrouted.

Tool:

bash
gcloud compute networks list  # view all secondary ranges

Network Policy Impact on IP Planning ​

With GKE Dataplane V2 (eBPF), network policy enforcement has eBPF map limits:

  • Max 260K endpoints across all services

But this is endpoint count, not IP count. No direct impact on secondary CIDR sizing.

Expansion Strategy: When Secondary CIDR Exhausted ​

Scenario: Cluster started with 10.4.0.0/14 (64K IPs), now at 80K Pods, exhausted.

Options:

  1. VPC expansion: Allocate additional secondary CIDR range

    • GKE supports multiple secondary ranges
    • Can add 10.6.0.0/14 without restarting cluster
    • Nodes created after will use new range
    • Old nodes keep original range
    • Risk: Network policy, firewall rules might need updates

  2. Cluster recreation: Create new cluster with larger secondary CIDR

    • Cost: Downtime, workload migration
    • Only if add secondary range not option

  3. IP space compact: Use IPv6 (different CIDR space)

    • GKE supports dual-stack (IPv4 + IPv6)
    • IPv6 has massive address space (2^128)
    • Complexity: requires IPv6 infrastructure planning

Real-World Scenario: IP Exhaustion During Autoscale ​

Case: Cluster autoscaler scales from 100 to 1000 nodes in 5 minutes.

Timeline:

  1. New nodes added (gke-node-101, gke-node-102, ...)
  2. GKE allocates secondary IP range to each node (/24 per node)
  3. But secondary CIDR size = 10.4.0.0/14 = 64K IPs
  4. 1000 × /24 = /14 = 64K exactly (no buffer)
  5. gke-node-993 onward: can't allocate /24 (exhausted)
  6. Nodes stuck in "NotReady" state (no IPs for kubelet)
  7. Autoscaler stops scaling

Resolution:

  • Pre-allocate larger secondary CIDR (10.4.0.0/13 = 128K)
  • Or: configure per-node CIDR size = /25 instead of /24 (squeeze 2 nodes per /24)

References ​