Workload Distribution: Topology Spread, Affinity, Bin-Packing

Placement Strategy Impact

How Pods distributed across nodes affects:

1. Failure resilience: if one node fails, how many Pods?
2. Network performance: traffic locality, latency
3. Resource utilization: underutilized nodes, cost
4. Upgrade impact: how many Pods disrupted during node upgrade

Topology Spread Constraints (TSC)

Goal: Distribute Pods evenly across topology domains (zones, nodes, racks).

Example: Multi-Zone Spread

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 100
  template:
    spec:
      topologySpreadConstraints:
      - maxSkew: 10      # allow up to 10 Pods difference between zones
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
      containers:
      - name: web
        image: web:v1
        resources:
          requests:
            cpu: 100m
            memory: 128Mi

Behavior:

• 100 replicas across 3 zones
• Target: ~33 Pods per zone
• maxSkew=10: allow 23-43 Pods per zone (difference ≤10)
• whenUnsatisfiable=DoNotSchedule: if can't meet constraint, don't schedule

Trade-off:

• Pro: Failure-resilient (zone failure = max 43 Pods lost)
• Con: Might constrain scheduling (some nodes empty if constraint can't be met)

Hostname-Level Spread

topologySpreadConstraints:
- maxSkew: 2
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: ScheduleAnyway

Behavior: Spread across nodes, allow up to 2 Pods difference per node.

Example (100 replicas, 1000 nodes):

• Without TSC: 10 nodes fully used, 990 nodes empty (bin-packing)
• With TSC: ~50-100 nodes with 1-2 Pods each (spreading)
• Memory footprint: spread uses more nodes, less efficient

When to use:

• Latency-sensitive apps (fault-tolerance > cost)
• Apps with high per-Pod resource (can't co-locate many)

Anti-Affinity Constraints

PodAntiAffinity: prevent co-location of related Pods.

Required Anti-Affinity

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - web
        topologyKey: kubernetes.io/hostname

Behavior: Each "web" Pod must be on different node (required).

At scale (1000 nodes, 500 web Pods):

• Requires at least 500 different nodes
• If cluster has exactly 500 nodes → can place all
• If fewer nodes → unsatisfiable → some Pods unscheduled

Cost: Forces spreading (worse bin-packing).

Preferred Anti-Affinity

podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - weight: 100
    podAffinityTerm:
      labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - web
      topologyKey: kubernetes.io/hostname

Behavior: Try to spread, but allow co-location if no other option.

At scale (1000 nodes, 2000 web Pods):

• Try to spread (weight=100 → high priority)
• If no nodes available, allow 2 Pods per node
• More flexible than required

Bin-Packing Strategy

Goal: Use fewest nodes (cost optimization).

Configuration:

spec:
  affinity:
    podAntiAffinity: null  # no spreading
  priorityClassName: high-priority

Combined with scheduler priority tuning:

• Disable spreading policies
• Use LeastRequestedPriority (no spreading) or MostRequestedPriority (dense packing)

Example (1000 Pod batch job, 100 nodes):

• Spreading: 1000 nodes needed (1 Pod per node)
• Bin-packing: 100 nodes used (10 Pods per node)
• Cost difference: 10x

Trade-off:

• Pro: Cost (fewer nodes)
• Con: Single node failure = 10 Pods lost (vs 1)

When to use:

• Batch/non-critical workload
• Stateless (can handle node failures via retry)

Affinity to Specific Hardware

Example: GPU workload

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: accelerator
            operator: In
            values:
            - nvidia-a100

Behavior: Only schedule on nodes with GPU.

At scale (50 GPU nodes, 200 training jobs):

• 50 nodes available for GPU workload
• Average 4 jobs per node
• If jobs have high GPU utilization → good
• If jobs light GPU usage → over-provisioned (waste)

Scheduler Decisions Under TSC

Scenario: Deploy 500 web Pods with TSC (maxSkew=10, zone spread).

Scheduler algorithm:

1. Fetch all nodes
2. Filter by predicates (resource, affinity)
3. For each feasible node, check if adding Pod violates TSC
   • Count Pods per zone (current)
   • If adding to node = zone already at (current_count + maxSkew) → skip
4. Score remaining nodes
5. Pick best

Complexity: O(nodes) per Pod × O(topology_keys) checks = O(1000) per Pod.

For 500 Pods, = 500 * 1000 = 500K predicate evaluations. Scheduler latency ~100-500ms.

Real-World Scenario: TSC Blocking Deployment

Case: 1000-node cluster (10 zones, 100 nodes per zone). Deploying 200 Pods with:

topologySpreadConstraints:
- maxSkew: 5
  topologyKey: topology.kubernetes.io/zone

Target: 20 Pods per zone (uniform). Max allowed: 25 Pods per zone (20 + maxSkew).

Timeline:

1. Deploy 200 Pods
2. Scheduler tries even distribution
3. After scheduling ~150 Pods: 5 zones hit 25-Pod limit
4. Remaining 50 Pods can't place anywhere (all zones at capacity)
5. Pods stuck Pending

Resolution:

• Increase maxSkew (e.g., 10)
• Or increase replicas (so 20+maxSkew allows more)
• Or use ScheduleAnyway instead of DoNotSchedule (allow constraint violation)

References

• Pod Topology Spread Constraints
• Pod Affinity
• GKE Scheduling