High-Performance Networking — InfiniBand, A3 Clusters, Compact Placement, H4D

Tại sao HPC/Supercomputer-grade networking quan trọng

Large-scale distributed training + HPC simulations:

100-GPU training:
  Communication frequency: every iteration (~1 second)
  Allreduce size: 100GB (gradient aggregation)
  Network latency: 1ms vs 100µs (100x difference)
  
With 1ms latency:
  Allreduce wait time: ~50ms
  
With 100µs latency:
  Allreduce wait time: ~0.5ms
  
Over 1M training steps: 50,000ms vs 500ms = 100x difference in total sync time

Mental model: For tightly-coupled systems (all-reduce every step), latency dominates cost. Sub-microsecond latency is worth premium.

---

A3 machine series: GPU+InfiniBand architecture

Machine types

a3-highgpu-8g
  8 × H100 GPUs (80GB each)
  GPUDirect-TCPX network
  32 vCPU, 256GB memory

a3-megagpu-8g
  8 × H100 GPUs
  GPUDirect-TCPXO network (further optimized)
  32 vCPU, 256GB memory

a3-edgegpu-8g
  8 × H100 GPUs
  Standard Ethernet (cheaper alternative)
  
a3-ultrapcpugpu-8g (Ironwood era)
  16 × H200 GPUs (141GB HBM3 each)
  GPUDirect RDMA
  64 vCPU, 1TB memory

TCPX network characteristics

Bandwidth:

Per-GPU: ~25 Gbps sustained
8 GPUs: ~200 Gbps aggregate
(vs 100 Gbps standard Ethernet)

Latency:

p50: 1-2 microseconds (within same node)
p99: 10-20 microseconds
(vs 100-1000 microseconds Ethernet)

Topology:

8 GPU nodes cluster (64 GPUs):
Node-to-node latency: 5-50 microseconds (depends on cluster fabric)
This enables true HPC-class collective operations

Dataplane V2 requirement

A3 clusters must use GKE Dataplane V2:

gcloud container clusters create a3-cluster \
  --cluster-version=1.34+ \
  --enable-dataplane-v2 \
  --machine-type=a3-highgpu-8g

Why: Dataplane V2 offloads networking to custom silicon (not software kernel). Required to achieve TCPX throughput.

Implication: Standard VPC networking insufficient. Uses custom fabric.

---

Compact placement for topology-aware scheduling

Cơ chế

Compact placement policy ensures all Pod replicas run on geographically close nodes:

Without compact placement:
Node A: Zone=us-central1-a, Pod-0
Node B: Zone=us-central1-b, Pod-1
Node C: Zone=us-central1-a, Pod-2

→ All-reduce: a→b→c→a (crosses zones frequently)
→ Cross-zone latency: ~1ms
→ Inter-zone bandwidth: bottleneck (GCP routes cross-zone traffic through fabric)

With compact placement:
Node A: Zone=us-central1-a, Pod-0
Node B: Zone=us-central1-a, Pod-1
Node C: Zone=us-central1-a, Pod-2

→ All-reduce: a→a→a (single zone)
→ Same-zone latency: 5-50µs
→ Same-zone bandwidth: full fabric capacity

Setup

gcloud container clusters create hpc-cluster \
  --zone=us-central1-a \
  --enable-compact-placement-policy \
  --compact-placement-policy-name=hpc-compact

gcloud container node-pools create gpu-pool \
  --cluster=hpc-cluster \
  --machine-type=a3-highgpu-8g \
  --placement-type=COMPACT

Pod scheduling (automatic):

spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: distributed-training
        topologyKey: topology.kubernetes.io/zone

Constraints

Compact placement not all machine types. Supported only on:

✓ A2, A3, A4 (GPU)
✓ C2, C2D, C3, C3D, C4, C4D (Compute)
✓ G2, G4 (GPU older gen)
✓ H3, H4D (HPC)
✓ N2, N2D (General)

✗ n1, n2 (standard general-purpose)
✗ e2 (budget)

---

H4D: HPC-optimized machine type

Hardware specs

h4-standard-96
  96 vCPU (AMD EPYC Genoa, Titanium)
  768 GB memory
  Up to 15 Gbps local SSD throughput
  Cloud RDMA (low-latency network)

h4d-standard-96
  96 vCPU
  768 GB memory
  8TB Local SSD (NVMe) ← KEY DIFFERENCE
  Local SSD for fast I/O (HPC simulation data)

Use cases

H4D designed for tightly-coupled HPC simulations:

Lattice Boltzmann simulation (fluid dynamics):
- 100 nodes × 96 cores = 9,600 cores
- Each iteration communicates with neighbors (stencil communication)
- Requires <100µs latency, high bandwidth
- Local SSD: store checkpoint data

CFD simulation (Computational Fluid Dynamics):
- Similar tight coupling, neighbor communication
- Compact placement ensures low-latency mesh

Machine learning + HPC (training + physics-based loss):
- Distributed training (all-reduce) + HPC simulation (stencil)
- Benefits from RDMA + compact placement

Automatic compact placement for H4D

gcloud container node-pools create h4d-pool \
  --cluster=my-cluster \
  --machine-type=h4d-standard-96
  # Compact placement automatically enabled

GKE automatically:

1. Enables compact placement policy
2. Configures cloud RDMA
3. Sets up local SSD volumes for Pod I/O

---

Multi-NIC Pods: Advanced networking

Motivation

Single NIC bottleneck (even with TCPX):

100-GPU training:
  Allreduce requires ~10 Tbps cross-section bandwidth
  Single NIC: 200 Gbps
  Needed NICs: 10 Tbps / 200 Gbps = 50 NICs equivalent

Solution: Multi-NIC Pods
  Each Pod attaches 2-4 NICs
  Each NIC handles different flow
  → Aggregate 400-800 Gbps per node

Setup (A3 Mega with multi-NIC)

apiVersion: v1
kind: Pod
metadata:
  name: training-job
spec:
  containers:
  - name: trainer
    image: training:latest
    resources:
      limits:
        google.com/gke-multi-nic: 4  # 4 network interfaces
        nvidia.com/gpu: 8

  # Network attachments
  networks:
  - name: default  # Primary network
  - name: secondary-1
  - name: secondary-2
  - name: secondary-3

GKE provisioning:

Node has 4 × TCPX NICs
Pod attaches all 4 NICs
Training framework (PyTorch, JAX) uses NCCL Fast Socket on all 4 NICs
Aggregate bandwidth: 4 × 200Gbps = 800 Gbps

NCCL over multi-NIC

# NCCL auto-detects multi-NIC and parallelizes
export NCCL_DEBUG=INFO

# Check: NCCL messages should show
# "Selected transport: TCPX" (all NICs)

---

Cluster scaling patterns

Small cluster (1-8 nodes)

1-8 × H100 GPU nodes:
  Latency within cluster: <100µs (same rack)
  No special networking needed
  Standard Ethernet sufficient
  
Configuration:
  n1-highmem-8 + 8 GPU + standard VPC
  Cost: $3-5/hour per node

Medium cluster (8-32 nodes, <32 GPUs)

8-32 × H100 nodes:
  Intra-cluster latency: 1-100µs
  Cross-zone traffic becomes relevant
  
Configuration:
  a3-highgpu-8g + TCPX + compact placement
  Cost: $20-30/hour per node
  
Training: 8-node 4x4x4 TPU equivalent performance (but with GPUs)

Large cluster (32+ nodes, 256+ GPUs)

64-256 GPUs (8-32 nodes A3 Mega):
  Intra-cluster latency: <50µs
  Single zone mandatory
  Multi-NIC highly beneficial
  
Configuration:
  a3-megagpu-8g + multi-NIC (2-4 per node) + Dataplane V2
  Cost: $25-35/hour per node
  
Critical: RDMA becomes preferred over TCPX

HPC cluster (H4D, extreme scale)

1000+ vCPU (10-20 × H4D):
  Local SSD for I/O bandwidth
  Compact placement policy
  Cloud RDMA networking
  
Configuration:
  h4d-standard-96 + Cloud RDMA + compact placement
  Cost: $15-25/hour per node
  
Use case: Coupled simulations (CFD, weather, seismic)

---

Operational challenges

Blast radius: cascading failures

A3 cluster, 8 nodes, single fabric:

Node 5 NIC fails
→ All-reduce rings blocked
→ All 8 Pods timeout (waiting for Node 5)
→ Training job fails entirely

vs single-node failure in distributed training:
→ Can often skip failed batch, continue
→ A3: cannot (collective operations atomic)

→ Solution: ProvisioningRequest with restart checkpoint
           or switch to non-collective-heavy architecture

Network congestion

Multiple training jobs on same cluster:

Job A: allreduce pattern (ring)
Job B: allreduce pattern (ring)
→ Both compete for fabric bandwidth
→ Both experience 2x latency degradation

vs separate clusters:
Job A on cluster A: full fabric
Job B on cluster B: full fabric

→ Trade-off: cost (multi-cluster) vs performance isolation

Upgrade complexity

Upgrading A3 cluster:

To add nodes to running cluster:
  Drain existing nodes (pause training)
  Add new nodes
  Restart training (from checkpoint)

vs Standard GKE:
  Add nodes while cluster runs
  Workloads migrate seamlessly

→ A3 clusters require coordination (not seamless)

---

Cost analysis

Training 70B LLM, 1 epoch (1M steps, 1 hour):

Standalone GPU (8× h100, $3/hr):
  Time: 8 hours (slow all-reduce, no TCPX)
  Cost: $24

A3 Mega cluster (2 nodes, TCPX, $50/hr):
  Time: 1 hour (fast all-reduce)
  Cost: $100
  
Break-even: A3 is 4x more expensive per job, but 8x faster
           → 2x more expensive per unit training time
           
But if you can't train at all without TCPX (>4 nodes):
           → A3 is mandatory, not optional

---

Mental model: Networking tiers

Ethernet standard (100 Gbps):
  Use for: inference, batch jobs, async training
  Latency: 1-10ms, sufficient for most

NCCL Fast Socket (200 Gbps, <1ms):
  Use for: GPU training (8-32 GPUs), prefer cost+latency balance

GPUDirect-TCPX (200+ Gbps, 1-50µs):
  Use for: large GPU training (32-128 GPUs), mandatory for scale

GPUDirect RDMA (900+ Gbps, <1µs):
  Use for: HPC simulations, extreme-scale training (128+ GPUs)

Compact placement:
  Use for: any tightly-coupled workload (training, HPC)
  Enable whenever available

---

References

• GKE A3 High/Mega Cluster Setup
• GKE Compact Placement
• GKE H4D HPC Workloads
• Compute Engine H4D Machine Type
• GKE Multi-NIC
• Dataplane V2 Architecture