GPU Inter-GPU Communication — NCCL Fast Socket, GPUDirect-TCPX/TCPXO, RDMA

Tại sao GPU communication là bottleneck

Training large distributed model:

Local compute (GPU internal):
  ↓ h100.compute_capability = 14000+ TFLOPS
  
Inter-GPU compute synchronization (NCCL allreduce):
  ↓ h100.network_bw (default Ethernet) = 100 Gbps = ~12.5 GB/s
  
Implication:
  - 8 GPUs sync gradients every iteration
  - Gradient size: 70B model = ~140 GB (fp32)
  - Allreduce time: 140GB / 12.5GB/s = 11 seconds
  - GPU compute time: ~0.5 seconds
  
Bottleneck: 95% time on network, 5% on compute
Result: 8 GPU training slower than 1 GPU (because synchronization overhead)

This is why GPU communication is critical path for large-scale training.

---

NCCL (NVIDIA Collective Communication Library)

What NCCL does

NCCL is user-space library (not kernel, not hardware-specific):

Application (PyTorch, JAX, TensorFlow):
  ↓ 
  torch.distributed.all_reduce()
  ↓
  NCCL runtime
  ↓
  GPU driver + CUDA kernel
  ↓
  NIC (network interface card)

NCCL implements collective ops: allreduce, allgather, reduce-scatter, broadcast.

Baseline NCCL performance

Using standard Ethernet (1 NIC per GPU):

Network bandwidth: 100 Gbps = 12.5 GB/s
NCCL allreduce latency (8 GPUs, 1GB gradient):
  ≈ 100ms (includes kernel launch overhead + PCIe overhead)

Sustained throughput: 1GB / 0.1s = 10 GB/s (approaching NIC limit)

---

NCCL Fast Socket: Andromeda optimization

Cơ chế

NCCL Fast Socket is transport plugin (sits between NCCL + NIC driver):

NCCL allreduce()
  ↓
  [NCCL Fast Socket plugin]
  ↓ (optimize via Andromeda virtual network stack)
  NIC
  ↓
  Network

Andromeda integration

Google Cloud's Andromeda is virtual network stack with hardware offload:

Traditional network stack:
  App → kernel TCP/IP → driver → NIC
  (expensive context-switches, memory copies)

Andromeda:
  App → gVNIC (Google Virtual NIC, hardware-assisted)
  (fewer copies, better batching, lower CPU overhead)

NCCL Fast Socket leverages Andromeda to:

1. Reduce overhead: fewer CPU context switches
2. Better batching: group multiple NCCL ops into single network packet
3. Dynamic load balancing: adapt to network congestion

Performance gains

Published benchmark (Google Cloud blog):

Baseline NCCL (TCP):          1.0x (baseline)
NCCL Fast Socket:            1.3-2.6x faster (depending on model)

Example (8 H100s, all-reduce 10GB gradient):

Baseline:       ~800ms
Fast Socket:    ~300ms (2.6x improvement)

8 iterations/sec:
- Baseline: 8 × 0.8s = 6.4s per step
- Fast Socket: 8 × 0.3s = 2.4s per step
→ Training 2.6x faster

Setup requirements

Node pool creation:

gcloud container node-pools create gpu-pool \
  --cluster=my-cluster \
  --machine-type=n1-highmem-8 \
  --accelerator=type=nvidia-tesla-h100,count=8 \
  --enable-nccl-fast-socket=true

GKE automatically:

1. Enable gVNIC on VMs
2. Install NCCL Fast Socket plugin
3. Configure NCCL env vars

Constraints

Hardware: Only H100, H200 (and newer). Not on A100, V100.

❌ Not supported:
  - A100 GPU (older architecture)
  - Memory-optimized machine types
  
✓ Supported:
  - n1-highmem-* (general purpose)
  - h3-standard-* (with H100)

Cannot stack with GPUDirect:

# ❌ INVALID
Both NCCL Fast Socket AND GPUDirect-TCPX
→ GKE rejects node pool creation

GKE version: Requires GKE 1.25.2+ (Standard) or 1.30.2+ (Autopilot).

---

GPUDirect-TCPX/TCPXO: Direct GPU-NIC path

Cơ chế

GPUDirect (NVIDIA) allows direct memory transfer from GPU VRAM → NIC, bypassing CPU/system memory:

Traditional path:
  GPU VRAM → CPU memory (PCIe) → NIC → network
  
GPUDirect path:
  GPU VRAM → NIC (direct, no CPU bounce)
  
Benefit: CPU not involved, less overhead, higher bandwidth

TCPX vs TCPXO vs RDMA

GPUDirect-TCPX (A3 High, A3 Mega):

Bandwidth: ~200 Gbps (vs 100 Gbps vanilla Ethernet)
A3 High:   8 × H100 + TCPX → 1.6 Tbps aggregate
Latency:   1-2 microseconds (within-cluster)

GPUDirect-TCPXO (A3 Mega specialized):

Further optimization: GPU-to-VM communication
Bandwidth: ~600 Gbps (3x TCPX)
Latency:   <1 microsecond

GPUDirect RDMA (A3 Ultra, A4, B200):

Remote Direct Memory Access: no CPU intervention at all
Bandwidth: ~900 Gbps (multiple connections)
Latency:   sub-microsecond
Supported: A3 Ultra (H200), A4 (B200 GPU), future gen

Setup: GPUDirect-TCPX (A3 High example)

gcloud container node-pools create a3-high-pool \
  --cluster=my-cluster \
  --machine-type=a3-highgpu-8g \
  --accelerator=type=nvidia-tesla-h100,count=8 \
  --enable-gpudirect-tcpx=true

GKE:

1. Creates 8 × H100 GPUs on a3-highgpu-8g
2. Enable TCPX path (driver config)
3. Configure Andromeda for direct GPU transfers
4. Expose "nvidia.com/gpu: 8" resource

Pod spec (no special config needed):

spec:
  containers:
  - name: training
    image: training-tcpx:latest
    resources:
      limits:
        nvidia.com/gpu: 8
  # NCCL + TCPX auto-detected via driver

TCPX architecture

8 GPUs on a3-highgpu-8g:

GPU 0 — \
GPU 1 — |— TCPX NIC 0 —|
GPU 2 — |               | → Inter-VM fabric (A3 network)
GPU 3 — /               |
GPU 4 — \               |
GPU 5 — |— TCPX NIC 1 —|
GPU 6 — |
GPU 7 — /

Each NIC handles 4 GPUs (H100 pairs)
Inter-NIC links: high-speed InfiniBand-like

---

GPUDirect RDMA: Latest technology

Architecture (A3 Ultra example)

A3 Ultra machine type:
- 16 × H200 GPUs (141GB HBM3 each)
- RDMA (no TCP, true RDMA protocol)
- IB (InfiniBand) switch topology

RDMA flow:
GPU VRAM addr 0x1000 (on node A)
↓ (RDMA verb: rdma_write)
NIC (node A) → switch → NIC (node B)
↓
GPU VRAM addr 0x2000 (on node B)

No CPU involvement, no kernel syscall
Direct DMA: bandwidth = full NIC speed × #connections

Performance (A3 Ultra vs A3 High)

8-node training cluster (64 GPUs):

A3 High (TCPX):
  Allreduce (10GB gradient): ~150ms
  Training step: 1.5s
  
A3 Ultra (RDMA):
  Allreduce (10GB gradient): ~40ms
  Training step: 0.5s
  
Speed improvement: 3x faster training

Constraints

• Cost: A3 Ultra expensive (~$50-80/hour vs A3 High $20-30/hour)
• Limited availability: Only us-central1 region (as of 2026)
• Requires Dataplane V2: GKE must use Dataplane V2 (not standard VPC)
• Network capacity: 8+ nodes typically needed for inter-node scaling

---

Network scaling patterns

Single-node training (no inter-GPU network)

1 × node with 8 GPUs
All communication intra-node (PCIe):
  PCIe 4.0 bandwidth: ~32 GB/s
  NCCL allreduce (1GB): ~30ms
  
No network involved → NCCL Fast Socket/GPUDirect not needed

2-node training (inter-node communication)

Network becomes critical:

2 nodes × 8 GPUs = 16 GPUs total
Allreduce: every node must sync with every other node

Bandwidth requirement:
  ~100GB/s per all-reduce
  Ethernet (100 Gbps): 12.5 GB/s → saturated
  → NCCL Fast Socket (2.6x) or GPUDirect needed

8+ node training (collective operations bottleneck)

8 nodes × 8 GPUs = 64 GPUs

Allreduce (ring topology):
  Node 0 → Node 1 → Node 2 → ... → Node 7 → Node 0
  Bandwidth: limited by slowest link

With TCPX (200 Gbps):
  Allreduce time: 10GB ÷ (200Gbps/8 nodes) ≈ 40ms
  
With RDMA (900 Gbps full duplex):
  Allreduce time: 10GB ÷ (900Gbps) ≈ 10ms
  
Scaling law: RDMA mandatory for 16+ GPU clusters

---

Practical tuning

NCCL environment variables

# Force Fast Socket plugin
export NCCL_PLUGIN=com_google_gke_nccl_fast_socket

# Enable profiling (debug performance)
export NCCL_DEBUG=INFO

# Custom timeout (for large models)
export NCCL_TIMEOUT=300  # 300 seconds

# Disable nvlink fallback (force TCPX)
export NCCL_PROTO=SIMPLE

Bandwidth testing

# Check NCCL bandwidth (8 GPUs)
docker run --rm --gpus all nvidia/samples:nccl-tests \
  /usr/bin/all_reduce_perf -b 1G -e 10G

# Expected output with TCPX:
# algo: Ring
# time: ~150ms (for 10GB allreduce)

Collective operation optimization

# PyTorch DDP configuration
model = DistributedDataParallel(
    model,
    device_ids=[0,1,2,3,4,5,6,7],
    
    # Gradient bucketing (reduce allreduce latency)
    broadcast_buffers=False,
    find_unused_parameters=False,  # Optimize for dense models
    
    # NCCL optimization
    backend='nccl',  # Use NCCL (not GlooOP)
)

# Enable NCCL Fast Socket at runtime
os.environ['NCCL_PLUGIN'] = 'com_google_gke_nccl_fast_socket'

---

Failure modes

Bandwidth degradation (undetected)

Training job with A3 High cluster:
  Expected allreduce: 8 × 200 Gbps = 1.6 Tbps
  Observed allreduce: 400 Gbps
  
→ Silent 4x degradation (no error, just slow training)
  
Cause options:
  1. TCPX driver not loaded (fallback to TCP)
  2. Network congestion (other VMs on fabric)
  3. CPU bottleneck (training job CPU-bound, NCCL starved)
  
Investigation:
  # Check NCCL debug output
  export NCCL_DEBUG=INFO
  python training.py 2>&1 | grep "NCCL"
  # Look for: "[NCCL] Selected transport: TCPX" (good)
  #           vs "[NCCL] Selected transport: TCP" (bad)

Topology mismatch

Training code assumes all 8 GPUs same location (same NIC):
  GPU 0-3 on NIC 0, GPU 4-7 on NIC 1

But cluster admin mistakenly split:
  GPU 0-2 on Node A, GPU 3-7 on Node B

→ Allreduce pattern optimized for single-node
  → crosses node boundary frequently
  → bandwidth degradation (inter-node slower)

---

Mental model: Network tier selection

GPU cluster size:
  1-8 GPUs     → No special networking needed
                 (intra-node PCIe, or TCP fine)
  
  9-32 GPUs    → NCCL Fast Socket recommended
                 (low cost, good gains)
  
  32-128 GPUs  → GPUDirect-TCPX required
                 (mandatory for training at scale)
  
  128+ GPUs    → GPUDirect RDMA ideal
                 (best throughput, lowest latency)
  
  HPC/simulation → RDMA mandatory
                   (tight latency coupling)

---

References

• GKE NCCL Fast Socket
• GKE GPU Bandwidth Optimization (Standard)
• GKE GPU Bandwidth Optimization (Autopilot)
• GKE A3 Cluster Setup
• NVIDIA NCCL Documentation
• GPUDirect RDMA Documentation