LLM Serving Patterns — vLLM, TGI, Triton, Optimization

Tại sao LLM serving khác biệt so với training

Training:

Input: fixed batch, all sequences same length
Compute: dense matrix multiply, high utilization
Output: logits (small)

Inference (serving):

Input: variable batch size, variable sequence length
Compute: autoregressive generation (iterative)
Output: variable length tokens

Scaling challenge: Training benefits from batch size (linear speedup to ~1000). Serving batching is latency-constrained (SLA: respond within 100ms), so effective batch is 10-100.

---

vLLM: PagedAttention + continuous batching

Key innovation: PagedAttention

Traditional attention (decoder phase):

For each new token generation:
  Load entire KV cache (key-value) from memory
  Compute attention
  Output 1 new token
  
KV cache allocation: contiguous block
Problem: if different sequence lengths, memory fragmented

vLLM PagedAttention:

KV cache allocated in "pages" (blocks)
Each page = 16 tokens worth of KV data
  
Sequence A: pages [0, 1, 5] (3 pages = 48 tokens)
Sequence B: pages [2, 3, 4] (3 pages = 48 tokens)
(pages are non-contiguous, but paged memory maps them)

Benefit: no fragmentation, high KV cache utilization
Drawback: must implement page-aware kernels (vLLM does this)

Continuous batching

Without batching (static batch):

Batch 1: [Seq A (length 50), Seq B (length 50)]
  t=0-10: generate token 1 for both (pad to max length)
  t=10-20: generate token 2 for both
  Throughput: 2 seqs × 100 tokens = 200 tokens/sec

With continuous batching:
  t=0: [Seq A (length 50), Seq B (length 50)]
  t=1: [Seq A (51), Seq B (51), Seq C (1)]
  t=2: [Seq A (52), Seq B (52), Seq C (2)]
  (Seq A/B finish, new seqs enter continuously)
  Throughput: 3 seqs × 100 tokens = 300 tokens/sec (50% better)

GKE deployment

apiVersion: v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - --model=meta-llama/Llama-2-70b-hf
        - --tensor-parallel-size=2  # Distributed across 2 GPUs
        - --max-num-seqs=256  # Continuous batching: up to 256 concurrent requests
        resources:
          limits:
            nvidia.com/gpu: 2
      # 3 replicas × 2 GPUs = 6 GPUs total

---

Text Generation Inference (TGI): HuggingFace's serving framework

Optimizations

TGI provides:

1. Continuous batching: similar to vLLM
2. Tensor parallelism: automatically shard model across GPUs
3. Token streaming: return tokens as they generate (low latency perceived)
4. Watermark handling: for token classification

Setup

apiVersion: v1
kind: Deployment
metadata:
  name: tgi-server
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: tgi
        image: ghcr.io/huggingface/text-generation-inference:latest
        env:
        - name: MODEL_ID
          value: meta-llama/Llama-2-7b
        - name: CUDA_MEMORY_FRACTION
          value: "0.9"  # Use 90% of GPU VRAM
        ports:
        - containerPort: 80
        resources:
          limits:
            nvidia.com/gpu: 1
        livenessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 30

TGI vs vLLM trade-off

Feature	vLLM	TGI
Batching	PagedAttention	Standard batching
Latency	Better (p50)	Comparable
Throughput	Higher	Comparable
Setup	More complex	Simpler
Community	Growing	Established

---

NVIDIA Triton + TensorRT-LLM: Enterprise serving

Architecture

Triton Inference Server
├─ TensorRT-LLM backend
│  ├─ Kernel fusion (layer operations)
│  ├─ In-flight batching (similar to vLLM)
│  ├─ Quantization support (INT8, INT4)
│  └─ Optimized for NVIDIA GPUs
├─ Model repository
│  ├─ Config files
│  └─ Engine files (pre-compiled)
└─ HTTP/gRPC endpoints

Key advantage: Multi-model serving

# Triton can serve multiple models on same GPU
models:
- name: llama2-7b
  backend: tensorrt_llm
  instance_group:
  - count: 1
    kind: GPU
    gpus: [0]
  
- name: mistral-7b
  backend: tensorrt_llm
  instance_group:
  - count: 1
    kind: GPU
    gpus: [1]

# Or time-share on single GPU (time-slicing)
- name: llama-5b
  instance_group:
  - count: 2  # 2 concurrent requests
    kind: GPU
    gpus: [0]
    profile: time-shared

GKE setup

# Deploy Triton server
helm install triton nvcr.io/nvidia/triton-server:latest \
  --set image.pullPolicy=IfNotPresent \
  --set gpuCount=4 \
  --set model.repository.path="/models"

# Models stored in GCS, mounted via Hyperdisk ML

---

Batch size tuning: latency vs throughput

Trade-off

Batch size = 1:
  Latency: 50ms (model forward pass)
  Throughput: 20 tokens/sec

Batch size = 32:
  Latency: 80ms (batching overhead, queueing)
  Throughput: 400 tokens/sec (20x!)

Batch size = 256:
  Latency: 150ms (queue depth too high)
  Throughput: 1000 tokens/sec
  But 150ms > SLA (100ms), unacceptable

Optimization: Use batching until latency SLA breached, then reject new requests (backpressure).

---

Model parallelism strategies

Tensor parallelism (TP)

Split model layers across GPUs:

Layer 0 matmul (Linear, 14000×14000):
  GPU 0: compute left half (14000×7000)
  GPU 1: compute right half (14000×7000)
  Result: reduce-sum (1 all-reduce per layer)
  
Cost: 1 all-reduce per layer
Performance: linear scaling (7 GPUs → 7x faster)

Pipeline parallelism (PP)

Split model stages across GPUs:

Layers 0-20: GPU 0
Layers 21-40: GPU 1
...
Layers 60-80: GPU 3

Forward pass:
  GPU 0 runs layers 0-20 → GPU 1
  GPU 1 runs layers 21-40 → GPU 2
  (pipelined, but GPU 0 idle while GPU 1 computing)
  
Cost: activation checkpointing (memory overhead)
Performance: sub-linear (dependent on pipeline depth)

Optimal for LLM inference

Most inference frameworks use Tensor Parallelism:

70B Llama model:
  Without TP: fit on 1 H100 (80GB)
  With TP across 2 GPUs: better memory distribution
  With TP across 4 GPUs: highest throughput (smaller stages)
  
Recommendation:
  7B model → 1 GPU (TP not needed)
  13B → 1 GPU (barely fits)
  70B → 2-4 GPUs (TP across 2-4)
  175B (GPT-3 scale) → 4-8 GPUs (TP critical)

---

Autoscaling metrics for inference

Kubernetes autoscaling (HPA) typically uses CPU/memory, but these don't correlate with LLM latency:

CPU 30% ← CPU idle while GPU busy
Memory 50% ← model weights static, not growing

Better metrics for LLM:
- Queue depth: number of pending requests
- Token generation rate: tokens/sec (should be constant)
- Time-to-first-token (TTFT): latency for first token
- Time-between-tokens (TBT): latency for subsequent tokens

Custom autoscaling

apiVersion: autoscaling.custom.metrics.k8s.io/v1beta1
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-server
  minReplicas: 2
  maxReplicas: 10
  
  metrics:
  - type: Pods
    pods:
      metricName: vllm_queue_depth  # Custom metric
      targetAverageValue: "20"  # Scale when queue > 20 requests per Pod
  
  # Also can use:
  # - vllm_time_to_first_token (should be < 100ms)
  # - vllm_token_throughput (tokens/sec)

---

Common patterns and optimizations

Pattern 1: Streaming response

# Client streams tokens as they arrive (latency perceived as lower)
from openai import OpenAI

client = OpenAI(
    base_url="http://vllm-server:8000/v1",
    api_key="dummy"
)

response = client.chat.completions.create(
    model="meta-llama/Llama-2-70b",
    messages=[{"role": "user", "content": "What is AI?"}],
    stream=True  # Stream tokens
)

for chunk in response:
    print(chunk.choices[0].delta.content, end="", flush=True)

Pattern 2: Prompt caching

# If same prompt prefix reused (e.g., system message)
# Cache KV cache to avoid recomputing

system_prompt = """You are a helpful assistant..."""
# vLLM automatically detects repeated prefixes
# and reuses KV cache (if prompt cache enabled)

Pattern 3: Speculative decoding

# Use smaller model to predict next tokens
# Verify with larger model (reduces latency)

Small model (7B): predicts tokens A, B, C
Large model (70B): confirms A, B but rejects C
Cost: 1 large forward pass to verify ~3 predictions
Result: ~2x faster generation with quality loss <<1%

---

Failure modes

Memory OOM during batching

vLLM with batch_size=64:
  KV cache grows with batch
  After 5 concurrent long-context requests
  KV cache pressure → GPU OOM

Fix:
  1. Reduce batch size
  2. Enable quantization (reduce model size)
  3. Use LoRA (parameter-efficient serving)

Latency spikes

Scenario:
  Steady state: 50 concurrent requests, p99 latency 80ms
  Spike: 500 requests arrive (e.g., popular query)
  
Result:
  Queue depth explodes
  p99 latency jumps to 500ms
  Requests timeout
  
Solution:
  1. Autoscale (HPA) but takes 30s to provision
  2. Request queuing with deadline rejection
  3. Load shedding (reject new requests gracefully)

---

Mental model: Inference tier selection

Throughput requirement:
  <100 tokens/sec    → Single GPU (7-13B model)
  <1000 tokens/sec   → 2-4 GPUs (TP enabled)
  >1000 tokens/sec   → Multi-replica + TP (8+ GPUs)

Latency requirement:
  p99 < 50ms  → Single GPU, small batch
  p99 < 100ms → TP (2-4 GPUs), moderate batch
  p99 < 500ms → Multi-replica + load balancing

Multi-model serving:
  <3 models    → Separate deployments
  3-10 models  → Triton multi-model (time-share)
  >10 models   → LoRA adapters (parameter efficiency)

---

References

• vLLM Documentation
• HuggingFace TGI
• NVIDIA TensorRT-LLM
• GKE LLM Best Practices
• GKE LLM Autoscaling
• Serving Gemma on GKE