Sidecar Performance — Overhead, Latency Impact và Resource Tuning

Sidecar Overhead là Real Cost

Mỗi Pod trong service mesh cần chạy thêm Envoy sidecar (istio-proxy). Đây không phải overhead nhỏ:

• CPU: Envoy cần CPU để process mỗi packet (L7 inspection, mTLS crypto, telemetry)
• Memory: Envoy cache xDS config, connection pools, trace buffers
• Latency: Mỗi request hop thêm 1-5ms (outbound Envoy + inbound Envoy)
• Connections: Mỗi Pod thêm N connections đến Istiod (xDS streams)

Với cluster 500 Pods, sidecar overhead có thể là:

Memory: 500 pods × 128Mi base = 64GB thêm
CPU: 500 pods × 100m base = 50 CPU cores thêm
Latency: 2ms per call × 1M RPS = significant tail latency impact

Hiểu overhead giúp bạn:

1. Size resource requests/limits đúng cho Envoy
2. Tune concurrency để optimize throughput
3. Quyết định khi nào cần sidecarless (ambient mesh)

Đo Lường Overhead Thực Tế

Benchmark Setup

Baseline (no mesh):
  Client → Service A → Service B

With mesh:
  Client → [Envoy A outbound] → [Envoy B inbound] → Service A
         → [Envoy A outbound] → [Envoy B inbound] → Service B

CPU Overhead

Envoy CPU consumption phụ thuộc vào:

1. Request rate (RPS):

   • Low RPS (<100): ~10-20m CPU (mostly idle)
   • Medium RPS (1000): ~50-200m CPU
   • High RPS (10000): ~500m-2 CPU

2. Protocol: HTTP/2 > HTTP/1.1 (multiplexing overhead)

   • HTTP/1.1 at 1000 RPS: ~50m CPU
   • HTTP/2 at 1000 RPS: ~70m CPU (multiplexing benefits at scale)

3. mTLS: Adds ~10-20% CPU overhead vs plain text

   • TLS 1.3 faster than TLS 1.2
   • AES-NI hardware acceleration giảm crypto overhead

4. Telemetry: Metrics emission, access log writing, trace reporting

   • Disabled: baseline
   • Metrics only: +5% CPU
   • Metrics + tracing (1%): +2% CPU
   • Metrics + tracing (100%): +20% CPU

Thực tế đo trong production:

Service: gRPC service với 5000 RPS
Envoy sidecar CPU (measured):
- P50: 80m cores
- P99: 150m cores
- Spike to 300m during connection churn (new pods rolling)

Memory Overhead

Envoy memory consumption:

Base (no connections): ~40-60Mi
Per active upstream cluster: ~2-5MB (xDS cache)
Per active connection: ~32KB (connection buffer)
Per active HTTP/2 stream: ~16KB
Metrics timeseries: ~10-50Mi

Estimate formula:
  memory = 60Mi base
         + (N_clusters × 2MB)
         + (N_connections × 32KB)
         + (N_streams × 16KB)
         + telemetry_buffer

Typical Pod trong medium-traffic service: 128-256Mi

Memory spike scenarios:

• Rolling update: Khi Pods replace, connections cũ không close ngay → memory tạm thời 2×
• Large xDS push: Istiod push update với nhiều clusters → memory spike trong vài giây
• Connection surge: Traffic burst → nhiều connections cùng lúc

Latency Overhead

Mỗi request hop trong mesh adds:

Client request đến Service A:
  Original (no mesh): 1ms network + 5ms app = 6ms total
  
  With mesh:
    + 0.2ms Envoy A outbound (routing, mTLS setup cached)
    + 1ms network
    + 0.2ms Envoy B inbound (mTLS verify, policy check)
    + 5ms app
    = 6.4ms total
    
  Overhead: 0.4ms per hop (P50)
  
  P99 overhead: 1-5ms per hop
  (High latency cases: mTLS re-handshake, xDS update applying)

Latency contributors:

1. L7 parsing: Envoy parse HTTP/gRPC headers để apply routing rules

   • HTTP/1.1: ~0.05ms
   • HTTP/2: ~0.1ms (header decompression, HPACK)

2. mTLS handshake: Chỉ xảy ra khi connection mới

   • TLS handshake: ~1-3ms (one-time cost)
   • Subsequent requests trên same connection: 0 overhead

3. Telemetry: Stats update, access log write

   • ~0.05ms per request (batched writes)

4. Policy check: AuthorizationPolicy evaluation

   • Simple policy: <0.1ms
   • Complex Rego policy với OPA: up to 2ms

Tuning Envoy Concurrency

Envoy Worker Threads

Envoy dùng single-threaded event loop per worker thread. Mặc định, số worker threads = số CPU cores:

1 CPU core → 1 worker thread
2 CPU cores → 2 worker threads
...

Vấn đề: Nếu bạn allocate 100m CPU (0.1 core) cho Envoy, Kubernetes có thể schedule Pod trên node 32-core, và Envoy sẽ tạo 32 worker threads — nhưng chỉ có 0.1 core budget!

Giải pháp: Set concurrency explicitly:

# Trong Pod annotation
annotations:
  proxy.istio.io/config: |
    concurrency: 2   # 2 worker threads, match CPU limit

# Hoặc trong MeshConfig (global default)
meshConfig:
  defaultConfig:
    concurrency: 2

Heuristic:

• 100m-500m CPU limit → concurrency: 1
• 500m-2 CPU limit → concurrency: 2
• 2-4 CPU limit → concurrency: 4
• >4 CPU limit → concurrency: min(cores, 8)

Envoy không benefit từ >8 workers vì overhead của cross-thread synchronization tăng.

CPU và Memory Requests/Limits

# Trong Deployment (Envoy sidecar được inject bởi webhook)
# Nhưng có thể override qua annotations
annotations:
  sidecar.istio.io/proxyCPU: "100m"
  sidecar.istio.io/proxyMemory: "128Mi"
  sidecar.istio.io/proxyCPULimit: "2000m"
  sidecar.istio.io/proxyMemoryLimit: "1024Mi"

Resource sizing guidelines theo load profile:

Traffic Pattern	CPU Request	CPU Limit	Memory Request	Memory Limit
Low (<100 RPS)	10m	500m	64Mi	256Mi
Medium (100-1000 RPS)	100m	2000m	128Mi	512Mi
High (1000-10000 RPS)	500m	4000m	256Mi	1024Mi
Very High (>10000 RPS)	1000m	8000m	512Mi	2048Mi

Tại sao request thấp nhưng limit cao:

• Envoy idle thường rất nhẹ
• Nhưng peak (connection churn, config push) cần CPU burst
• CPU throttling gây latency spike nguy hiểm hơn OOM
• → Limit cao để allow burst

Connection Pool Tuning

apiVersion: networking.istio.io/v1
kind: DestinationRule
spec:
  trafficPolicy:
    connectionPool:
      tcp:
        # Max connections Envoy duy trì đến upstream
        maxConnections: 100
        
        # TCP keepalive để detect dead connections
        tcpKeepalive:
          time: 7200s   # Start keepalive sau 2h idle
          interval: 75s  # Probe interval
          probes: 9      # Số probes trước khi declare dead
      
      http:
        # HTTP/1.1: pending requests khi no available connections
        http1MaxPendingRequests: 1000
        
        # HTTP/2: max concurrent streams
        http2MaxRequests: 10000
        
        # Close connection sau N requests (giúp load distribute khi pods scale)
        maxRequestsPerConnection: 100
        
        # Idle connection timeout
        idleTimeout: 300s
        
        # Upgrade HTTP/1 → HTTP/2 nếu possible
        h2UpgradePolicy: UPGRADE

Connection pool sizing:

Đối với HTTP/1.1:
  N_connections cần = RPS × P99_latency_seconds
  Ví dụ: 1000 RPS × 0.05s = 50 connections

  maxConnections = 2 × calculated (buffer cho bursty traffic)
  http1MaxPendingRequests = 10% × maxConnections (queue khi connections full)

Đối với HTTP/2:
  Một connection có thể handle nhiều streams
  http2MaxRequests = max concurrent requests bạn expect

Sidecar Resource Monitoring

Key Metrics để Monitor Envoy Health

# Envoy memory usage per Pod
container_memory_working_set_bytes{
  container="istio-proxy",
  namespace="production"
}

# Envoy CPU usage
rate(container_cpu_usage_seconds_total{
  container="istio-proxy",
  namespace="production"
}[5m])

# Envoy connection pool overflow (circuit breaker tripped)
increase(envoy_cluster_upstream_cx_overflow{
  namespace="production"
}[5m])

# Pending requests (queue depth)
envoy_cluster_upstream_rq_pending_active{
  namespace="production"
}

# Envoy memory pressure (active connections)
envoy_cluster_upstream_cx_active{
  namespace="production"
}

Envoy Admin Stats

# Access Envoy admin
kubectl port-forward pod/my-pod 15000:15000

# Memory stats
curl localhost:15000/memory
# heap_size: 134217728  (128MB)
# allocated: 95237120   (91MB actually used)

# Worker thread stats
curl localhost:15000/server_info | jq '.concurrency'

# Connection pool stats
curl localhost:15000/stats | grep "upstream_cx_active"

# Watchout metrics:
# upstream_cx_overflow > 0 → connection pool full, requests dropped
# upstream_rq_timeout > 0 → upstream slow
# upstream_cx_connect_fail > 0 → network issues

Optimization Strategies

1. Disable Telemetry cho Low-Value Services

# Disable access logging để reduce CPU (không cần cho all services)
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: disable-access-log
  namespace: production
spec:
  selector:
    matchLabels:
      app: high-throughput-internal-service
  accessLogging:
  - disabled: true  # Disable cho services với very high RPS

2. Reduce Tracing Overhead

# Giảm sampling rate cho non-critical services
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: reduce-tracing
spec:
  selector:
    matchLabels:
      app: batch-processor
  tracing:
  - randomSamplingPercentage: 0.1  # 0.1% cho batch jobs

3. HTTP/2 Optimization

# Enable HTTP/2 cho gRPC và high-concurrency services
# (Thường tự động với gRPC, nhưng cần confirm)
destinationRule:
  trafficPolicy:
    connectionPool:
      http:
        h2UpgradePolicy: UPGRADE  # Force HTTP/2

# Với HTTP/2, một connection support nhiều concurrent requests
# → Giảm số connections cần duy trì
# → Giảm TLS handshake overhead

4. Sidecar Resource với WasmPlugin (Advanced)

# Nếu dùng custom Envoy WASM filter → tăng memory limit
annotations:
  sidecar.istio.io/proxyMemoryLimit: "512Mi"  # WASM filters tốn memory

# Specify WASM execution mode
# (isolation: "sandbox" vs "none" - sandbox an toàn hơn nhưng chậm hơn)

5. Exclude Non-Mesh Services từ Sidecar

# Services không cần mesh features → opt-out
apiVersion: networking.istio.io/v1
kind: Sidecar
metadata:
  name: batch-job-sidecar
  namespace: production
spec:
  workloadSelector:
    labels:
      app: batch-processor
  egress:
  # Chỉ allow egress đến các services cần thiết
  - hosts:
    - "production/database-service"
    - "production/object-storage"
    # Không include toàn bộ mesh → giảm xDS config size → giảm memory

Ambient Mesh: Sidecarless Future

Tại sao Ambient Mesh?

Sidecar model có fundamental limitations:

1. Per-Pod overhead: 128Mi memory × 10000 Pods = 1.25TB memory chỉ cho proxies!
2. Startup latency: Init container + sidecar bootstrap = 1-3s delay
3. Operational complexity: Sidecar upgrades require Pod restarts (rolling restart toàn fleet)
4. NET_ADMIN requirement: Hoặc phải dùng CNI plugin

Ambient mesh approach:

• Không inject sidecar vào Pods
• Traffic xử lý bởi node-level ztunnel (per-node L4 proxy)
• L7 features handled bởi waypoint proxy (per-namespace hoặc per-service)

Sidecar model:
  [Pod A: app + Envoy] → [Pod B: app + Envoy]
  Mỗi Pod: +128Mi memory, +0.2ms latency

Ambient model:
  [Pod A: app only] → ztunnel → [Pod B: app only]
  Per-node ztunnel: shared cost
  Waypoint proxy: chỉ khi cần L7 features

Ambient Mesh với CSM

CSM đang preview support ambient mesh. Để enable:

# Enable ambient mode (preview)
istioctl install --set profile=ambient

# Label namespace để use ambient
kubectl label namespace production istio.io/dataplane-mode=ambient

# Không cần istio-injection=enabled
# Không có istio-init container
# Không có istio-proxy container

Khi nào cân nhắc Ambient Mesh:

• AI/ML workloads với many Pods cần GPU nodes → save resource
• Batch processing với thousands of short-lived Jobs
• Cost-sensitive environments

Khi nào ở lại Sidecar Model:

• Production workloads cần stable, well-tested
• Cần full L7 features cho mọi Pod
• Team đã familiar với sidecar debugging

Sidecar Startup và Lifecycle

Cold Start Time

Pod startup timeline với sidecar:
  t=0ms: Pod spec assigned to node
  t=50ms: istio-init container start
  t=150ms: istio-init complete (iptables rules set)
  t=200ms: istio-proxy (Envoy) container start
  t=800ms: Envoy connect đến Istiod, receive xDS config
  t=1000ms: Envoy READY (passes health check)
  t=1100ms: Application container start (nếu holdApplicationUntilProxyStarts=true)
  t=2000ms: Application READY (passes readiness probe)
  
  Total overhead vs no-mesh: ~1000ms

Giảm cold start:

# Disable holdApplicationUntilProxyStarts nếu app có retry logic
annotations:
  proxy.istio.io/config: |
    holdApplicationUntilProxyStarts: false
    # App start ngay, Envoy catch up sau
    # OK nếu app có retry/backoff cho initial connections

Graceful Shutdown

# Đảm bảo Envoy không shutdown trước app
annotations:
  proxy.istio.io/config: |
    terminationDrainDuration: 5s
    # Envoy wait 5s trước khi shutdown
    # Cho phép in-flight requests hoàn thành

# Application container terminationGracePeriodSeconds phải > terminationDrainDuration
spec:
  terminationGracePeriodSeconds: 30

Shutdown sequence:

1. SIGTERM gửi đến Pod
2. Envoy set "draining" mode → không nhận new connections
3. Wait terminationDrainDuration (5s)
4. Drain in-flight requests
5. Application shutdown
6. Envoy shutdown

Production Sizing Checklist

# Example: Medium-traffic service, 500 RPS, HTTP/1.1
annotations:
  # 2 worker threads (500m CPU burst)
  proxy.istio.io/config: |
    concurrency: 2
  
  # Resource sizing
  sidecar.istio.io/proxyCPU: "100m"          # Request: normal idle
  sidecar.istio.io/proxyCPULimit: "1000m"     # Limit: allow burst
  sidecar.istio.io/proxyMemory: "128Mi"       # Request: normal usage
  sidecar.istio.io/proxyMemoryLimit: "512Mi"  # Limit: peak + buffer

# DestinationRule cho connection pool
trafficPolicy:
  connectionPool:
    tcp:
      maxConnections: 200        # 500 RPS × 0.05s × 2 (buffer) = 50, round up
    http:
      http1MaxPendingRequests: 100
      maxRequestsPerConnection: 100
  
  outlierDetection:
    consecutiveGatewayErrors: 5
    interval: 30s
    baseEjectionTime: 30s
    maxEjectionPercent: 50       # Never eject more than 50% of endpoints

Kết luận

Sidecar performance là real concern trong large-scale deployments:

1. CPU: Scale với RPS, mTLS crypto, telemetry
2. Memory: Scale với number of clusters, connections, xDS config size
3. Latency: 0.4ms P50, 1-5ms P99 per hop
4. Concurrency: Set explicitly để match CPU allocation

Key tuning levers:

• concurrency: Worker threads, match CPU limit
• Resource requests/limits: Generous limits để allow burst
• Connection pool: Size theo RPS × latency formula
• Telemetry: Reduce sampling và disable access log cho high-RPS services

Khi overhead quá lớn: cân nhắc Ambient Mesh (sidecarless) hoặc selective mesh (chỉ services cần L7 features).

References

• Envoy performance tuning
• Istio performance and scalability
• Sidecar resource configuration
• Ambient Mesh overview
• Cloud Service Mesh resource sizing
• Envoy concurrency model