Service Mesh Scalability: xDS, Sidecar Overhead, Endpoint Limits

Service Mesh Architecture at Scale

Service mesh (Istio, Google Cloud Service Mesh) injects sidecar proxy (Envoy) into every application Pod. Sidecar intercepts traffic:

Client Pod → Envoy sidecar (outbound) → Service IP → Envoy sidecar (inbound) → Server Pod

At 1000 nodes, 100K Pods:

• 100K Envoy sidecars
• Each sidecar maintains connections to every upstream service
• Central control plane (istiod) pushes configuration to all sidecars

Bottleneck: Control plane configuration push, sidecar memory, endpoint cardinality.

xDS Protocol: Configuration Push Model

xDS = extension Discovery Service (gRPC-based protocol). istiod pushes to sidecars:

• Cluster discovery (CDS): "Services available in cluster"
• Endpoint discovery (EDS): "Which Pods behind Service"
• Route discovery (RDS): "How to route traffic"
• Listener discovery (LDS): "Which ports listen"

Push model:

istiod watches Kubernetes services/endpoints
  ↓
Computes Envoy configuration
  ↓
Pushes to all connected Envoy sidecars (gRPC streaming)
  ↓
Envoy applies configuration

Push latency: ~100-500ms from service creation to Envoy having routing info.

Bottleneck 1: Endpoint Cardinality

Problem: Endpoint = one Pod backend. Service with 1000 replicas = 1000 endpoints.

EDS message size per service:

Endpoint entry = ~500 bytes (Pod IP, port, metadata)
1000 endpoints = 500KB per service
100 services = 50MB message

Memory per Envoy sidecar:

• Endpoints loaded into Envoy memory
• 50K endpoints (cluster-wide visible) × 500 bytes = 25MB per sidecar
• 100K sidecars × 25MB = 2.5TB (impossible!)

Solution: Scoping

Envoy doesn't need endpoints for all services. Scoping:

• Namespace scoping: Only see Services in same namespace
• Sidecar resource: Define which services visible to which Pods

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  namespace: default
spec:
  workloadSelector:
    labels:
      app: web
  egress:
  - hosts:
    - default/*      # only services in default namespace
    - mesh/*         # mesh-wide but filtered

Result: Sidecar sees ~100-1000 endpoints instead of 50K, memory = 50-500MB (acceptable).

Bottleneck 2: Control Plane Push Scalability

istiod (control plane) computes and pushes configuration. Push is not instant.

Timeline for service change:

kubectl apply service
  ↓ (100ms, API server)
etcd write
  ↓ (100ms, informer cache)
istiod detects change
  ↓ (100-500ms, compute new Envoy config)
istiod pushes to 100K Envoy sidecars
  ↓ (depends on connection rate)
Envoy receives and applies

Push bottleneck: istiod threads, network bandwidth, gRPC connection pool.

At scale:

• istiod typically single pod (or 2 replicas)
• 100K Envoy connections from sidecars
• If single istiod, connection pool = bottleneck
• Push time = 5-10 seconds for large service change

Solution: Multi-replica istiod

kubectl scale deployment istiod -n istio-system --replicas=5

Distributed istiod replicas share load. But must ensure Envoy sidecars distribute connections (not all to one replica).

Sidecar Memory Overhead

Envoy sidecar per Pod:

• Base memory: 20-30MB (just Envoy binary)
• Configuration memory: 10-100MB (depends on endpoint count, routes)
• Buffer memory: 50-100MB (traffic buffering, connection tracking)
• Total: 100-200MB per sidecar

At scale:

• 100K Pods × 150MB avg = 15TB! (clearly unsustainable)
• But: not all Pods need sidecar (batch jobs, non-service traffic)
• Realistic: 30-50K sidecars × 150MB = 4.5-7.5TB (still large, requires high-memory nodes)

Mitigation:

1. Selective sidecar injection:

   namespace-label: istio-injection=enabled  # inject only in labeled namespaces

2. Resource requests for sidecar:

   resources:
     requests:
       cpu: 100m
       memory: 128Mi    # ensure scheduler accounts for sidecar overhead

3. Optimize Envoy config:

   • Use VirtualService instead of multiple services
   • Reduce route complexity
   • Enable compression

Endpoint Limit: GKE Dataplane V2 Constraint

GKE Dataplane V2 (eBPF-based networking) has:

Max 260K endpoints across all services (hard limit).

Implication for Service Mesh:

• If you have 1000 services, average endpoints per service = 260K / 1000 = 260
• If one service has 500 replicas, it alone uses 500 endpoints → remaining 999 services split 259.5K
• Not practical for typical microservices scales

Workaround:

1. Federation: Split services across multiple clusters (100 services per cluster)
2. Aggregation: Group related Pods into fewer, larger services
3. External load balancing: For high-fanout (one service to many backends), use GCP Cloud Load Balancer instead of Kubernetes Service

Example:

# Bad: 1000 services, each 100 replicas = 100K endpoints
# Good: 100 services, each 1000 replicas = max 100K endpoints
# OR: 10 services with 10K replicas (if practical)

Real-World Scenario: Control Plane Upgrade at Scale

Case: Upgrade istiod from 1.20 to 1.21. Control plane in rolling upgrade mode.

Timeline:

1. Drain one istiod replica
2. Envoys reconnect to remaining replicas
3. Remaining replicas get additional load
4. Push latency increases (overloaded)
5. Service changes delayed 10+ seconds
6. Client timeouts if they expect fast configuration push
7. New istiod replica comes up, rebalance

Risk: Uncontrolled cascading delays.

Mitigation:

• Scale istiod replicas to 2x during upgrade
• Use pod disruption budgets (PDB)
• Monitor istiod CPU/memory, alert if saturated
• Have gradual rollout (10% Envoys per step)

Diagnosis: Service Mesh Bottleneck

Metrics to check:

istiod_total_configurations  # config objects computed
istiod_pushes_total          # number of pushes
envoy_stats_inuse_connections  # connection count
envoy_stats_memory           # sidecar memory usage
pilot_push_time_bucket       # push latency

Warning signs:

• istiod CPU >80%
• Push latency >5 seconds
• Envoy restart loop (OOMKilled)
• Service change not visible in Envoy for >10 seconds

Action:

• Scale istiod horizontally
• Reduce endpoint cardinality (namespace scoping, Sidecar resources)
• Review service count, consolidate if possible

References

• Istio Scalability
• xDS Protocol
• Google Cloud Service Mesh
• GKE Dataplane V2 Limits