etcd Scalability: Object Storage, Compaction, Spanner Migration

etcd's Role in Kubernetes Clusters

etcd là single source of truth cho mọi Kubernetes cluster state. Không etcd = cluster không tồn tại (configuration, Deployments, Secrets, etctera).

Scale impact:

• 1000 nodes, 100 Pods/node = 100K Pods
• • 50K Services
• • 200K ConfigMaps, Secrets, RBAC
• Total: ~350K objects trong etcd

Memory footprint: 350K objects × average 2KB per object = 700MB data + ~2-3GB overhead (indexes, write-ahead log) = 3-4GB etcd process memory.

Write rate: Pod churn, rolling updates, controller reconciliation = 1000-5000 writes/sec peak.

Ở thang độ này, etcd stability directly determines cluster stability.

Internal Model: etcd Architecture

Storage Engine

etcd sử dụng BoltDB (embedded key-value store sử dụng B+ tree). BoltDB characteristics:

• ACID transactions: Multi-key mutations atomic
• Versioning: Mỗi write increments revision number
• Range queries: efficient via B+ tree
• No sharding: Single replica process handles everything (Raft replication handles availability)

Data structure:

/kubernetes.io/pods/namespace/pod-name → {Pod object JSON}
/kubernetes.io/services/namespace/svc-name → {Service object JSON}
/kubernetes.io/nodes/node-name → {Node object JSON}

Raft Consensus

etcd uses Raft consensus (3-way replication typical):

Client write
  ↓
etcd leader receives request
  ↓
Leader appends to Raft log
  ↓
Leader replicates to followers (network RPC)
  ↓
Followers ACK
  ↓
Leader waits for quorum ACK (2 of 3 = majority)
  ↓
Leader applies to state machine (BoltDB)
  ↓
Response to client: "write committed"

Implication: Write latency = leader → followers RPC latency + disk fsync. Typically 10-50ms per write.

Compaction & Revision History

etcd maintains revision history — mỗi object change appends new version, previous versions kept.

Why? Client watches reconnect at previous revision:

watch := etcd.Watch(
  ctx,
  "/pods",
  client.WithRev(500),  // reconnect from revision 500
)

Watch buffer gets all changes since revision 500 up to latest.

Problem: Revision history grows unbounded. With 1000s writes/sec:

• 86400 seconds/day
• 5000 writes/sec × 86400 = 430M revisions/day
• Storage grows exponentially

Solution: Compaction

Compaction is deletions of old revisions.

compaction --revision 1000000000  # delete revisions < this

After compaction, etcd can't serve watch reconnect to deleted revisions → watches must restart from "latest" (slow).

Mechanics:

1. etcd leader marks old revisions for deletion
2. Compaction process (runs periodically) removes from BoltDB
3. Compact removes revision history, reclaims space

Default: GKE runs compaction every 30 minutes.

Bottleneck:

Compaction is blocking (leader stops accepting writes briefly):

• Acquires lock on database
• Scans BoltDB for old revisions
• Deletes entries (slow if many)
• Release lock

If compaction takes >1 minute, cluster experiences:

• API latency spike (writes blocked)
• Watch lag (events buffered, not flowing)
• Client timeouts

Compaction Strategy at Scale

Aggressive Compaction (Default)

--auto-compaction-mode=periodic
--auto-compaction-retention=1m  # keep last 1 minute revisions

Effect: Compact every compaction period (~1m).

Pros: Space efficient, etcd database doesn't bloat Cons: Frequent compaction = high CPU, write stalls, watch lag

Lenient Compaction

--auto-compaction-retention=30m  # keep last 30 minutes

Pros: Fewer compactions, clients can reconnect watches longer back Cons: etcd database grows, memory/disk pressure

Hybrid Strategy (Recommended for large clusters)

--auto-compaction-mode=revision  # compact by revision, not time
--auto-compaction-revision=1000000

Compact based on revision count (every N revisions), rather than time. More predictable.

Recommended setting for 1000+ node cluster:

--auto-compaction-retention=500000

Means: keep last 500K revisions (at 5000 writes/sec = ~100 seconds history). Sufficient for watch reconnect, minimal compaction overhead.

GKE's Spanner-based etcd (2025+)

Problem: etcd Scalability Ceiling

Traditional etcd:

• Single leader (write throughput capped)
• BoltDB (8GB size limit, configurable to 16GB)
• No sharding (all keys in one database)
• Compaction blocking (cluster stalls)

GKE hit ceiling with 65,000-node clusters (per blog post). Google Cloud team decided: replace etcd with Google's Spanner (distributed database used internally at Google scale).

Spanner as etcd Replacement

Characteristics:

• Distributed (sharded) database
• Strong consistency (still satisfies Raft-like guarantees)
• Automatic replication across regions
• Managed (no manual compaction)
• Virtually unlimited scaling

API compatibility:

• Spanner-etcd exposes same etcd v3 API
• Clients (API server, kubelets) unaware of change
• Transparent migration

Migration process (GKE-managed):

1. Create Spanner-backed etcd backend in parallel
2. Replicate state from legacy etcd
3. Switchover (atomic)
4. Decommission legacy etcd

Performance Characteristics

Spanner-based etcd shows:

• Lower latency variance (p99 latency more stable)
• Higher throughput (10,000+ writes/sec, no backoff)
• No compaction stalls (compaction doesn't block cluster)
• Automatic scaling (no manual capacity planning)

Operational Implications

For end-users (minimal impact):

• No manual intervention
• Cluster upgrades might stall briefly during switchover
• No API changes

For operators:

• Remove etcd monitoring (no longer visible) — GKE exposes equivalent metrics
• Stop tuning etcd flags — Spanner-backed etcd ignores many flags

Diagnosis: etcd Performance Issues

Metrics to Watch

From Prometheus (if cluster exported metrics):

etcd_server_has_leader  # is etcd healthy (1 = has leader)
etcd_server_leader_changes_seen_total  # leader churn (>1/hour = instability)
etcd_server_health_success  # health check success rate
etcd_commit_duration_seconds{quantile="0.99"}  # write latency
etcd_server_slowApplies_total  # high = disk I/O bottleneck

Warning Signs

1. etcd leader CPU spike:

   • Indicates write storm or compaction running
   • Check Pod churn rate (rolling updates, autoscaling)
   • Verify compaction isn't stuck

2. Watch lag (client observing stale data):

   • Means events not flowing from etcd to API server
   • Typically: API server watch buffer backlog
   • Check API server goroutine count, concurrent watches

3. Recurring leader re-elections:

   • etcd leader election cycle restarting
   • Indicates leader failing health check
   • Usually disk I/O (etcd WAL sync slow) or network partition

Remediation

If approaching limits:

1. Reduce object churn: Batch Pod creations, use StatefulSets (less rolling updates)
2. Delete unused objects: Pruning old Pods, Services, ConfigMaps
3. Shard cluster: Split into multiple clusters, aggregate at application layer
4. Upgrade to GKE with Spanner-etcd: (automatic with cluster upgrade)

Real-World Scenario: Debugging etcd Bloat

Case: Cluster with 500-node, 50K Pods, etcd using 8GB memory (limit hit).

Investigation:

1. Check top object types: etcdctl --endpoints=... endpoint health
2. Find culprit: likely ConfigMaps (10MB each, hundreds created during deployment)
3. Verify: kubectl get configmap --all-namespaces | wc -l

Solution:

• Delete unused ConfigMaps
• Switch to external config store (e.g., Cloud Secret Manager)
• Or: migrate cluster to Spanner-etcd (auto-scaling handles this)

References

• etcd Documentation
• Kubernetes etcd Concepts
• GKE Best Practices for Large Clusters
• How We Built a 130,000-Node GKE Cluster