My Awesome Project

Appearance

NodeLocal DNSCache: DNS Performance at Large Scale ​

DNS Bottleneck in Kubernetes Clusters ​

Without NodeLocal DNSCache:

Pod → CoreDNS (kube-dns service) → Upstream DNS (Cloud DNS)

At 1000 nodes, 100K Pods:

  • Each Pod might resolve service names multiple times (app startup, health checks, periodic lookups)
  • CoreDNS replicas (typically 2-4) must handle all cluster DNS requests
  • Network overhead: Pod → CoreDNS always requires network traversal (even if on same node)

Scale calculation:

  • 100K Pods, each 5 DNS requests per minute = 500K requests/minute
  • CoreDNS throughput: ~5,000 qps (queries per second) = 300K requests/minute
  • Overload: 200K excess requests/minute

Result:

  • CoreDNS CPU saturates
  • DNS queries timeout
  • Pods fail to resolve service names
  • App startup delayed or fails

Why DNS Matters at Scale ​

DNS lookup latency impacts:

  1. Pod startup: First DNS query to resolve service → critical path
  2. Health checks: Periodic HTTP health checks to services (DNS resolution per check)
  3. Observability: Metrics scraping involves DNS resolution of targets
  4. Outbound connections: Client connects to service name → DNS lookup first

Even 100ms extra per DNS query × 500K queries/minute = 8+ hour of cumulative delay/day.

NodeLocal DNSCache Solution ​

Architecture ​

Deploy CoreDNS as DaemonSet on every node, listening on node IP (not via Service).

Pod → localhost:53 (CoreDNS on same node)
         ↓ (cache hit?)
         Return immediately
         ↓ (cache miss)
         Query CoreDNS service (Cloud DNS)

Benefits:

  • DNS queries never leave node (no network hop)
  • Cache hit rate = 80-90% typical (same services queried repeatedly)
  • Reduces load on central CoreDNS by 10x

Deployment ​

Enable via GKE cluster creation:

bash
gcloud container clusters create mycluster \
  --enable-dns-cache

Or enable on existing cluster (beta):

bash
gcloud container clusters update mycluster \
  --enable-dns-cache

GKE automatically:

  1. Deploys CoreDNS DaemonSet (node-local-dns) on every node
  2. Patches kubelet configuration (--cluster-dns=169.254.20.10)
  3. Configures localhost DNS stub

Caching Mechanism ​

NodeLocal CoreDNS caches with TTL:

Service SRV record: _http._tcp.svc.cluster.local → 30s TTL (default)
A record (service IP): nginx-svc.default.svc.cluster.local → 30s TTL
External domain: example.com → 300s TTL

Cache effectiveness:

  • Service queries: 90%+ hit rate (same services accessed repeatedly)
  • External queries: 70-80% hit rate (depends on domain diversity)

When NodeLocal DNSCache is Mandatory ​

Scenario 1: High Pod Density (>80 Pods/node) ​

At 80+ Pods/node, central CoreDNS can't keep up. NodeLocal reduces load by 80%+.

Scenario 2: Frequent Service Name Resolution ​

Apps doing many DNS lookups (service discovery, health checks, client retries).

Scenario 3: Network Path Bottleneck ​

Pod and CoreDNS on different node. Network bandwidth constrained → DNS over network costly.

Scenario 4: Latency-Sensitive Apps ​

Apps where DNS lookup latency contributes to SLO (edge cases, but real).

Scenario 5: CoreDNS Already Saturated ​

Monitor CoreDNS metrics: if CPU >60%, latency >100ms, enable NodeLocal.

Operational Considerations ​

CPU Overhead per Node ​

NodeLocal CoreDNS DaemonSet:

  • Base memory: 20-30MB per node
  • CPU: 10-50m under normal load

Cluster-wide: 1000 nodes × 40m = 40 CPU cores just for NodeLocal DNS.

Worth it? Yes, because reduces central CoreDNS need from 4 replicas to 1-2 → net saving.

Cache Invalidation ​

Cache invalidated when:

  1. TTL expires: Default 30s for service records
  2. Explicit invalidation: (rare) etcd watch sees change
  3. Pod restart: Cache persists (no state in Pod)

Risk: If Service endpoint changes, cache might serve stale IP for up to 30s.

For critical services, consider lower TTL:

yaml
# In CoreDNS ConfigMap
cache 10  # cache 10 seconds instead of 30

Monitoring NodeLocal DNS ​

bash
# Check if enabled
kubectl -n kube-system get daemonset node-local-dns

# Check metrics (if Prometheus enabled)
kubectl top pod -n kube-system node-local-dns-xxxx

Metrics:

  • coredns_cache_hits_total: cache hits
  • coredns_cache_misses_total: cache misses
  • coredns_dns_request_duration_seconds_bucket: query latency

Target hit rate >75%.

Real-World Scenario: DNS Saturation During Surge ​

Case: Cluster autoscale from 100 to 1000 nodes (scaling event). New workload lands.

Timeline (without NodeLocal):

  1. New 900 nodes come online
  2. ~9000 new Pods spawn (10 Pods/node initially)
  3. All Pods initialize, resolve service names → 9000 concurrent DNS queries
  4. Central CoreDNS (2 replicas) flooded
  5. DNS timeout → Pod init fails
  6. Pods restart → more DNS queries → worse
  7. Cascading failure

With NodeLocal:

  1. Same scenario
  2. Each node has local DNS → 9000 queries handled locally (cache misses rare)
  3. Local CoreDNS serves from cache (80%+ hit rate)
  4. Central CoreDNS only sees 20% misses = 1800 queries → manageable
  5. Pod initialization succeeds

Disabling NodeLocal DNSCache (If Needed) ​

Reasons to disable (rare):

  • Custom DNS resolution needed (security reasons, special firewall rules)
  • Debugging DNS issues (need central logging)

Disable:

bash
gcloud container clusters update mycluster \
  --no-enable-dns-cache

References ​