My Awesome Project

Appearance

Network Policy Scalability: eBPF, Compilation, Cardinality ​

Network Policy at Large Scale ​

Network policy provides microsegmentation: control traffic between Pods based on labels.

yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector: {}
  ingress: []  # deny all incoming
  egress: []   # deny all outgoing

At 1000+ nodes:

  • 100K+ Pods, each subject to policies
  • 100+ NetworkPolicy objects
  • Egress rules to external networks (CIDR blocks)

Bottleneck: eBPF program generation, policy evaluation per packet.

eBPF-Based Enforcement (Dataplane V2) ​

Before eBPF (Legacy) ​

Network policy enforced via iptables rules on each node:

iptables -t filter -A FORWARD \
  -s 10.4.0.0/14 \    # Pod CIDR
  -d 10.4.0.0/14 \
  -j ACCEPT

Complexity: Manyiptables rules (1 rule per policy × selector complexity).

Performance: Linear scan through rules (O(rules)) per packet.

With eBPF (Dataplane V2) ​

Network policy compiled to eBPF bytecode (loaded into kernel):

c
// Simplified eBPF pseudocode
int policy_ingress(struct __sk_buff *skb) {
  struct iphdr *ip = load_header(skb);
  struct pod_id *pod = lookup_pod(ip->daddr);
  
  if (pod->labels[app] == "web") {
    if (allowed_sources.contains(ip->saddr)) {
      return ALLOW;
    }
  }
  return DROP;
}

eBPF program JIT-compiled to machine code → O(1) lookups (hash map).

Performance: ~100x faster than iptables.

eBPF Map Limits ​

Endpoint Count Limit (260K) ​

Dataplane V2 uses eBPF map (hash map in kernel):

Key: (protocol, service_ip, port, backend_pod_ip)
Value: (backend_pod_ip, backend_port)

Max entries: 260K across all services/endpoints.

Implication:

1000 services, avg 260 backends per service = 260K endpoints (max capacity)
If average 100 backends/service, can support 2600 services
If average 500 backends/service, can support 520 services

At large scale (100K Pods):

  • Few services with many backends (high fan-out) → consume limit fast
  • Many services with few backends (low fan-out) → fit within limit

Policy Rules Map Limit ​

NetworkPolicy rules also map-backed:

Key: (pod_ip, policy_name, rule_index)
Value: (action, target_selectors)

Limit: Also 260K-ish per cluster (shared with endpoint limit).

Policy Compilation Overhead ​

When new NetworkPolicy created/updated:

1. API server receives policy
2. etcd persists
3. Controller (Cilium operator) detects change
4. Compiles policy to eBPF bytecode
5. Loads bytecode into kernel
6. Programs attached to network interfaces

Compilation time: 100-500ms per policy (depends on rule complexity).

At scale (100+ policies, frequent changes):

  • Compile queue builds up
  • Policy update latency increases
  • Eventual consistency: new policy visible 1-2 seconds after apply

Example:

kubectl apply networkpolicy rule-1.yaml  # t=0
... (policy compiles, loads to eBPF) ... # t=100-500ms
Traffic matching rule-1 blocked (or allowed)  # t=500ms

Real-World Scenario: Policy Explosion ​

Case: 1000-node cluster, developing microservices. Team creates NetworkPolicy per service (100 services = 100 policies).

Topology:

service-a ↔ service-b ↔ service-c ↔ ... ↔ service-z

Each service has egress to 10 others (100 × 10 = 1000 rules).

Timeline:

  1. Create 100 NetworkPolicy objects (sequential apply)
  2. Each policy compiled: 1000 × 100ms = 100 seconds
  3. During compilation, traffic not fully enforced
  4. After 100 seconds: all policies loaded
  5. Traffic now microsegmented

Cost:

  • CPU on Dataplane (Cilium operator): 20% during compilation
  • Memory: policies + eBPF programs = 100-500MB
  • Latency: new policies visible after ~500ms

Optimization: Policy Aggregation ​

Instead of 1 policy per service:

yaml
# Before (100 policies)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: service-a-ingress
spec:
  podSelector:
    matchLabels:
      app: service-a
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: service-b
  - from:
    - podSelector:
        matchLabels:
          app: service-c
  # ... 8 more sources
yaml
# After (1 policy)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: core-services
spec:
  podSelector:
    matchLabels:
      tier: core
  ingress:
  - from:
    - podSelector:
        matchLabels:
          tier: core

Result: Fewer policies → fewer compilation cycles → less overhead.

Egress Policy Complexity ​

Egress to external (non-cluster) networks adds complexity:

yaml
spec:
  egress:
  - to:
    - namespaceSelector: {}
    - podSelector: {}
    ports:
    - protocol: TCP
      port: 443
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0  # any IP
        except:
        - 169.254.169.254/32  # block metadata service
    ports:
    - protocol: TCP
      port: 443

Compilation: Multiple CIDR blocks = multiple eBPF rules (O(CIDR count)).

Best practice: Aggregate CIDR blocks, use fewest rules.

Diagnosis: Network Policy Bottleneck ​

Metrics:

cilium_policy_regeneration_total  # policy compile count
cilium_policy_regeneration_time_bucket  # compile latency

Warning signs:

  • Policy compile latency >1 second
  • Cilium operator CPU saturated
  • New policies not visible 10+ seconds after apply

Remediation:

  1. Simplify policies (fewer rules, aggregate selectors)
  2. Batch policy updates (apply multiple policies, single compile cycle)
  3. Scale Cilium operator (more replicas)
  4. Review policy necessity (might not need all policies)

References ​