Metrics Cardinality Management: Explosion, Prevention, Cost Control

Cardinality Basics

Cardinality = number of unique time series. Each unique combination of metric + labels = 1 series.

Example:

kube_pod_status_phase{pod="pod-1", namespace="ns-1", phase="Running"}
kube_pod_status_phase{pod="pod-2", namespace="ns-1", phase="Running"}
kube_pod_status_phase{pod="pod-1", namespace="ns-2", phase="Running"}
= 3 unique time series

At 1000+ nodes, 100K Pods:

kube_pod_status_phase metric:
  100K Pods × 5 phases (Pending, Running, Succeeded, Failed, Unknown)
  = 500K time series

kube_pod_labels metric:
  100K Pods × avg 10 custom labels
  = 1M time series

Total: 500K + 1M + others = 2M+ time series

Cost: Cloud Monitoring charges per sample ingested (~$0.25 per million samples).

2M time series × 60 scrapes/hour × 24 hours = 2.88 billion samples/day ≈ $72/day ≈ $2160/month.

Cardinality Explosion Sources

1. High-Cardinality Labels (Anti-Pattern)

kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceMonitorSelector: {}
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-metrics
spec:
  endpoints:
  - port: metrics

If application exposes metrics like:

http_request_duration_seconds_bucket{user_id="alice", request_id="req-123", ...}

Each unique user_id and request_id = new series. With 100K users, = 100K+ series per metric.

Red flag: Metrics with unbounded label values (user IDs, request IDs, trace IDs).

2. Implicit Cardinality (Container Metrics)

Kubelet exports per-container metrics:

container_cpu_usage_seconds_total{pod="pod-1", container="app", namespace="ns"}

100K Pods × 2 containers per Pod = 200K series just for CPU.

Multiply by 10 metrics (cpu, memory, network, disk) = 2M series.

3. Label Explosion (kube-state-metrics)

kube-state-metrics exposes Kubernetes object state:

kube_pod_info{pod="pod-1", namespace="ns", node="node-1", pod_ip="10.4.0.1", ...}

Each label combination = series. With many labels, = many series.

100K Pods × 20 labels = 2M series.

Cardinality Limit & Enforcement

Cloud Monitoring sets cardinality hard limits:

• Default: 100K series per resource (GKE cluster)
• Adjustable: up to 1M+ (contact support, additional cost)

When hitting limit:

• Prometheus scrapes rejected (HTTP 400)
• Metrics dropped (observability gap)
• Alerts might miss (metric not ingested)

Impact: During cardinality spike, metrics silently dropped → blind to real issues.

Scrape Filtering Strategy

Strategy 1: Metric Relabeling (Drop High-Cardinality)

scrape_configs:
- job_name: kubernetes-pods
  metric_relabel_configs:
  # Drop user_id label (unbounded)
  - source_labels: [__name__]
    regex: '.*_user_id.*'
    action: drop
  # Keep only specific metrics
  - source_labels: [__name__]
    regex: 'http_requests_total|http_duration_seconds'
    action: keep

Effect:

• Drop metrics matching pattern
• Keep only allowlisted metrics

Example (before/after):

Before: 10M series
After:  100K series (10K series for kept metrics)
Cost: $75/month → $2.40/month

Strategy 2: Instance Filtering (Sample n of Targets)

scrape_configs:
- job_name: kubelet
  scrape_interval: 30s
  metric_relabel_configs:
  # Sample 10% of Pods
  - source_labels: [__address__]
    regex: '10\.4\.([0-9]+)\.([0-9]+):.*'
    action: keep
    # Keep only if last octet divisible by 10

Effect: Scrape subset of targets (e.g., 10% of Pods).

Trade-off:

• Lower cardinality (10% of series)
• Less detailed observability (might miss anomalies in unsampled Pods)

When to use: Metrics used for aggregated SLOs (cluster-level, not Pod-level drill-down).

Strategy 3: Aggregation Rules (Pre-aggregation)

Instead of storing individual Pod metrics:

# Expensive: per-Pod memory usage
container_memory_bytes{pod="pod-1"}
container_memory_bytes{pod="pod-2"}
...
container_memory_bytes{pod="pod-100000"}
= 100K series

Pre-aggregate to per-node:

groups:
- name: aggregation
  rules:
  - record: node:container_memory_bytes:sum
    expr: sum by (node) (container_memory_bytes)

Effect:

• 100K series → 1K series (1 per node)
• Alert on node-level, not Pod-level

Trade-off: Can't drill down to Pod-level (lost detail).

High-Cardinality Pattern Detection

Prometheus Cardinality Analysis

In Cloud Monitoring, Metrics Management page shows:

Highest cardinality metrics:
  kube_pod_labels: 2.1M samples/minute
  container_cpu_usage: 1.5M samples/minute
  node_cpu_seconds: 800K samples/minute

Action: Focus on top 10% (highest cardinality).

Local Prometheus Debugging

# Connect to Prometheus
kubectl port-forward -n prometheus svc/prometheus 9090:9090

# Query cardinality
curl -s 'http://localhost:9090/api/v1/label/__name__/values' | jq '. | length'
# Returns: number of unique metrics

# Top 10 metrics by cardinality (simplified)
curl -s 'http://localhost:9090/api/v1/query?query=count({__name__=~".+"}) by (__name__)' | \
  jq '.data.result | sort_by(.value | tonumber) | reverse | .[0:10]'

Real-World Scenario: Cardinality Explosion During Deployment

Case: Deploy 10K Pods (batch job). Metrics cardinality was 500K.

Timeline:

1. Before deploy: 500K series
2. Deploy 10K Pods
3. Each Pod creates:
   • kube_pod_info series
   • container_cpu_usage series
   • container_memory_usage series
   • network metrics = ~5 series per Pod
4. 10K × 5 = 50K new series
5. Total: 550K series (ok, still under 1M limit)

BUT: If Pods have custom labels (user_id, trace_id):

kubernetes.io/user_id="user-123"
kubernetes.io/trace_id="trace-xyz"

And metrics included these labels:

• 10K Pods × 10K users (worst case) = 100M series (EXPLODES)
• Cloud Monitoring ingestion fails
• Metrics lost

Resolution:

• Scrape filter: drop unbounded labels
• Pre-aggregation: group by node instead of Pod
• Cardinality alert: trigger when approaching limit

Monitoring Cardinality Health

Alert conditions:

# Alert if approaching cardinality limit
count(ALERTS{severity="warning", metric_cardinality_high=""}) > 0

# Alert if ingestion errors spike (dropped metrics)
increase(prometheus_tsdb_symbol_table_size_bytes[5m]) > 0.9 * max_cardinality

References

• Cloud Monitoring Metrics Management
• Prometheus Cardinality
• GKE Observability