My Awesome Project

Appearance

Logging at Scale: Volume Management, Sampling, Exclusion ​

Log Volume at Scale ​

Baseline: Each containerized Pod outputs logs (stdout/stderr).

At 1000 nodes, 100K Pods:

  • Average log rate: 10 KB/second per Pod (typical web service)
  • Total: 100K × 10 KB/s = 1 GB/s log volume
  • Per day: 1 GB/s × 86,400s = ~86 TB logs/day

Cost in Cloud Logging:

  • Ingestion: $0.50 per GB (first 50GB free)
  • Storage: $0.01 per GB/month
  • 86 TB/day × $0.50 = $43K/day ≈ $1.3M/month ingestion cost alone

Clearly unsustainable. Most organizations drop 90%+ of logs.

Log Exclusion Strategies ​

Strategy 1: Exclude Noisy Services ​

yaml
# CloudLoggingConfig (GKE)
apiVersion: logging.cnpg.io/v1beta1
kind: LoggingConfig
metadata:
  name: logging-config
spec:
  defaultExclusionFilter:
    - resource.labels.pod_name: "prometheus-.*"  # exclude Prometheus Pods
    - resource.labels.namespace_name: "kube-system"  # exclude system logs
    - severity: "DEBUG"  # exclude debug logs

Effect:

  • Prometheus Pods (Prometheus, Alertmanager, node-exporter): 1000 Pods, 100 KB/s each = 100 MB/s gone
  • kube-system namespace (system daemons): ~1000 Pods = 100 MB/s gone
  • Debug logs: ~20-30% of total
  • Total removed: ~300-500 MB/s = 25-40% volume reduction

Strategy 2: Namespace-Level Exclusion ​

yaml
# Only log production namespace, exclude dev/test
excludeNamespaces:
  - kube-system
  - kube-public
  - dev-*
  - staging-*
includeNamespaces:
  - production
  - monitoring

Trade-off:

  • Pro: 50%+ volume reduction (drop dev/staging logs)
  • Con: No logs from dev cluster (harder to debug dev issues)

Strategy 3: Severity Filtering ​

yaml
# Only log ERROR and above
severityFilter: [ERROR, CRITICAL]

Effect: Drop INFO, WARNING logs (typically 80-90% of volume).

Trade-off:

  • Pro: 80%+ volume reduction
  • Con: Miss context for warnings (leads to harder debugging)

Better approach: Log INFO, but drop repetitive messages.

Sampling Strategies ​

Instead of dropping all logs from source, sample them (keep 1 in N).

Probability Sampling ​

yaml
samplingConfig:
  initialRate: 0.1  # sample 10% initially
  finalRate: 0.01   # scale down to 1% after 1 million logs
  samplingSize: 1000000

Behavior:

  • Sample 100% logs until 1M logs collected
  • Then sample 10%, until 100M logs → 1%
  • Gradually reduce as volume grows

Effect: Early logs fully captured, later logs sparse.

Systematic Sampling ​

yaml
samplingConfig:
  rate: 100  # keep 1 in every 100 logs

Behavior:

  • Discard log 1-99, keep log 100, discard 101-199, keep 200, etc.
  • Uniform distribution across time

Pro: Predictable sampling (every Nth log kept) Con: Might miss critical events in between

Field-Level Exclusion ​

Instead of dropping entire logs, drop specific fields:

yaml
fieldExclusion:
  - field: jsonPayload.user_id
  - field: jsonPayload.request_body  # might contain PII
  - field: labels.trace_id

Effect: Keep logs, but redact sensitive fields.

Use case: Compliance (PCI-DSS, GDPR) requires not logging PII.

Log Aggregation & Pre-Processing ​

Instead of sending raw logs, aggregate/sample at source (before Cloud Logging ingestion).

Fluent Bit (Log Forwarder) ​

Deploy Fluent Bit DaemonSet per node:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [SERVICE]
      Flush         5
      Daemon        off
      Log_Level     info
    
    [INPUT]
      Name              tail
      Path              /var/log/containers/*.log
      
    [FILTER]
      Name    modify
      Match   *
      Remove  log_id     # drop high-cardinality field
      Remove  span_id
    
    [FILTER]
      Name    sampling
      Match   *
      Sample  100  # keep 1 in 100 logs
      
    [OUTPUT]
      Name stackdriver
      Match *

Effect:

  • Each node's Fluent Bit instance samples logs
  • Pre-processes before sending to Cloud Logging
  • 90%+ volume reduction

Cost:

  • Fluent Bit CPU overhead: 50m per node = 50 cores for 1000 nodes
  • Acceptable trade-off (saves $1M+ per month in logging costs)

Real-World Scenario: Log Volume Explosion During Incident ​

Case: Application bug causes infinite loop logging.

Before mitigation:

  • Log rate suddenly spikes from 10 KB/s to 10 MB/s per affected Pod
  • 100 affected Pods × 10 MB/s = 1 GB/s (was 100 MB/s baseline)
  • Cloud Logging ingestion bill jumps $50K in 1 hour
  • Alert: cardinality spike, but incident ongoing

With exclusions/sampling:

  • Bug still spikes log volume
  • But sampled at 1% → 10 MB/s total (manageable)
  • Logs still flow to Cloud Logging, can diagnose bug
  • Cost spike reduced

Resolution:

  • Find root cause (bug in logging statement)
  • Increase severity filter (ERROR+ only)
  • Restart Pods to clear log buffer
  • Cost impact: $5K (instead of $50K) per hour

Monitoring Log Volume Health ​

Key metrics:

cloud.googleapis.com/logging/user/log_bytes_ingested  # total bytes ingested
cloud.googleapis.com/logging/user/log_entry_count      # number of log entries

Alert conditions:

# Alert if log volume increases 10x
increase(log_bytes_ingested[1h]) > 10 * avg_log_bytes

Log Retention & Cost ​

Cloud Logging default retention: 30 days (can adjust).

Retention cost: $0.01 per GB per month (storage).

Example (1M logs/day = 100 GB/day):

  • 30-day retention = 3TB stored
  • Monthly storage cost: 3000 × $0.01 = $30

Optimization: Reduce retention to 7 days (faster troubleshooting, lower cost).

References ​