Error Reporting - Grouping, Fingerprinting, và Notifications

Tại sao error reporting không phải chỉ logging

Khi hệ thống xử lý 1 triệu request/giây, thậm chí 0.01% error rate = 100 errors/giây.

Nếu bạn log tất cả:

8.6 triệu errors/ngày
Log storage: terabytes/năm
Alert: 100 notifications/giây (khiến ops team bị overwhelmed)

Error Reporting giải quyết bằng cách:

Group similar errors - "These 10,000 errors là cùng một issue"
Calculate metrics - "Error rate 100/sec, 50K unique users affected, appeared first ở 10:30 AM"
Smart notifications - "Alert only when new error appears hoặc rate spike"

Error Fingerprinting - Group Similar Errors

The Fingerprint Concept

Fingerprint là một identifier cho "error group":

Error 1: "TypeError: Cannot read property 'id' of undefined" at line 42
Error 2: "TypeError: Cannot read property 'id' of undefined" at line 42
Error 3: "TypeError: Cannot read property 'id' of undefined" at line 42
...
Error 10000: "TypeError: Cannot read property 'id' of undefined" at line 42

Fingerprint: "1a2b3c4d5e6f7a8b" ← Same fingerprint, all grouped

Fingerprint Generation

GCP Error Reporting calculates fingerprint based on:

Error type - Exception class
Stack trace - Call stack at error
Application/service - Which service errored
Resource type - App Engine, Cloud Functions, GKE, etc

Algorithm (simplified):

python

def calculate_fingerprint(error):
    # Extract key parts
    error_type = error.exception_type
    
    # Top 3 functions in stack trace
    stack_summary = [
        error.stack_trace[0].function,
        error.stack_trace[1].function,
        error.stack_trace[2].function,
    ]
    
    service_name = error.service
    
    # Combine into fingerprint
    input_string = f"{error_type}:{stack_summary}:{service_name}"
    fingerprint = hash(input_string) % (2^64)  # 64-bit hash
    
    return fingerprint

Example:

Error A:
  Type: ValueError
  Stack: api.py:handle_request() → model.py:validate() → utils.py:parse_json()
  Service: user-service
  Fingerprint = hash("ValueError:handle_request:validate:parse_json:user-service")

Error B:
  Type: ValueError
  Stack: api.py:handle_request() → model.py:validate() → utils.py:parse_json()
  Service: user-service
  Fingerprint = hash("ValueError:handle_request:validate:parse_json:user-service")
  
→ Same fingerprint! Grouped together.

Error C:
  Type: ValueError
  Stack: api.py:handle_request() → model.py:validate() → utils.py:parse_int()  ← Different stack!
  Service: user-service
  Fingerprint = hash("ValueError:handle_request:validate:parse_int:user-service")
  
→ Different fingerprint! Separate error group.

Fingerprint Stability

Fingerprint should be stable - same error always gets same fingerprint.

But what if code changes?

OLD code:
  def validate():
      utils.parse_json()

Error fingerprint: "ValueError:handle_request:validate:parse_json"

NEW code (refactored):
  def validate():
      parser.parse_json()  ← function name changed

Error fingerprint: "ValueError:handle_request:validate:parse_json" ← SAME
(Because GCP only looks at TOP-level functions, not intermediate callers)

Stability considerations:

Minor refactoring (rename internal functions) → fingerprint stable ✓
Major refactor (move error location) → fingerprint changes ✗

When fingerprint changes:

Old group and new group are separate
You see duplicate "new error" notification
User needs to manually correlate (annoying)

Best practice: Document why error is logged, use meaningful error messages.

Error Group Management

Error Group Lifecycle

NEW ERROR DETECTED
    ↓
[Open] (default state)
    ↓
Dev triages → [Acknowledged]
    ↓
Dev works on fix
    ↓
Deploy fix
    ↓
[Resolved] (manually marked)
    ↓
If error appears again → Auto-revert to [Open]
    ↓
[Muted] (ignore this error group)

Muting Errors (False Positives)

Some errors are expected:

User mistypes password → "Authentication failed" (expected)
Network timeout from external API → expected
Rate limit hit on free tier → expected

python

# Error Reporting configuration
ERROR_IGNORE_LIST = [
    "AuthenticationFailedError",  # Expected, clients mistype
    "ExternalApiTimeout",          # External service unreliable
    "QuotaExceededException",      # Expected for free tier
]

def report_error(exception):
    if exception.type in ERROR_IGNORE_LIST:
        return  # Don't report
    
    cloud_error_reporting.report(exception)

Benefits:

Cleaner error dashboard
Fewer false positive alerts
Focus on real issues

Error Group Queries

Query error groups:

python

from google.cloud import error_reporting_v1beta1

client = error_reporting_v1beta1.ErrorGroupServiceClient()

# List all error groups
error_groups = client.list_error_groups(
    name=f"projects/my-project"
)

# Filter by time
import datetime
now = datetime.datetime.now(datetime.timezone.utc)
one_hour_ago = now - datetime.timedelta(hours=1)

recent_errors = [
    g for g in error_groups
    if g.last_seen_time >= one_hour_ago
]

Affected User Detection

Metric: Affected Users

Error Event 1: user_id=123, trace_id=abc, time=10:30:00
Error Event 2: user_id=123, trace_id=def, time=10:30:05 ← Same user!
Error Event 3: user_id=456, trace_id=ghi, time=10:30:10 ← Different user

Error Group:
- Frequency: 3 errors
- Affected users: 2 (users 123 and 456)

How GCP determines affected users:

Extract user_id from error metadata (if available)
Track unique user IDs in error group
Report "X unique users affected"

Extracting User ID from Error

Must manually pass user context:

python

from google.cloud import error_reporting

def handle_request(user_id):
    try:
        process_order()
    except Exception as e:
        # Report error with user context
        client = error_reporting.Client()
        client.report_exception(
            exception=e,
            labels={
                "user_id": str(user_id),
                "request_id": get_request_id(),
            }
        )

In Error Reporting dashboard:

Error Group: "ValueError: Invalid order"
- Frequency: 100
- Affected users: 47
- Request IDs: req-123, req-456, req-789, ...

Use Cases for Affected Users

Priority calculation:

python

def should_alert(error_group):
    # Alert if:
    # - Affecting > 100 users, OR
    # - Affecting VIP customer tier, OR
    # - Error rate > 10/minute
    
    if error_group.affected_user_count > 100:
        return True
    
    if any(user.tier == "enterprise" for user in error_group.affected_users):
        return True
    
    if error_group.frequency > 10:
        return True
    
    return False

Error Events & Sampling

Data Retention

Cloud Error Reporting keeps only 1,000 samples per error group:

Error Group "ValueError in process_order":
- Total occurrences: 50,000
- Stored samples: 1,000 (2% of total)
- Lost: 49,000

This is intentional (cost reduction). When 50,000 errors identical, storing all is wasteful.

Accessing Full Error History

To query beyond 1,000 samples, use Cloud Logging (Cloud Logs Explorer):

sql

SELECT
  jsonPayload.error_message,
  labels.user_id,
  severity,
  timestamp
FROM `projects.DATASET.logs`
WHERE jsonPayload.error_type = "ValueError"
  AND jsonPayload.function = "process_order"
  AND timestamp > TIMESTAMP_SUB(NOW(), INTERVAL 7 DAY)

This gives you full history (depends on log retention setting, typically 30 days).

Error Deduplication

Error Reporting has internal deduplication to prevent duplicate reporting:

Error A reported at t=10:30:00.100
Error B (same fingerprint) reported at t=10:30:00.105 (5ms later)

Deduplication window: 5 seconds

→ Error B deduplicated, not counted twice
→ Frequency shows: 1 (not 2)

Deduplication helps prevent inflated frequency counts.

Error Notifications & Alerts

Notification Types

Notification Type	When Triggered	Example
New error	First time fingerprint seen	"New: ValueError in process_order"
Recurring error	Error seen again after resolved	"Recurring: ValueError in process_order"
Error spike	Frequency increases suddenly	"Spike: ValueError errors up from 1/min to 50/min"

Notification Configuration

python

from google.cloud import error_reporting_v1beta1

client = error_reporting_v1beta1.ErrorGroupServiceClient()

# Get error group
error_group = client.get_group(name="projects/my-project/groups/123")

# Update to enable notifications
error_group.tracking.update_time = time.time()

client.update_group(group=error_group)

Or via GCP console:

Go to Error Reporting dashboard
Click on error group
Click "Enable notifications"

Notification Channels

Error Reporting can notify via:

Channel	Format
Email	Sent to team email
Slack	Posted in #alerts channel
PagerDuty	Create incident
Cloud Pub/Sub	Custom integration
Webhooks	Custom webhook endpoint

Example: Slack Notification

python

from google.cloud import monitoring_v3

client = monitoring_v3.NotificationChannelServiceClient()

channel = monitoring_v3.NotificationChannel(
    type_="slack",
    display_name="alerts-channel",
    labels={
        "channel_name": "#alerts",
    },
    enabled=True,
)

created_channel = client.create_notification_channel(
    name="projects/my-project",
    notification_channel=channel,
)

Integration with Issue Trackers

Linking Error Groups to Issues

Error Group: "ValueError in process_order"
├── Issue Tracker Link: https://github.com/myapp/issues/1234
├── Status: In Progress
└── Assignee: alice@example.com

Manual Linking

In Error Reporting console:

Click error group
Click "Link to issue"
Paste issue URL

Programmatic Linking

python

from google.cloud import error_reporting_v1beta1

client = error_reporting_v1beta1.ErrorGroupServiceClient()

error_group = client.get_group(name="projects/my-project/groups/123")

# Update with issue link
error_group.tracking.links = [
    {"url": "https://github.com/myapp/issues/1234"}
]

client.update_group(group=error_group)

Benefits:

Team sees error context directly in GitHub/Jira
GitHub sees error spike when pull request ships
Helps track "is this bug fixed?"

Error Report Payload

What gets sent to Error Reporting

When you report error:

python

client = error_reporting.Client()

try:
    risky_operation()
except Exception as e:
    client.report_exception(
        exception=e,  # Automatically captures stack trace
        labels={
            "user_id": "123",
            "feature_flag": "new_checkout",
        }
    )

Payload sent to GCP:

json

{
  "service_context": {
    "service": "user-service",
    "version": "v1.2.3"
  },
  "timestamp": "2026-06-24T10:30:00.000Z",
  "error_context": {
    "http_request": {
      "url": "https://api.example.com/orders/123",
      "method": "POST",
      "status_code": 500
    },
    "user_id": "123",
    "resource_type": "gke_container"
  },
  "exception_type": "ValueError",
  "stack_trace": [
    {
      "file_name": "order.py",
      "line_number": 42,
      "function_name": "process_order"
    }
  ]
}

Automatic Field Capture

GCP automatically captures (when running on GCP services):

HTTP request details (URL, method, status)
Resource context (service name, version, pod name)
Timestamp
Environment (staging vs production)

Optional manual fields:

User ID
Request ID
Custom labels

Production Patterns

Pattern 1: Error Budget Alerting

Track error budget (SLO-based):

python

def calculate_error_budget_remaining():
    """Return % of error budget remaining"""
    slo_error_rate = 0.001  # 0.1% error rate SLO
    
    actual_error_rate = get_error_rate_1h()
    
    if actual_error_rate > slo_error_rate:
        budget_remaining = 0
    else:
        budget_consumed = actual_error_rate / slo_error_rate
        budget_remaining = (1 - budget_consumed) * 100
    
    return budget_remaining

# Alert if budget low
budget = calculate_error_budget_remaining()
if budget < 10:
    notify(f"ERROR: Only {budget}% error budget remaining!")

Pattern 2: Gradual Rollout with Error Monitoring

Deploy new version with error monitoring:

python

# Deploy version 2 to 10% traffic
# Monitor error rate vs version 1

def should_increase_rollout():
    error_rate_v1 = get_error_rate("version-1")
    error_rate_v2 = get_error_rate("version-2")
    
    # If v2 error rate is worse, rollback
    if error_rate_v2 > error_rate_v1 * 1.1:  # 10% worse
        return False
    
    # Otherwise, increase traffic
    return True

# In deployment automation:
while traffic_v2 < 100:
    if should_increase_rollout():
        traffic_v2 += 10
    else:
        rollback_to_v1()
        break

Pattern 3: Error Rate SLO Tracking

Track if error rate within SLO:

python

def track_error_slo():
    """Track if error rate is within SLO"""
    slo_error_rate = 0.001  # 0.1%
    window = 5 * 60  # 5 minutes
    
    error_rate = get_error_rate(window)
    
    if error_rate > slo_error_rate:
        metric.record(
            "error_slo_violation",
            labels={"severity": "high"},
            value=1
        )
        alert(f"Error rate {error_rate} exceeds SLO {slo_error_rate}")
    else:
        metric.record(
            "error_slo_ok",
            labels={"severity": "normal"},
            value=1
        )

Constraints & Limitations

Limitation 1: Fingerprinting Edge Cases

Fingerprinting can group unrelated errors:

Error A: ValueError at process_order:42
Error B: ValueError at process_order:42 (but different root cause)

Same fingerprint → Grouped together, even though different cause!

Mitigation: Use meaningful error messages.

python

# BAD
raise ValueError("Invalid value")

# GOOD
raise ValueError(f"Invalid order amount: {amount} (expected > 0)")

More specific error messages → better fingerprinting.

Limitation 2: Stack Trace Variation

If stack trace slightly varies, fingerprint might change:

Scenario 1:
  Exception raised at:
  api.py:42 → model.py:100 → utils.py:50

Scenario 2:
  Exception raised at:
  api.py:42 → model.py:110 → utils.py:50  ← Different line in model.py

Same error logically, but GCP might assign different fingerprint (depends on stack depth used).

Limitation 3: Privacy - Sensitive Data in Stack Traces

Stack traces might contain sensitive data:

python

def process_payment(card_number, cvv):
    try:
        validate_card(card_number)
    except ValueError as e:
        raise ValueError(f"Card validation failed: {card_number}")  # ← LEAK!

Stack trace in Error Reporting:

ValueError: Card validation failed: 4532123456789012

Credit card number exposed!

Mitigation: Scrub sensitive data before logging.

python

def sanitize_error_message(msg):
    # Remove credit card patterns
    msg = re.sub(r"\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}", "****", msg)
    # Remove SSN patterns
    msg = re.sub(r"\d{3}-\d{2}-\d{4}", "***", msg)
    return msg

Error Reporting - Grouping, Fingerprinting, và Notifications ​

Tại sao error reporting không phải chỉ logging ​

Error Fingerprinting - Group Similar Errors ​

The Fingerprint Concept ​

Fingerprint Generation ​

Fingerprint Stability ​

Error Group Management ​

Error Group Lifecycle ​

Muting Errors (False Positives) ​

Error Group Queries ​

Affected User Detection ​

Metric: Affected Users ​

Extracting User ID from Error ​

Use Cases for Affected Users ​

Error Events & Sampling ​

Data Retention ​

Accessing Full Error History ​

Error Deduplication ​

Error Notifications & Alerts ​

Notification Types ​

Notification Configuration ​

Notification Channels ​

Example: Slack Notification ​

Integration with Issue Trackers ​

Linking Error Groups to Issues ​

Manual Linking ​

Programmatic Linking ​

Error Report Payload ​

What gets sent to Error Reporting ​

Automatic Field Capture ​

Production Patterns ​

Pattern 1: Error Budget Alerting ​

Pattern 2: Gradual Rollout with Error Monitoring ​

Pattern 3: Error Rate SLO Tracking ​

Constraints & Limitations ​

Limitation 1: Fingerprinting Edge Cases ​

Limitation 2: Stack Trace Variation ​

Limitation 3: Privacy - Sensitive Data in Stack Traces ​

References ​

Error Reporting - Grouping, Fingerprinting, và Notifications

Tại sao error reporting không phải chỉ logging

Error Fingerprinting - Group Similar Errors

The Fingerprint Concept

Fingerprint Generation

Fingerprint Stability

Error Group Management

Error Group Lifecycle

Muting Errors (False Positives)

Error Group Queries

Affected User Detection

Metric: Affected Users

Extracting User ID from Error

Use Cases for Affected Users

Error Events & Sampling

Data Retention

Accessing Full Error History

Error Deduplication

Error Notifications & Alerts

Notification Types

Notification Configuration

Notification Channels

Example: Slack Notification

Integration with Issue Trackers

Linking Error Groups to Issues

Manual Linking

Programmatic Linking

Error Report Payload

What gets sent to Error Reporting

Automatic Field Capture

Production Patterns

Pattern 1: Error Budget Alerting

Pattern 2: Gradual Rollout with Error Monitoring

Pattern 3: Error Rate SLO Tracking

Constraints & Limitations

Limitation 1: Fingerprinting Edge Cases

Limitation 2: Stack Trace Variation

Limitation 3: Privacy - Sensitive Data in Stack Traces

References