Failure Handling & Observability

Tại Sao Quan Trọng

Default behavior: Tasks fail → retry → exceed max attempts → silently dropped (unless DLQ).

Nếu không có observability:

• Bạn không biết tasks đang fail (silent)
• Bạn không biết rate (how many?)
• Bạn không biết why (underlying cause?)
• Bạn không recover (manual investigation at 3 AM)

---

Default Failure Behavior

Task Exceeds Max Attempts

Cloud Tasks config:
  max_attempts: 100
  max_backoff: 1 hour

Task fails on attempt 1:
  Retry scheduled in 100ms

Retries continue:
  Attempt 2, 3, ..., 99 all fail
  Attempt 100 fails

Result:
  Task deleted from queue (silently)
  No alert, no notification
  (Unless DLQ configured)

Why Silent Dropout?

GCP philosophy: Tasks are "fire and forget" by default
- Publisher doesn't expect acknowledgment
- Pub/Sub has DLQ, but Tasks doesn't (by default)
- Operator must explicitly enable observability

---

Dead Letter Queue (DLQ)

Enable DLQ

queue := &Queue{
    Name: "projects/my-project/locations/us-central1/queues/main",
    DeadLetterConfig: &DeadLetterConfig{
        DeadLetterQueue: "projects/my-project/locations/us-central1/queues/dead-letter",
        MaxDeliveryAttempts: 5,  // Move to DLQ after 5 attempts (instead of 100)
    },
}

client.UpdateQueue(ctx, &UpdateQueueRequest{
    Queue: queue,
})

Flow

Task in "main" queue
  ↓ (fails)
Retry 1, 2, 3, 4
  ↓ (all fail, attempt 5 scheduled)
Task moved to "dead-letter" queue
  ↓
Operator inspects dead letter queue
  ↓
Investigate + fix root cause
  ↓
Re-enqueue task to "main" queue

Inspect Dead Letter Queue

gcloud tasks list-tasks \
    --queue=dead-letter \
    --location=us-central1 \
    --project=my-project

Re-enqueue from DLQ

# Option 1: Copy task back to main queue
gcloud tasks create-http-task \
    --queue=main \
    --location=us-central1 \
    --uri="https://handler.example.com" \
    --message-body='...' \
    --project=my-project

# Option 2: Manually investigate + fix, then re-create

---

Logging & Monitoring

Cloud Logging Integration

Cloud Tasks automatically logs to Cloud Logging:

{
  "timestamp": "2026-06-24T10:00:00Z",
  "severity": "ERROR",
  "logName": "projects/my-project/logs/cloudtasks.googleapis.com",
  "resource": {
    "type": "cloud_tasks_queue",
    "labels": {
      "queue_name": "projects/my-project/locations/us-central1/queues/my-queue",
      "location_id": "us-central1"
    }
  },
  "jsonPayload": {
    "task_name": "projects/my-project/locations/us-central1/queues/my-queue/tasks/task-id",
    "response_code": 500,
    "dispatch_deadline": "2026-06-24T10:10:00Z",
    "dispatch_time": "2026-06-24T10:00:00Z",
    "attempt_number": 1,
    "response_body": "Internal server error"
  }
}

Key Fields to Monitor

response_code: HTTP status from handler
  200-299 → success
  5xx → retry
  
attempt_number: Which retry (1 = first attempt)
  
dispatch_deadline: When task must complete (for App Engine)
  
response_body: Error message from handler
  useful for debugging

Query Failed Tasks

SELECT
  task_name,
  attempt_number,
  response_code,
  dispatch_time,
  response_body
FROM `my-project.cloudtasks_googleapis_com_cloud_tasks`
WHERE
  severity = "ERROR"
  AND resource.labels.queue_name = "projects/my-project/locations/us-central1/queues/my-queue"
  AND timestamp > TIMESTAMP_SUB(NOW(), INTERVAL 1 HOUR)
  AND response_code != 200  # Failed requests
ORDER BY timestamp DESC
LIMIT 100

---

Alerting Patterns

Alert 1: Max Attempts Reached

# Create alert in Cloud Monitoring
alert_policy = {
    "display_name": "Cloud Tasks Max Attempts Reached",
    "conditions": [
        {
            "display_name": "Max attempts counter > 0",
            "condition_threshold": {
                "filter": """
                    resource.type = "cloud_tasks_queue"
                    AND metric.type = "cloudtasks.googleapis.com/task/attempt_count"
                    AND metric.labels.outcome = "max_attempts_exceeded"
                """,
                "comparison": "COMPARISON_GT",
                "threshold_value": 0,
                "duration": "300s",  # 5 minutes
                "aggregations": [
                    {
                        "alignment_period": "60s",
                        "per_series_aligner": "ALIGN_RATE",
                    }
                ],
            },
        }
    ],
    "notification_channels": ["email@example.com"],
}

Action: Investigate handler, logs, data integrity.

Alert 2: High Failure Rate

alert_policy = {
    "display_name": "Cloud Tasks High Failure Rate",
    "conditions": [
        {
            "display_name": "Failure rate > 5%",
            "condition_threshold": {
                "filter": """
                    resource.type = "cloud_tasks_queue"
                    AND metric.type = "cloudtasks.googleapis.com/queue/task_attempt_count"
                """,
                "comparison": "COMPARISON_GT",
                "threshold_value": 0.05,  # 5%
                "duration": "300s",
            },
        }
    ],
}

Action: Check handler health, dependencies (DB, API), rate limits.

Alert 3: Task Age (Queue Backlog)

# Tasks pending for > 1 hour = backlog
alert_policy = {
    "display_name": "Cloud Tasks Queue Backlog",
    "conditions": [
        {
            "display_name": "Oldest task age > 1 hour",
            "custom_query": """
                fetch cloud_tasks_queue
                | metric 'cloudtasks.googleapis.com/queue/task_count'
                | value [val('task_count')]
                | condition val('task_count') > 1000
                  && resource.queue_name == 'projects/my-project/...'
            """,
        }
    ],
}

Action: Queue dispatcher might be throttled, increase rate limits or add queues.

---

Observability in Handler

Structured Logging

import "cloud.google.com/go/logging"

func PaymentHandler(w http.ResponseWriter, r *http.Request) {
    client := logging.NewClient(r.Context(), projectID)
    defer client.Close()
    
    // Extract task metadata
    taskName := r.Header.Get("X-CloudTasks-TaskName")
    retryCount := r.Header.Get("X-CloudTasks-TaskRetryCount")
    
    // Parse payload
    var req PaymentRequest
    json.NewDecoder(r.Body).Decode(&req)
    
    // Log with context
    client.Logger("payment-handler").Log(logging.Entry{
        Severity: logging.Info,
        Payload: map[string]interface{}{
            "task_name": taskName,
            "retry_count": retryCount,
            "payment_id": req.PaymentID,
            "amount": req.Amount,
            "status": "processing",
        },
    })
    
    // Process
    result, err := chargePayment(r.Context(), req)
    
    if err != nil {
        client.Logger("payment-handler").Log(logging.Entry{
            Severity: logging.Error,
            Payload: map[string]interface{}{
                "task_name": taskName,
                "retry_count": retryCount,
                "payment_id": req.PaymentID,
                "error": err.Error(),
                "error_type": fmt.Sprintf("%T", err),
            },
        })
        w.WriteHeader(http.StatusInternalServerError)
        return
    }
    
    client.Logger("payment-handler").Log(logging.Entry{
        Severity: logging.Info,
        Payload: map[string]interface{}{
            "task_name": taskName,
            "retry_count": retryCount,
            "payment_id": req.PaymentID,
            "status": "completed",
            "transaction_id": result.TransactionID,
        },
    })
    
    w.WriteHeader(http.StatusOK)
}

Metrics in Handler

import "cloud.google.com/go/monitoring/apiv3/v2"

func PaymentHandler(w http.ResponseWriter, r *http.Request) {
    metric := &monitoring.MetricDescriptor{
        Type: "custom.googleapis.com/payment/charges",
        Labels: []*label.LabelDescriptor{
            {Key: "status", ValueType: label.LabelDescriptor_STRING},
        },
    }
    
    // Try to charge
    err := chargePayment()
    
    // Record metric
    status := "success"
    if err != nil {
        status = "failure"
    }
    
    timeSeriesData := &monitoring.TimeSeries{
        Metric: &metric.Type.Metric{
            Type: "custom.googleapis.com/payment/charges",
            Labels: map[string]string{
                "status": status,
            },
        },
        Points: []*monitoring.Point{
            {
                Interval: &monitoring.TimeInterval{
                    EndTime: &timestamp.Timestamp{
                        Seconds: time.Now().Unix(),
                    },
                },
                Value: &monitoring.TypedValue{
                    Value: &monitoring.TypedValue_Int64Value{
                        Int64Value: 1,
                    },
                },
            },
        },
    }
    
    // Send to Cloud Monitoring
    client.CreateTimeSeries(ctx, &monitoring.CreateTimeSeriesRequest{
        Name: fmt.Sprintf("projects/%s", projectID),
        TimeSeries: []*monitoring.TimeSeries{timeSeriesData},
    })
}

---

Troubleshooting Patterns

Pattern 1: Task Stuck in Queue

Symptom: Task age > 1 hour, not dispatched

Diagnosis:
  1. Check queue state: is it RUNNING or PAUSED?
  2. Check rate limits: dispatch_rate = 0?
  3. Check handler: returning 503 (Service Unavailable)?
  4. Check network: connectivity to handler endpoint?

Pattern 2: All Tasks Failing

Symptom: 100% failure rate

Diagnosis:
  1. Check handler logs: what error being returned?
  2. Check handler code: did deployment introduce bug?
  3. Check dependencies: is database down, API unavailable?
  4. Check network: connectivity, firewall rules?
  
Debug:
  curl -v https://handler.example.com/webhook \
    -H "Content-Type: application/json" \
    -d '{"test": "data"}'

Pattern 3: Cascade Failure

Symptom: Initial handler failure → retry storm → more handlers down

Root cause: Backoff too short, retry rate too high

Diagnosis:
  1. Check retry config: min_backoff, max_backoff, max_doublings
  2. Check queue rate: max_dispatches_per_second, max_concurrent_dispatches
  3. Check handler capacity: can it handle retry load?

Fix:
  1. Increase backoff (min 1 second, max 1 hour)
  2. Decrease concurrent (reduce concurrent_dispatches)
  3. Pause queue during incident recovery

Pattern 4: Duplicate Execution

Symptom: Data appears twice (payments charged twice)

Root cause: Handler not idempotent

Diagnosis:
  1. Check handler code: does it check for duplicates?
  2. Check logs: how many times was it called?
  
Fix:
  1. Implement idempotent handler (check database first)
  2. Use deduplication key (trace ID in database)
  3. Add unique constraint to prevent duplicate data

---

Recovery Procedures

Incident: Handler Crashed

Step 1: Pause queue (stop new dispatches)
  gcloud tasks queues update my-queue --pause

Step 2: Investigate handler logs
  Check application logs, errors, root cause

Step 3: Deploy fix
  Fix code, redeploy handler

Step 4: Resume queue
  gcloud tasks queues update my-queue --resume

Step 5: Monitor
  Watch failure rate, retry rate, task age
  Adjust rate limits if needed

Incident: Database Down

Step 1: Pause queue

Step 2: Wait for database recovery (or restore)

Step 3: Test handler manually
  curl https://handler.example.com/test

Step 4: Resume queue (tasks start retrying)

Step 5: Monitor recovery
  Ensure backlog drains (task age decreases)
  Ensure failure rate drops to 0%

Incident: Permanent Data Corruption

Step 1: Pause queue (don't make things worse)

Step 2: Investigate failed tasks (DLQ)
  Which tasks failed? Why?
  
Step 3: Fix data manually (or rollback)
  Restore from backup, fix inconsistencies
  
Step 4: Resume queue
  
Step 5: Optional: Re-enqueue failed tasks from DLQ
  If new attempts would succeed with fixed data

---

References

• Cloud Logging Integration
• Cloud Monitoring Metrics
• Dead Letter Queue Setup
• Troubleshooting Guide