Retry & Exponential Backoff

Tại Sao Quan Trọng

Retry cấu hình sai gây:

• Cascade failure — backoff quá ngắn → retry storm → hơn nữa down
• Silent drops — max attempts vượt quá trước handler có cơ hội retry
• Resource waste — exponential backoff không cấu hình → burst retries

Bạn cần hiểu:

1. Retry được trigger khi nào (HTTP status, timeout)
2. Exponential backoff công thức
3. Max attempts, min/max backoff, doubling factor
4. SLA: task có retry bao lâu trước khi dropped

---

Retry Trigger Conditions

Cloud Tasks retry task nếu:

1. HTTP Status Code Không 200-299

200 OK → SUCCESS (task deleted)
201 Created → SUCCESS
299 Whatever → SUCCESS

300 Redirect → RETRY (suspicious, shouldn't happen for task)
400 Bad Request → RETRY (might be transient)
401 Unauthorized → RETRY (token might expire and refresh)
500 Server Error → RETRY (temporary server issue)
503 Service Unavailable → RETRY (backoff kicked in)

BUT:
- Response không đến (network partition) → TIMEOUT → RETRY

2. Timeout

Handler không gửi response trong timeout duration (App Engine standard: 10 minutes).

Dispatch sent: 10:00:00
Timeout configured: 10 minutes
If no response by 10:10:00 → assume FAILED → RETRY

3. Handler Crash / Crash Loop

Handler nhận request
Handler crashed mid-processing
No response sent
Timeout → RETRY

---

Exponential Backoff Formula

Config Parameters

type RetryConfig struct {
    MaxAttempts     int32          // Default: 100
    MinBackoff      time.Duration  // Default: 100ms
    MaxBackoff      time.Duration  // Default: 3,600s (1 hour)
    MaxDoublings    int32          // Default: 16
}

Formula

delay_i = min(
    max_backoff,
    min_backoff * (2 ^ min(i, max_doublings))
) + random_jitter

Dimana:

• i = retry attempt number (0-indexed)
• 2^i = exponential growth
• max_doublings = caps exponential (prevents infinite growth)
• random_jitter = ±10% random, prevent thundering herd

Default Config Example

min_backoff = 100ms
max_backoff = 3600s (1 hour)
max_doublings = 16

Attempt 0: delay = 100ms
Attempt 1: delay = 100ms * 2^1 = 200ms
Attempt 2: delay = 100ms * 2^2 = 400ms
Attempt 3: delay = 100ms * 2^3 = 800ms
Attempt 4: delay = 100ms * 2^4 = 1.6s
Attempt 5: delay = 100ms * 2^5 = 3.2s
...
Attempt 16: delay = 100ms * 2^16 = 6,553.6s (~1.8 hours) ← capped!
Attempt 17+: delay = 3,600s (1 hour) ← flat at max_backoff

Total retry window:

With max_attempts = 100:
Retries scheduled over ~100 hours (4+ days)
If task succeeds on attempt 50, it was retried ~25 times

---

Backoff Curve — Visualization

Delay (seconds)

3600 ─────────────────────────────────
      ╱─────────────────────────────┐
1000  │                             │
      │   ╱───────────────────┐    │
100   │  ╱                    │    │
      │ ╱                     │    │
1     ├┘                      │    │
      │ 0  5  10 15 20 25 30  │    │ 50   100
      └────────────────────────────────────── Attempt
                     max_doublings=16 capped →

Characteristics:

• First 16 attempts: exponential growth (2x each)
• After attempt 16: flat at max_backoff (1 hour)
• Total duration: ~4+ days for 100 attempts

---

Jitter — Preventing Thundering Herd on Retry

Problem:

All 1,000 tasks fail at 10:00:00
Without jitter: all 1,000 retry at 10:00:100ms
Result: thundering herd, handler overload

With jitter:

All 1,000 tasks fail at 10:00:00
Each scheduled retry with ±10% random delay:
  Task 1: retry at 10:00:090ms
  Task 2: retry at 10:00:095ms
  Task 3: retry at 10:00:105ms
  ...
  Task 1000: retry at 10:00:110ms

Result: spread retries over ~20ms window (smooth, no herd)

---

Max Attempts Semantics

Default: 100 Attempts

Attempt 1: immediate
Attempt 2: 200ms later
Attempt 3: 400ms later
...
Attempt 100: ~3 days later
Attempt 101: DROPPED (max reached)

If task not succeed after 100 attempts:

• Task is silently dropped (unless DLQ configured)
• No alert, no notification
• This is why monitoring is critical

Custom Config

client.UpdateQueue(ctx, &Queue{
    Name: "projects/X/locations/Y/queues/critical",
    RetryConfig: &RetryConfig{
        MaxAttempts: 50,     // Shorter retry window
        MinBackoff: 1*time.Second,
        MaxBackoff: 60*time.Second, // 1 minute, not 1 hour
        MaxDoublings: 10,    // Cap exponential sooner
    },
})

// Now: retry window ~30 minutes (not 4 days)

---

Failure Modes & Boundaries

Mode 1: Transient Failure (Network Blip)

Attempt 1 (10:00:00): Network timeout
Attempt 2 (10:00:00.2s): Network recovered
→ SUCCESS
Total duration: 200ms

Mode 2: Handler Bug (Slow Recovery)

Attempt 1 (10:00:00): Handler crashed
Attempt 2 (10:00:00.2s): Still crashed
Attempt 3 (10:00:00.4s): Still crashed
... 
Attempt 10 (10:00:51.2s): Handler recovered, restarted
→ SUCCESS
Total duration: 51 seconds

Mode 3: Permanent Failure (No Recovery)

Attempt 1 (10:00:00): Database corrupt, data lost forever
Attempt 2 (10:00:00.2s): Still corrupt
...
Attempt 100 (72 hours later): Still corrupt
→ DROPPED (max attempts reached)

Task is gone, no retry anymore

Mode 4: Cascade Failure

Attempt 1 (10:00:00): Handler normal
  → Fails due to resource contention (DB locked)
  → Throws 500 error

Attempt 2 (10:00:00.2s): Retry
  → DB still locked
  → Throws 500 error

Retry storm: exponential backoff eventually kicks in
But if:
  - min_backoff too short
  - max_doublings too small
  - max_backoff too small

Result: Retries still flood handler, preventing recovery

---

Configuring Backoff for Different Workloads

Pattern 1: Strict SLO (Critical Path)

// E.g., payment processing

queue := &Queue{
    RetryConfig: &RetryConfig{
        MaxAttempts: 10,              // Only 10 retries
        MinBackoff: 100*time.Millisecond,
        MaxBackoff: 10*time.Second,   // Cap at 10 sec
        MaxDoublings: 5,               // Exponential only 5x
    },
}

// Retry window: ~30 seconds total
// If fail after 30s, incident (investigate immediately)

Pattern 2: Batch Processing (Relaxed SLO)

// E.g., daily report generation

queue := &Queue{
    RetryConfig: &RetryConfig{
        MaxAttempts: 100,             // Full retries
        MinBackoff: 1*time.Second,
        MaxBackoff: 1*time.Hour,      // Full 1 hour
        MaxDoublings: 16,             // Full exponential
    },
}

// Retry window: ~4 days
// Task can take time, eventually succeeds

Pattern 3: Transient-Heavy (Quick Recovery)

// E.g., cache invalidation

queue := &Queue{
    RetryConfig: &RetryConfig{
        MaxAttempts: 20,              // Medium retries
        MinBackoff: 10*time.Millisecond,
        MaxBackoff: 1*time.Second,    // Short max
        MaxDoublings: 8,              // Quick plateau
    },
}

// Retry window: ~5 seconds
// Assumes transient issues resolve quickly

---

Observability & Monitoring

What to Track

retry_rate (%)
  → % of tasks that needed retry
  → Alert if > threshold (e.g., > 5%)

max_attempts_reached
  → Count of tasks dropped due to max attempts
  → Should be 0 for healthy system

retry_latency_p99
  → How long between attempt N and N+1
  → Verify backoff working as expected

task_age_seconds
  → Age of oldest pending task
  → If very old, something stuck

attempt_distribution
  → Histogram: how many at attempt 1, 2, 3, ..., 100
  → Should be skewed towards low attempts
  → If skewed towards 100, many permanent failures

Alert Rules

Alert: max_attempts_reached > 0
  → Permanent failures happening
  → Action: Debug and fix handler / data integrity

Alert: retry_rate > 10% for > 5min
  → Handler instability
  → Action: Investigate handler, logs, dependencies

Alert: task_age_seconds > 3600 (1 hour)
  → Queue stuck
  → Action: Check if queue paused, if rate limits hit, handler down

Alert: attempt_distribution tail heavy (> 50% at attempt 20+)
  → Slow recovery from transient failure
  → Action: Shorten max_backoff? Increase rate limits?

---

Manual Retry & Force Run

Force Run Task (Immediate Execution)

// Task stuck in retry, you want to retry now (not wait for backoff)
client.RunTask(ctx, &RunTaskRequest{
    Name: "projects/X/locations/Y/queues/Z/tasks/my-task",
})

// Task immediately dispatched (bypass exponential backoff)

Quota: 60 ForceRun per minute per region.

Practical Use Case

Handler bug discovered, fix deployed 10:30:00
But failing tasks are scheduled to retry at:
  10:30:30 (attempt 2)
  10:31:00 (attempt 3)
  10:32:00 (attempt 4)

Instead of wait, ForceRun:
```bash
gcloud tasks run --queue=my-queue --location=us-central1 my-task

Task retried immediately (if still failing).

---

Dead Letter Queue Integration (Preview)

Scenario: Task reaches max attempts, dropped normally.

With DLQ: Task sent to separate "dead letter" queue for analysis.

queue := &Queue{
    Name: "projects/X/locations/Y/queues/main",
    DeadLetterConfig: &DeadLetterConfig{
        DeadLetterQueue: "projects/X/locations/Y/queues/dead-letter",
    },
}

// If task fails max_attempts in "main" queue
// → Task automatically created in "dead-letter" queue
// → Operator manually investigates, fixes data, re-enqueues

---

References

• Retry Configuration
• Understanding Retries
• Run Tasks