Rate Limiting & Dispatch Configuration

Tại Sao Quan Trọng

Rate limiting sai cách gây:

• Thundering herd — handler bị bombard, CPU spike, request timeout
• Cascade failure — handler down → retry storm → hơn nữa down
• Cost explosion — retry loop đắt tiền

Đúng cách bạn cần hiểu:

1. Cấu hình gì (QPS, burst, concurrent)
2. Cách chúng tương tác (không independent)
3. Effective dispatch rate formula
4. Khi nào multi-queue required

---

Ba Knobs Rate Limiting

1. max_dispatches_per_second (QPS)

Định nghĩa: Tối đa bao nhiêu tasks được dispatch mỗi second, trung bình.

Default: 500 QPS

Ảnh hưởng:

10,000 tasks pending
500 QPS → dispatch trong 20 seconds
5,000 QPS → dispatch trong 2 seconds

Điều cần biết:

• Con số này là per queue (không per project)
• Request API gửi task không bị rate limit (chỉ dispatch phase)
• Tùy chỉnh tới ~1,000,000 QPS (yêu cầu quota increase)

2. max_burst_size (Burst)

Định nghĩa: Trong 1 second, dispatch có thể send tối đa bao nhiêu tasks cùng lúc.

Default: 100

Ảnh hưởng:

max_dispatches_per_second = 500
max_burst_size = 100

Second 1: Dispatch 100 tasks cùng lúc
Second 2: Dispatch 100 tasks cùng lúc
Second 3: Dispatch 100 tasks cùng lúc
Second 4: Dispatch 100 tasks cùng lúc
Second 5: Dispatch 100 tasks cùng lúc

(Tiếp tục 500 tasks/second trung bình)

Scenario sai:

max_dispatches_per_second = 500
max_burst_size = 500  ← SIDA!

Second 1: Dispatch 500 tasks cùng lúc → handler overwhelmed
(Nếu handler process 10 request/second, bạn đã shock 50x capacity)

3. max_concurrent_dispatches

Định nghĩa: Tối đa bao nhiêu HTTP requests flying in parallel tại bất kỳ lúc nào.

Default: 1,000

Ảnh hưởng:

max_concurrent_dispatches = 1,000
Handler latency = 10 seconds (slow)

Dispatch rate: 1,000 requests / 10 seconds = 100 QPS effective
(Dù config 500 QPS, bạn chỉ dispatch được 100 QPS vì concurrent slots full)

Handler latency = 100ms (fast)
Dispatch rate: 1,000 / 0.1 = 10,000 QPS effective

---

Cách Chúng Tương Tác — Effective Dispatch Rate

Công thức:

Effective dispatch rate (QPS) = min(
    max_dispatches_per_second,
    max_burst_size × 1 (per second),
    max_concurrent_dispatches / avg_handler_latency_seconds
)

Example 1: Slow Handler

Config:
  max_dispatches_per_second = 500
  max_burst_size = 100
  max_concurrent_dispatches = 1,000

Handler latency = 10 seconds (slow endpoint)

Effective = min(500, 100, 1000 / 10) = min(500, 100, 100) = 100 QPS

Handler chỉ xử lý được 100 requests/second vì concurrent slots:

• 1,000 slots / 10 second latency = 100 QPS

Example 2: Fast Handler

Config:
  max_dispatches_per_second = 500
  max_burst_size = 100
  max_concurrent_dispatches = 1,000

Handler latency = 100ms (fast endpoint)

Effective = min(500, 100, 1000 / 0.1) = min(500, 100, 10,000) = 100 QPS

Rate limiting được burst size, không concurrent slots:

• Burst 100 per second → 100 QPS effective

Example 3: Multiple Queues

Project limit: 10,000 QPS total

Single queue:
  max_dispatches_per_second = 10,000 → dispatch toàn bộ

Multiple queues (separate concerns):
  Queue A (critical): 5,000 QPS
  Queue B (batch): 3,000 QPS
  Queue C (reporting): 2,000 QPS
  Total: 10,000 QPS

---

Preventing Thundering Herd

Anti-Pattern: Handler Slow + High Burst

// WRONG
queue := &Queue{
    RateLimits: &RateLimits{
        MaxDispatchesPerSecond: 1000,
        MaxBurstSize: 1000,           // ← DANGER
        MaxConcurrentDispatches: 5000, // ← DANGER
    },
}

// Handler: 2 seconds latency
// Dispatch: 5000 concurrent / 2 sec = 2,500 QPS effective
// Actual handler capacity: 500 requests/sec
// Result: Handler overwhelmed, CPU spike, request timeout

Pattern: Conservative Burst

// CORRECT
queue := &Queue{
    RateLimits: &RateLimits{
        MaxDispatchesPerSecond: 1000,
        MaxBurstSize: 50,             // Conservative burst
        MaxConcurrentDispatches: 1000,
    },
}

// Dispatch: max 50 tasks/second in burst
// Effective: 50 QPS (limited by burst)
// Handler receives steady stream, no sudden spike

Pattern: Load Testing First

// Step 1: Load test handler
// Find max sustainable QPS
maxQPS := loadTest(handler)  // Returns 500 QPS

// Step 2: Configure queue at 80% of max
queue := &Queue{
    RateLimits: &RateLimits{
        MaxDispatchesPerSecond: int32(maxQPS * 0.8),  // 400 QPS
        MaxBurstSize: int32(maxQPS * 0.1),            // 50 burst
        MaxConcurrentDispatches: int32(maxQPS * 0.5), // 250 concurrent
    },
}

// Step 3: Monitor, gradually increase if stable

---

Cascade Failure Pattern

Scenario: Failure Loop

Time 0: Handler healthy, dispatch 500 QPS

Time 10s: Handler CPU spike (reason: unrelated)
  → Handler starts returning 500 errors
  → Cloud Tasks triggers retry
  → Retry rate = 500 failed tasks per second

Time 20s: Retry + new tasks = 1,000 dispatch attempts
  → Handler now 2x overloaded
  → Error rate 100%
  → More retries scheduled
  
Time 30s: Cascade: original errors + retries + retries-of-retries
  → Handler completely down

Prevention: Backoff + Circuit Breaking

// Handler side
circuit := circuitbreaker.New()

func PaymentHandler(w http.ResponseWriter, r *http.Request) {
    if !circuit.Allow() {
        w.WriteHeader(http.StatusServiceUnavailable) // 503
        return
    }
    
    err := processPayment()
    if err != nil {
        circuit.RecordFailure()
        w.WriteHeader(http.StatusInternalServerError) // 500
        return
    }
    
    circuit.RecordSuccess()
    w.WriteHeader(http.StatusOK)
}

Cloud Tasks respect 503 (retries with backoff). Handler returns 503 instead of 500 → backoff kicked in → not overwhelming.

---

Multi-Queue Pattern — Horizontal Scaling

Problem: Single queue có limit. Project có queue limit (1,000 queues/region).

Solution: Split tasks into multiple queues:

// Queue 1: High-priority, critical path
queue1 := "projects/X/locations/Y/queues/critical"
// RateLimits: 5,000 QPS

// Queue 2: Medium-priority
queue2 := "projects/X/locations/Y/queues/standard"
// RateLimits: 3,000 QPS

// Queue 3: Low-priority, batch
queue3 := "projects/X/locations/Y/queues/batch"
// RateLimits: 2,000 QPS

// Total project: 10,000 QPS (if quota allows)

// When creating task:
switch priority {
case CRITICAL:
    return createTaskInQueue(queue1, task)
case STANDARD:
    return createTaskInQueue(queue2, task)
case BATCH:
    return createTaskInQueue(queue3, task)
}

Advantages:

• Isolation: high-priority tasks không bị batch tasks block
• Independent rate tuning
• Separate handler endpoints with separate scaling

---

Quota vs Limits

Quota (Adjustable)

Queues per project per region: 1,000 (default)
API requests per minute per region: 6,000,000

Request increase via Cloud Console → GCP đánh giá → approve/deny.

System Limits (Fixed, Cannot Change)

Max dispatch rate per queue: 500 QPS (default)
Max task size: 1 MB
Max concurrent dispatches per queue: 1,000

Để vượt qua, dùng multiple queues.

---

Operational Patterns

Pattern: Pause Queue During Incident

// Handler had a bug, deploy fix in progress
client.UpdateQueue(ctx, &Queue{
    Name: "projects/X/locations/Y/queues/payment",
    State: PAUSED, // Stop dispatching
})

// ... fix deployed ...

client.UpdateQueue(ctx, &Queue{
    Name: "projects/X/locations/Y/queues/payment",
    State: RUNNING, // Resume dispatching
})

// Tasks that were pending during pause now dispatch normally

Pattern: Separate Control Planes

// Different services, different SLOs

// Service A: strict SLO (100ms p99)
queueA := &Queue{
    RateLimits: &RateLimits{
        MaxDispatchesPerSecond: 100,
        MaxBurstSize: 10,
        MaxConcurrentDispatches: 50,
    },
}

// Service B: relaxed SLO (5sec p99)
queueB := &Queue{
    RateLimits: &RateLimits{
        MaxDispatchesPerSecond: 10000,
        MaxBurstSize: 1000,
        MaxConcurrentDispatches: 5000,
    },
}

---

Monitoring & Alerting

Key Metrics

dispatch_rate_actual (QPS)
  → Compare with max_dispatches_per_second
  
burst_spike_detected (yes/no)
  → Detect sudden burst of tasks

handler_latency (milliseconds)
  → If ↑, effective QPS ↓ (due to concurrent limit)
  
retry_rate (%)
  → If high, handler problem (investigate)
  
task_age_seconds
  → If high, queue backlog (capacity issue)

Alert Examples

Alert: handler_latency_p99 > 5000ms AND task_age > 60s
  → Handler slow + queue backlog
  → Action: Increase max_concurrent_dispatches or scale handler
  
Alert: retry_rate > 10% for > 5min
  → Handler instability
  → Action: Investigate handler, consider circuit breaker
  
Alert: task_creation_rate >> dispatch_rate for > 10min
  → Producer faster than consumer
  → Action: Increase queue rate limits or add more queues

---

References

• Creating and Managing Queues
• Queue Configuration
• Rate Limiting Best Practices