Load Shedding & Request Prioritization

Tại sao load shedding quan trọng

When system is at max capacity:

Scenario 1 (No load shedding):
  100 requests in queue
  Service can handle 50 requests/sec
  All wait, all degrade equally
  
  Critical user request: Waits 2 seconds
  Logging/analytics request: Also waits 2 seconds
  
  Result: Critical user unhappy, queue never drains

Scenario 2 (Load shedding):
  100 requests in queue
  Service can handle 50 requests/sec
  50 critical requests
  50 non-critical requests
  
  Shed non-critical requests
  50 critical requests processed
  Queue drains
  Critical users happy
  
  Result: Some requests rejected, but service stays responsive

Load shedding is about choosing who to disappoint rather than disappointing everyone equally.

Internal Model: Load shedding strategies

Strategy 1: Server-side rejection (simplest)

When service queue too deep, reject new requests:

// Pseudo-code
func Handle(request Request) Response {
    if requestQueue.Length() > maxQueueSize {
        return Error("Service overloaded, try again later")  // HTTP 503
    }
    
    // Process request normally
    result := Process(request)
    return result
}

Benefit: Simple, no special client knowledge needed.

Cost: Clients must handle 503 and retry later.

Strategy 2: Priority-based shedding

Classify requests by importance, shed low-priority ones first:

Request types:
  - P1 (Critical): User-facing reads/writes, revenue impacting
  - P2 (Important): Internal operations, analytics
  - P3 (Non-critical): Logging, telemetry, debug operations

Under load:
  1. Accept all P1 requests
  2. Accept P2 if capacity left
  3. Shed P3 requests first

Implement:

func Handle(request Request) Response {
    queueLength := requestQueue.Length()
    
    if queueLength > highCapacity {
        // Queue very deep
        if request.Priority != P1 {
            return Error(503, "Overloaded")
        }
    }
    
    if queueLength > mediumCapacity {
        // Queue medium
        if request.Priority == P3 {
            return Error(503, "Overloaded")
        }
    }
    
    // Process request
    return Process(request)
}

Strategy 3: Probabilistic shedding

Instead of hard threshold, use probability:

Queue depth: 50%
  → Shed 10% of requests (random)
  
Queue depth: 80%
  → Shed 50% of requests
  
Queue depth: 95%
  → Shed 90% of requests

Benefit: Smooth degradation, not cliff-like.

shedProbability := (queueLength - minQueue) / (maxQueue - minQueue)

if rand.Float64() < shedProbability {
    return Error(503, "Overloaded")
}

Internal Model: Request prioritization

How to prioritize?

Not all requests are equal:

Hierarchy:
  1. User-facing (directly affects user experience)
  2. Critical backend (payment processing, auth)
  3. Internal operations (batch jobs, maintenance)
  4. Analytics (logging, monitoring)

When overload:
  Shed analytics first
  Then internal operations
  Then backend
  Last, shed user-facing

Signaling priority

Need way for service to know request priority:

Option 1: HTTP header

POST /api/transfer
Priority: critical
Content-Type: application/json

{data}

Service checks header, prioritizes accordingly.

Option 2: Request path

/api/v1/critical/transfer (priority: high)
/api/v1/normal/analytics (priority: normal)

Option 3: Service mesh (Istio)

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api
spec:
  http:
  - match:
    - sourceLabels:
        priority: "critical"
    route:
    - destination: api
      weight: 100
    timeout: 30s
    retries:
      attempts: 3
  - route:
    - destination: api
      weight: 100
    timeout: 5s
    retries:
      attempts: 1

Critical requests get longer timeout and more retries. Non-critical get strict limits.

Production patterns

Pattern 1: Local load shedding (single service)

Service monitors own queue depth:

Queue length 0-10: Accept all
Queue length 10-50: Shed non-critical (P3)
Queue length 50-100: Shed P2, accept only P1
Queue length >100: Shed everything except P0 (payment processing)

Simple, no dependencies on other services.

Pattern 2: Upstream load shedding (API Gateway)

API Gateway (Cloud Armor, Apigee) sheds before requests reach backend:

Client → API Gateway → Backend Service

API Gateway monitors backend health:
  - If backend queue deep, reject requests at gateway
  - Prevent wasted network traffic
  - Shed at the edge

Benefit: Reduced load on backend.

Cost: API Gateway must know which requests to shed.

Pattern 3: Graceful degradation + load shedding

Combine graceful degradation with load shedding:

E-commerce checkout:

Normal load:
  - Full feature: real-time inventory, recommendations, shipping calc
  
High load (70% capacity):
  - Degrade: cached inventory, no recommendations
  
Critical load (90% capacity):
  - Shed: drop analytics, drop non-critical internal requests
  - Keep: checkout, payment processing
  
Overload (>100%):
  - Shed everything except payment + checkout

Implementing load shedding on GCP

Option 1: Application-level

// Go service
type RequestHandler struct {
    queue chan Request
    maxQueueSize int
    prioritizer Prioritizer
}

func (h *RequestHandler) Handle(r Request) Response {
    if len(h.queue) > h.maxQueueSize {
        if r.Priority < CriticalPriority {
            return Response{Status: 503, Error: "Overloaded"}
        }
    }
    
    h.queue <- r
    result := <-h.process(r)
    return result
}

Option 2: Cloud Load Balancer with rate limiting

Cloud Load Balancer can shed load based on rules:

Default: 1000 requests/sec per backend
If exceeded: Return 503 to excess requests

Can configure per:
  - Client IP (rate limit aggressive clients)
  - Request path (rate limit expensive endpoints)
  - User identity (rate limit based on tier)

Option 3: Istio circuit breaker + outlier detection

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: api-lb
spec:
  host: api
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
        maxRequestsPerConnection: 2
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s

Istio ejects (removes from load balancing) pods that are failing, sheds requests to them.

Common mistakes

Mistake 1: Shed too aggressively

At 50% capacity, start shedding
Result: Many valid requests rejected
Service only using 50% capacity
Missing revenue

Fix: Shed only when truly needed (>80-90% capacity).

Mistake 2: No priority distinction

All requests equal priority
Under load, shed randomly
Result: Critical user requests shed equally to analytics

Fix: Distinguish P1/P2/P3, shed based on priority.

Mistake 3: Clients don't handle 503

Service returns 503 (overloaded)
Clients interpret as error
Don't retry
Users see failure

Fix: Clients must handle 503 and retry with backoff.

Mistake 4: Shed without clear rules

"If queue > X, shed Y% of requests"
But what is X? What is Y?
No documentation
Behavior changes with load pattern
Hard to debug

Fix: Document thresholds explicitly.

Shedding policy:
  Queue 50-100: Shed 10% non-critical requests
  Queue 100-200: Shed 50% non-critical
  Queue 200+: Shed 80% non-critical

Trade-offs

Load shedding vs graceful degradation

Graceful degradation:
  - User requests processed (with reduced functionality)
  - Better UX (something vs nothing)
  - Slower recovery (more processing)

Load shedding:
  - Some requests rejected (fail fast)
  - Faster recovery (less processing)
  - Some users affected (explicit rejection)

Choose based on:

• If you can degrade → use graceful degradation
• If you can't degrade → use load shedding
• If you can degrade AND shed → use both

Queue length monitoring vs latency monitoring

Monitor: Queue length
  Pro: Direct signal of overload
  Con: Doesn't account for CPU-bound work (no queue buildup, just slow)

Monitor: Latency
  Pro: User-visible metric
  Con: Reacts to overload too late (P99 already high)

Best: Monitor both. Shed when:

• Queue too deep (queuing delay visible) OR
• Latency too high (processing slow)

Testing load shedding

Test 1: Shed works under load

Load test: 1000 req/sec
Service capacity: 500 req/sec
Expected behavior:
  - Service handles 500 req/sec
  - Sheds ~500 req/sec
  - Shed requests return 503
  - Service stays responsive (P99 latency < 100ms)

Test 2: Priority respected

Load test: 500 P1 + 500 P3 requests
Service capacity: 600 req/sec

Expected:
  - All 500 P1 requests processed
  - ~100 P3 requests processed
  - ~400 P3 requests shed (503)

Test 3: Clients retry on 503

Load test: 1000 requests, service sheds 50%
Verify: Clients retry shed requests
Verify: Retried requests succeed (after load decreases)

Summary

Load shedding protects critical services:

□ Server-side rejection when queue too deep
□ Priority-based shedding to protect critical paths
□ Probabilistic shedding for smooth degradation
□ Upstream shedding to prevent wasted traffic
□ Combine with graceful degradation for best UX
□ Monitor queue depth + latency to trigger shedding
□ Test shedding to ensure it works under load

---

References

• Google SRE Book: Handling Overload
• Google Cloud: Protecting Services from Overload
• Cloud Load Balancer: Rate Limiting
• Istio: Circuit Breaker