Resilience Patterns — Graceful Degradation, Canaries, Circuit Breakers

Tại sao resilience patterns quan trọng

Một dependency fails. A service slows down. Network partition happens.

Without resilience:

• Request times out waiting for dependency
• User experiences timeout error
• Or worse, cascading failure kills your entire service

With resilience patterns:

• Service detects failure
• Degrades gracefully (serves something rather than nothing)
• Recovers when dependency heals
• User experience degraded but acceptable

Resilience patterns are defensive mechanisms — they can't prevent all failures, but they contain them and minimize impact.

Internal Model: Graceful degradation

What is graceful degradation?

Graceful degradation means: when some part of system is unavailable, serve degraded functionality rather than fail completely.

Examples:

E-commerce website:
  - Full feature: Show product + real-time inventory + recommendations + shipping cost
  - Degraded (if inventory service down): Show product + cached inventory + no recommendations + estimated shipping
  - User still gets useful experience

Search service:
  - Full feature: Real-time search with instant completion suggestions
  - Degraded (if index down): Search previous results from cache
  - User gets search results, just not real-time

Social media timeline:
  - Full feature: Real-time feed from network
  - Degraded (if real-time service down): Show cached feed from last sync
  - User sees slightly stale content but not blank

Key distinction from circuit breaker

People often confuse graceful degradation with circuit breaking. They're different:

Graceful degradation:

• Try to serve reduced functionality
• Goal: maximize availability (serve some users, something)
• Example: use cached data if fresh data unavailable

Circuit breaker:

• Stop trying to call failing service
• Goal: fail fast (don't waste resources on failing service)
• Example: If service fails 3 times, stop calling it for next 30 seconds

When to use each:

Scenario: Database is slow (99% latency increase)
├─ Graceful degradation: Serve cached data from Redis instead
│  (User gets data, just cached)
└─ Circuit breaker: Stop querying database, return cached immediately

Scenario: Payment service is down
├─ Graceful degradation: Not applicable (can't degrade payment processing)
└─ Circuit breaker: Stop trying to charge, return error immediately

Designing degradation tiers

Not all features are equally important. Design degradation explicitly:

Service: E-commerce

Tier 1 (Critical): Must work
  - Browse products
  - Add to cart
  - Checkout

Tier 2 (Important): Should work
  - Personalized recommendations
  - Real-time inventory status
  - Product reviews

Tier 3 (Nice-to-have): Can degrade
  - Search suggestions
  - Similar products
  - Analytics tracking

During outage:
  - If database down → Tier 1 only (basic product catalog from cache)
  - If recommendation service down → Skip Tier 3, show Tier 1+2
  - If analytics down → Skip Tier 3, nothing else breaks

Implementing graceful degradation

Pattern 1: Fallback to cache

try:
  product = db.query("product", id)
catch DatabaseError:
  product = cache.get("product", id)  // older data, but something

Pattern 2: Feature flag to disable expensive operations

if feature_flags.enable_recommendations:
  recommendations = get_recommendations(user_id)
else:
  recommendations = []

Pattern 3: Timeout-based fallback

try:
  result = call_service_with_timeout(timeout=100ms)
except TimeoutError:
  result = get_cached_result()  // use stale data

Cost of graceful degradation

Graceful degradation requires:

• Caching layer (Redis, Memcached)
• Fallback logic in code
• Feature flags to disable expensive features
• Testing to ensure degraded path works

Cost is real. Only use for genuinely important features.

Internal Model: Canary deployments

What is canary deployment?

Canary deployment means: release change to small subset of users first, monitor, then roll to everyone.

Canary deployment timeline:

Time 0: Deploy to canary (5% of users)
        Monitor for 15 minutes
        Metrics look good?
        
Time 15min: Deploy to 25% of users
           Monitor for 15 minutes
           Metrics look good?
           
Time 30min: Deploy to 50% of users
           Monitor for 15 minutes
           Metrics look good?
           
Time 45min: Deploy to 100% of users
           Rollout complete

Canary vs big bang deployment

Big bang:

Deploy feature to 100% of users
1% chance it breaks → affects 100% of users
Recovery takes 30 minutes
Impact: 300,000 users affected for 30 min

Canary (5% initial):

Deploy feature to 5% of users
1% chance it breaks → affects 5% of users
Detect issue within 1 minute
Rollback
Impact: 15,000 users affected for 1 min

Canary reduces blast radius by 10x.

Metrics to watch during canary

Don't just deploy and ignore. Monitor actively:

Before deployment:
  Error rate: 0.1%
  p99 latency: 150ms
  CPU: 40%

After canary deployment (5% of traffic):
  Error rate: 0.12% (slight increase, within variance)
  p99 latency: 155ms (acceptable)
  CPU: 40% (unchanged)
  
✓ Metrics look normal, proceed to next stage

After 25% deployment:
  Error rate: 0.8% (significant increase!)
  p99 latency: 300ms (doubled!)
  CPU: 60% (increased)
  
✗ Metrics bad, ROLLBACK immediately

Implementing canary with Istio on GKE

GCP's service mesh (Istio) makes canary easy:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-service
spec:
  hosts:
  - my-service
  http:
  - match:
    - headers:
        user-agent:
          exact: "canary-test"
    route:
    - destination:
        host: my-service
        subset: v2
      weight: 100
  - route:
    - destination:
        host: my-service
        subset: v1
      weight: 95
    - destination:
        host: my-service
        subset: v2
      weight: 5  # Send 5% traffic to v2

This sends 5% traffic to v2, 95% to v1. Google Cloud Monitoring watches error rate, latency. If metrics bad, you can revert this config instantly (no pod restart needed).

Common canary mistakes

Mistake 1: Canary period too short

• Deploy to canary, wait 1 minute, then roll to everyone
• Issue takes 5 minutes to surface
• Result: Issue hits 100% of users

Fix: Wait long enough for issue to surface (usually 15–30 minutes)

Mistake 2: Canary too small

• Deploy to 0.1% of users
• Statistical noise masks real issue
• Issue hits 100% users later

Fix: Canary should be 5–10% (enough to detect real issues)

Mistake 3: Canary in isolated geography

• Canary only to US users
• US load pattern different from India
• Issue only surfaces in India

Fix: Canary should span geographies

Internal Model: Circuit breaker pattern

What is circuit breaker?

Circuit breaker pattern: if service fails repeatedly, stop calling it and return error immediately.

Three states:

CLOSED (normal):
  - Service available
  - Calls go through
  - On failure, increment failure counter
  
OPEN (failing):
  - Failure counter exceeded threshold
  - Stop calling service
  - Return error immediately (don't wait for timeout)
  
HALF_OPEN (recovering):
  - After timeout period, try again
  - If succeeds, go back to CLOSED
  - If fails, go back to OPEN

Analogy: electrical circuit breaker

In electrical systems:

Normal flow: Current flows → CLOSED
Short circuit: Too much current → OPEN (breaks circuit)
Recovery: After fix, try again → HALF_OPEN
If ok: Circuit holds → CLOSED
If still broken: Open again → OPEN

Same pattern applies to services.

When does circuit breaker help?

Scenario: Database is completely down

Without circuit breaker:
  Request 1: Try DB → timeout after 30s → fail
  Request 2: Try DB → timeout after 30s → fail
  Request 3: Try DB → timeout after 30s → fail
  (Each request wastes 30 seconds)

With circuit breaker (threshold=3 failures):
  Request 1: Try DB → timeout after 30s → fail (counter=1)
  Request 2: Try DB → timeout after 30s → fail (counter=2)
  Request 3: Try DB → timeout after 30s → fail (counter=3) → OPEN
  Request 4: Circuit open → return error immediately (no wait!)
  Request 5: Circuit open → return error immediately
  ...
  Request N: Circuit becomes HALF_OPEN → try DB

Benefit: Requests 4–N fail fast (milliseconds) instead of timing out (30 seconds).

Implementing circuit breaker on GCP

Option 1: Application-level (Go example)

import "github.com/grpc-ecosystem/go-grpc-middleware"

cb := circuitbreaker.New(
    Name: "database",
    MaxConsecutiveFailures: 3,
    OpenTimeout: 30 * time.Second,
)

result, err := cb.Execute(func() (interface{}, error) {
    return db.Query(ctx, sql)
})

Option 2: Istio / Service Mesh

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: database
spec:
  host: database
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 30s
      baseEjectionTime: 30s

Istio automatically stops sending traffic to pods failing health checks.

Circuit breaker vs retry

Don't confuse them:

Retry: Try again immediately (or with backoff)

Try request → fail → wait 100ms → try again

Circuit breaker: Stop trying for a period

3 failures → open circuit → fail fast → after 30s, try again

Together:

Try request → fail → retry with backoff → still failing → circuit open → fail fast

Cost of circuit breaker

If circuit opens:

• Service returns errors
• Users see "service unavailable"
• Availability goes down

But availability goes down less than if system was thrashing (retrying forever).

Trade-off: Explicit error vs thrashing.

Production architecture patterns using these

Pattern 1: Canary + monitoring

Deploy new version:
  1. Deploy to canary (5%)
  2. Watch error rate, latency
  3. If metrics good, proceed to 25%
  4. If metrics bad, automatic rollback
  
Benefit: Catch bad deployments before hitting everyone

Pattern 2: Graceful degradation + feature flags

Service dependency fails:
  1. Feature flag: enable_basic_mode = true
  2. API strips non-essential features
  3. Return degraded response
  4. When dependency recovers, flag: enable_basic_mode = false
  
Benefit: Service stays available, users get something

Pattern 3: Circuit breaker + fallback

Database slow/failing:
  1. Circuit breaker detects failures
  2. Stops sending traffic to database
  3. Returns cached data (stale, but something)
  4. When database recovers, circuit closes
  
Benefit: Stops wasting resources on slow database, returns data from cache

Summary

Resilience patterns prevent failures from cascading:

□ Graceful degradation → serve reduced functionality when dependency unavailable
□ Canary deployment → roll out slowly, detect issues early
□ Circuit breaker → fail fast when service failing, don't waste resources
□ Feature flags → disable expensive features without deploy
□ Monitoring → watch metrics during deployment and degradation
□ Fallback → use cache or alternative when primary unavailable

---

References

• Google Cloud: Graceful Degradation
• Google Cloud: Patterns for Scalable and Resilient Apps
• Istio: VirtualService Traffic Management
• Google SRE Book: Dealing with Overload