SLI, SLO, Error Budget — Định lượng độ tin cậy

Tại sao điều này quan trọng ở production

Hầu hết các engineering team đều nói rằng họ "care về reliability". Nhưng nếu bạn hỏi "chúng ta sẽ chấp nhận bao nhiêu downtime trong quý này?" không nhiều người có câu trả lời cụ thể.

Kết quả:

• Engineer A muốn deploy cùng lúc 5 features vì "chúng có thể improve user experience"
• Engineer B muốn dừng hết deployment trong quarter vì "chúng ta vừa có downtime"
• Manager không biết team sẽ release được bao nhiêu feature trong quarter vì không có quy tắc clear

Khi không có định lượng rõ ràng, reliability decisions dựa trên gut feeling thay vì facts. SLI, SLO, error budget là các công cụ để chuyển điều này thành hệ thống.

Internal Model: SLI, SLO là gì?

SLI (Service Level Indicator)

SLI là một measurement — "bao nhiêu % requests chúng tôi handled successfully trong time period này?"

Ví dụ:

• Availability SLI: (successful requests) / (total requests) — e.g., 99.95%
• Latency SLI: bao nhiêu % requests completed trong < 200ms — e.g., 95% requests < 200ms
• Error rate SLI: (non-5xx responses) / (total responses) — e.g., 99.8% non-5xx

SLI phải đo từ user perspective chứ không phải từ "server logs say everything is fine". Ví dụ:

• Sai: "Database engine reported 0 errors"
• Đúng: "How many user requests succeeded end-to-end?"

Google Cloud tài liệu khuyến cáo: "Một SLI tốt phải capture những gì users thực sự care về".

SLO (Service Level Objective)

SLO là một target cho SLI. Nó trả lời câu hỏi: "Mục tiêu của chúng tôi cho SLI này là gì?"

Ví dụ:

• SLI: availability = 99.95%
• SLO: availability >= 99.9% (rolling 30 days)

SLO không phải là "99.99% là best practice cho mọi service". Nó là một conscious choice:

• Nếu SLO là 99.99%, bạn chỉ có 4 phút downtime allowed per month
• Nếu SLO là 99%, bạn có 43 phút downtime allowed per month

Thường, SLO được chọn dựa trên:

1. User impact — cái gì user notice?
2. Business requirement — service này cần reliable đến đâu?
3. Cost of implementation — 99% cost × 10, 99.9% cost × 50, 99.99% cost × 200

Relationship giữa SLI, SLO, và reality

Reality (actual performance)
       ↓
SLI (measurement)
       ↓
SLO (target)
       ↓
Error Budget (allowance)
       ↓
Deployment Decisions

SLI là observable metric. SLO là target trên metric đó. Error Budget là "có bao nhiêu thời gian mà chúng ta được phép breached SLO?"

Internal Model: Error Budget

Định nghĩa

Error budget là thời lượng cho phép mà hệ thống có thể không tuân thủ SLO mà vẫn được coi là compliant.

Công thức:

Error Budget = (1 - SLO_target) × (period_length)

Ví dụ cụ thể:

SLO: 99.9% availability trong rolling 30 days.

Error Budget = (1 - 0.999) × 30 days
             = 0.001 × 30 × 24 × 60 minutes
             = 0.001 × 43,200 minutes
             = 43.2 minutes

Vậy trong 30 days, chúng tôi có 43 phút downtime allowed. Đó không phải là "mục tiêu vì chúng ta không track downtime tốt". Đó là explicit budget — nó có thể được sử dụng cho:

• Maintenance windows
• Risky deployments
• Infrastructure failures
• Test disaster recovery procedures

Error budget consumption — Ngân sách được dùng thế nào

Mỗi khi hệ thống breach SLO, error budget bị consume. Ví dụ:

• SLO: 99.9% availability
• Incident 1: hệ thống down 5 phút → consume 5 / 43.2 ≈ 11.6% of budget
• Incident 2: hệ thống down 20 phút → consume 20 / 43.2 ≈ 46% of budget
• Remaining budget: 43.2 - 5 - 20 = 18.2 minutes

Bây giờ, bạn chỉ có 18 phút còn lại để deploy, test, hoặc sự cố khác xảy ra trong tháng.

Rolling vs Calendar periods

Calendar period (e.g., "per month"):

• 30-day period: Jan 1–30, Feb 1–28, etc.
• Error budget được tính toán lại khi period kết thúc

Rolling period (e.g., "30-day rolling"):

• Last 30 days từ now
• Budget được tính toán liên tục — mỗi giây, một second cũ được dropped, một second mới được thêm
• Cho phép daily decisions — "budget có enough để deploy hôm nay không?"

Rolling period thường tốt hơn cho deployment decisions vì nó dynamic, không phải "wait cho cuối tháng rồi mới biết".

Mental Model: Error Budget là gì?

Nhiều engineer hiểu sai error budget là "target downtime mà chúng tôi aim for". Sai.

Error budget là:

• Permission to fail — không phải target mà downtime allocation
• Tool cho prioritization — "chúng ta nên deploy feature A hay B?" — answer phụ thuộc vào error budget
• Signal khi status tốt hoặc tệ — "budget depleted?" → pause risky changes; "budget healthy?" → safe to proceed

Một cách nghĩ khác:

• SLO 99.9% không nghĩa là "chúng tôi aspire tới 99.9% uptime"
• Nó nghĩa là "chúng tôi committed to 99.9%, và khi chúng tôi exceed 0.1% downtime, đó là incident"

Error budget tracking không phải về "vinh dự" — nó về decision-making infrastructure.

Why error budget > "just use SLO target"?

Một số team chỉ track "is SLI >= SLO target" − yes/no decision. Tại sao error budget tốt hơn?

1. Granular visibility — Bạn có thể thấy "chúng ta đã consume 60% budget" ngay trong tháng, không phải "status: compliant or not"
2. Enables planning — "chúng ta có enough budget để deploy này?" là calculable question
3. Aligns teams — Engineer, Product, Ops tất cả agree: deployment risk vs budget tradeoff

Ví dụ:

Team A có 2 risky deployments pending:

• Feature A: 70% chance success, 30% chance 15min downtime
• Feature B: 80% chance success, 20% chance 10min downtime

Remaining error budget: 30 minutes

Expected downtime from Feature A: 0.3 × 15 = 4.5 min
Expected downtime from Feature B: 0.2 × 10 = 2 min
Both are safe individually.
Can we do both? Expected total: 6.5 min — yes, still safe.

Nếu không track error budget, team không có framework để make decision.

SLO Design — Chọn SLI & Target một cách sáng suốt

Chọn SLI metric

SLI phải đo user-impacting characteristics. Các metric candidates:

1. Availability — successful requests / total requests

   • Good for: "Can I reach the service?"
   • Example: non-5xx responses

2. Latency — % requests < Xms

   • Good for: "Is the service fast enough?"
   • Example: 95% of requests < 200ms

3. Throughput — actual requests / expected requests

   • Good for: "Can I handle peak load?"
   • Rarely primary SLI; usually paired with availability

4. Custom metrics — application-specific

   • Example (payment service): % transactions processed without retry
   • Example (search): % queries returned results < 500ms

Rule of thumb: pick 1–3 metrics max, not 10. Too many metrics dilute focus.

Google Cloud best practice

Google's SRE book recommends:

• Start with availability SLI (most important)
• Add latency SLI if speed matters (e.g., user-facing)
• Add custom SLI only if it captures something not in above two

Chọn SLO target

SLO không phải "99.99% because it's standard". Nó phải được driven bởi:

1. User tolerance — Đặt bao nhiêu latency là user sẽ notice?

   • Consumer app: 200ms noticeable, 500ms unacceptable
   • Internal tool: 2s acceptable

2. Business context — Downtime cost?

   • Payment service: downtime = lost revenue → target 99.99%
   • Internal tool: downtime = reduced productivity → target 99%
   • Non-critical blog: downtime = few complaints → target 99%

3. Implementation cost — What's the cost of extra 9s?

   • 99% → 99.9% (10x more stringent): +20% infrastructure complexity
   • 99.9% → 99.99% (10x more stringent): +50% infrastructure complexity
   • 99.99% → 99.999% (10x more stringent): +200% infrastructure complexity

Google Cloud guidance: "SLOs shouldn't be higher than is necessary or meaningful for your users."

Common anti-pattern: SLO as "we're so reliable" statement

Bad:

• Team sets SLO 99.99% to "look good"
• Then they spend 6 months on infrastructure to hit it
• But users would have been happy with 99.5%

Good:

• Team asks "what does user care about?"
• Sets SLO 99.5% based on user expectations
• Spends engineering effort on features

Your SLO should reflect actual commitment, not aspiration.

Calculating Error Budget — Bagaimana math bekerja

Simple case: Availability SLI

SLI: "% of requests that didn't return 5xx" SLO: "≥ 99.9% over rolling 30 days"

Total eligible events in 30 days:
  - Assume 1000 requests/sec
  - 30 days = 2.592 million seconds
  - Total requests = 2.592 billion

Error budget:
  - (1 - 0.999) × 2.592 billion = 2.592 million bad requests allowed
  - Or in downtime equivalence: 0.001 × 30 days = 43.2 minutes

Actual compliance so far (mid-month):
  - 1.3 billion requests processed
  - 1 million returned 5xx
  - Actual SLI = (1.3B - 1M) / 1.3B = 99.923%
  - Current good-request budget: 0.001 × 1.3B = 1.3M
  - Consumed: 1M
  - Remaining: 300k requests

Latency SLI — trickier

SLI: "% of requests completed in < 200ms" SLO: "≥ 95% per rolling 30 days"

Total requests: 2.592 billion
Good requests (< 200ms): 95% = 2.46 billion
Bad requests (>= 200ms): 5% = 129.6 million allowed

If actual:
  - 2.55 billion < 200ms
  - 42 million >= 200ms
  - Actual SLI = 2.55B / 2.592B = 98.4%
  
Status: compliant, still have budget

Google Cloud documentation: "Error budget for an SLO represents total amount of time service can be noncompliant before violation."

Trade-offs in SLO Design

SLO too high — engineering tax

If you set SLO 99.99% when users only care about 99%:

• You overspend on redundancy, failover, monitoring
• You slow down feature velocity (more testing, more risk-averse)
• You burn out engineers on ops toil

SLO too low — customer unhappy

If you set SLO 99% when users expect 99.9%:

• Customers leave
• Reputation damage
• You already have infrastructure for 99.9%, so setting 99% is leaving money on table

Multiple SLOs for different tiers

Some services have multiple SLOs by user tier:

• Premium tier: 99.99%
• Standard tier: 99.9%
• Free tier: 99%

This allows prioritization — when system is overloaded, free tier gets dropped first.

GCP best practice: if you support multiple tiers, define SLO per tier explicitly.

Compliance Periods and Decision Making

Calendar vs Rolling

Calendar month (Jan 1–30, Feb 1–28):

• ✗ Can't make decisions mid-month "what's my budget?"
• ✓ Aligns with business planning (months)

Rolling 30 days:

• ✓ Can make decisions every day
• ✓ Incident 20 days ago falls out automatically
• ✗ Less alignment with quarterly planning

Recommendation: Use rolling period for deployment decisions (daily hygiene), but track calendar period for business reporting (executive review).

SLI/SLO in Cloud Monitoring

On GCP, you can define SLI/SLO directly in Cloud Monitoring:

Service > Cloud Monitoring > Service Level Indicators

Create SLI:
  - Metric: 200 or 300 responses / total responses
  - Filter: by endpoint, by client, etc.

Create SLO:
  - SLI: above SLI
  - Goal: 99.9%
  - Period: 30-day rolling
  - Compliance window: daily

Cloud Monitoring automatically calculates error budget burn rate and alerts you when you're burning budget faster than expected (e.g., "you're on track to deplete budget in 7 days if this trend continues").

This is actionable — it tells team "we need to act now".

Common mistakes and lessons

Mistake 1: Too many SLOs

Team defines SLO for:

• Availability
• Latency p50, p95, p99
• Error rate
• Custom business metrics (5 more) = Total 12 SLOs

Result: nobody knows which one to optimize. Focus diluted.

Fix: Max 3 SLOs. Usually availability + latency is enough.

Mistake 2: SLI doesn't match user experience

Team measures "application server responded" but SLI doesn't include:

• Database query time
• Network latency
• Frontend rendering

Result: SLI looks good but user experience terrible.

Fix: SLI must measure end-to-end from user perspective.

Mistake 3: SLO never changes

Team set SLO 99.99% three years ago. System changed, architecture changed, but SLO static.

Result: either SLO is meaningless (system always exceeds it), or team wastes resources on too-high target.

Fix: Review SLO quarterly. If system consistently exceeds target by 5x, lower it. If consistently near boundary, raise resources or lower commitment.

Mistake 4: Error budget "used up" is bad

Manager sees team consumed error budget and scolds them: "you weren't reliable enough!"

Wrong mental model. Error budget is meant to be used. It's the tool to enable safe velocity.

Fix: Educate leadership. Error budget depletion is not failure—it's optimization signal.

Summary: Mental model checklist

After reading this section, you should have built:

□ SLI = measurement of service success from user perspective
□ SLO = target on SLI, chosen based on user needs + business cost
□ Error budget = time allowed to breach SLO, enables decisions
□ Relationship: higher SLO = lower error budget = more carefully managed deployments
□ Design principle: SLO is not "how reliable can we be?" but "how reliable do users need?"
□ Rolling period preferred for daily ops decisions; calendar period for business reporting

---

References

• Google SRE Book: Service Level Objectives
• Google SRE Workbook: Implementing SLOs
• Google Cloud: Designing SLOs
• Google Cloud: Service Level Monitoring
• Google Cloud Blog: Understanding Error Budget Overspend