Error Budget Policy — Quản lý risk deployment

Tại sao cần policy chứ không chỉ tracking?

Error budget là một số: "bạn có 43 phút downtime allowed này tháng". Nhưng nếu không có policy — quy tắc rõ ràng về cách sử dụng nó — error budget chỉ là số trong dashboard.

Không có policy, điều này xảy ra:

Team A: "Chúng ta có 30 phút budget, nên chúng ta có thể deploy 5 features cùng lúc"
Team B: "Chúng ta có 30 phút budget, nên chúng ta không deploy gì cả vì quá nguy hiểm"
Team C: "Bạn không track error budget, sao phải care?"

Kết quả: inconsistent decisions, confusion, và incident được phân bổ ngẫu nhiên.

Error budget policy là quy tắc explicit: "khi error budget ở tình trạng X, chúng ta làm Y."

Internal Model: Policy Structure

Khi nào cần policy?

Policy liên kết error budget status với deployment decision:

Error Budget Status → Deployment Decision
  └─ budget > 50%? → Deploy at normal pace
  └─ budget 30-50%? → Deploy carefully, high-risk features frozen
  └─ budget < 30%? → Deploy only critical fixes, otherwise frozen
  └─ budget < 0%? → Incident mode, release freeze, focus on remediation

Anatomy của một error budget policy

Một good policy có thành phần:

Thresholds — error budget tại các điểm nào trigger action?
Actions per threshold — deployment rule là gì ở mỗi threshold?
Owner — ai quyết định escalate khi boundary vượt?
Communication — làm sao team biết status?
Review cadence — policy được review bao lâu một lần?

Threshold design — picking the right numbers

Common pattern:

Budget Level 1 (Healthy): > 50%
  - Can deploy features as usual
  - Risk appetite: normal

Budget Level 2 (Yellow): 30-50%
  - High-risk features frozen
  - Only essential, low-risk deploys allowed
  - Risk appetite: cautious

Budget Level 3 (Red): < 30%
  - Feature freeze (no feature deploys)
  - Only bugfixes + critical patches
  - Risk appetite: minimal

Budget Level 4 (Crisis): < 0%
  - Complete deployment freeze
  - All hands on deck for incident response
  - If another incident happens, further restriction possible

Why these numbers?

50%: Midpoint, psychological trigger to "pay attention"
30%: Still have runway, but limited
0%: Explicit failure to meet SLO, action required
Not 90%: Too many false alarms; teams ignore it

Different teams can use different thresholds based on risk appetite. E-commerce team might use 40/20/0. Internal tool might use 60/30/0.

Internal Model: What "deployment freeze" means

When people say "feature freeze", they mean different things:

Type 1: Feature Freeze (most common)

No new feature deployments. Exception: bugfixes and critical patches.

Allowed:
  ✓ Deploy p0/p1 bugfixes (customer-impacting bugs)
  ✓ Deploy security patches
  ✓ Deploy performance improvements (if low-risk)

Forbidden:
  ✗ New dashboard feature
  ✗ New API endpoint
  ✗ Refactoring for future scalability

This maintains reliability while still allowing critical fixes.

Type 2: Code Freeze (stricter)

No deployments at all except in dire emergencies.

Allowed:
  ✓ P0 incidents (customer-impacting outages)
  ✓ Active security exploit response

Forbidden:
  ✗ Everything else

Code freeze is temporary crisis mode, not normal state.

Type 3: Partial Freeze (middle ground)

Deploy to non-critical clusters/regions first. If stable, roll forward.

Phase 1: Deploy to canary (5% traffic)
Phase 2: If stable for 1 hour, deploy to 50%
Phase 3: If stable for 1 hour, deploy to 100%

This allows learning with limited blast radius.

GCP best practice: Default to Feature Freeze. Code Freeze only if actively losing money or customer trust.

Mental Model: Why freeze?

Freeze might seem like punishment ("team screwed up, so no feature work"). But it's actually optimization:

If error budget is nearly exhausted:

Next deployment has higher probability of causing incident
If incident happens, you'll exceed SLO and violate customer contract
Therefore: deprioritize risky changes until budget recovers

It's not "you failed", it's "system is at capacity, let's be conservative".

Relationship with deployment risk

Assume deployment has 5% failure probability. If error budget remaining is small:

Scenario A (healthy budget):
  - 5% chance deployment fails
  - Recover within 10 minutes
  - Still have 30 minutes budget left
  - → Deploy (acceptable risk)

Scenario B (depleted budget):
  - 5% chance deployment fails
  - Recover within 10 minutes
  - Now budget is -10 minutes (SLO breach)
  - Customer contract violated
  - → Don't deploy (unacceptable risk)

Freeze is automatic risk management — it prevents team from deploying when risk profile is bad.

Implementing Error Budget Policy

Step 1: Define thresholds and actions

Create a document:

Error Budget Policy v1

SLI: Availability (% non-5xx responses)
SLO: 99.9% over rolling 30 days
Period: Rolling 30 days

Thresholds and Actions:

1. Budget > 50%
   - Status: Green
   - Deployments: Allowed at normal pace
   - Change advisory: Not required
   - On-call escalation: None

2. Budget 30-50%
   - Status: Yellow
   - Deployments: High-risk features frozen
   - Allowed: Bugfixes, performance improvements, low-risk features
   - Change advisory: Required for all deploys
   - On-call escalation: None

3. Budget 10-30%
   - Status: Orange
   - Deployments: Feature freeze (only critical bugfixes)
   - Change advisory: Required, includes risk assessment
   - On-call escalation: Email oncall lead before deploy

4. Budget < 10%
   - Status: Red
   - Deployments: Code freeze (no changes except p0 incident response)
   - Change advisory: VP approval required
   - On-call escalation: Page oncall lead immediately

5. Budget < 0%
   - Status: Critical
   - Deployments: Complete freeze
   - Actions: Incident retrospective, budget recovery plan
   - On-call escalation: Escalation lead engaged

Step 2: Automate threshold alerts

Set up Cloud Monitoring alerts to notify when status changes:

Alert: "Error budget crossing 50% threshold"
  - Condition: error_budget <= 0.5
  - Notification: Email to #sre-alerts Slack channel
  
Alert: "Error budget crossing 30% threshold"
  - Condition: error_budget <= 0.3
  - Notification: Email to #sre-alerts, page on-call SRE
  
Alert: "Error budget crossing 0% threshold"
  - Condition: error_budget <= 0
  - Notification: Page on-call SRE immediately, escalate to SRE lead

Pro tip: Also alert on burn rate. If you're burning budget 10x faster than expected ("at this rate, you'll deplete budget in 3 days"), alert even if absolute budget is still high.

Burn rate calculation:
  Actual SLI = 99.95%
  Target SLO = 99.9%
  Burn rate = (99.95% - 99.9%) / 0.1% = 5x
  
Interpretation: You're exceeding budget 5x faster than expected

Step 3: Surface policy clearly to deployment team

Most teams won't read a long document. Make it visible at deployment time:

$ kubectl apply -f service.yaml

[DEPLOYMENT GATE CHECK]
Current error budget: 23.4% (YELLOW STATUS)
Policy action: Feature freeze in effect

Features allowed:
  ✓ P1 bugfixes
  ✓ Security patches
  ✗ New features

Current deployment: "Add new search filter UI" (NEW FEATURE)
Status: ✗ BLOCKED

To proceed:
  1. Change deployment to bugfix OR
  2. Get exception approval from SRE lead (@sre-lead on-call)

This requires integration between CD pipeline and budget monitoring.

Step 4: Weekly review

Review error budget status every week:

Weekly SRE Sync
├─ Budget status (current %)
├─ Burn rate (are we on track?)
├─ What consumed budget (incidents, deployments)
├─ Forecast (will we exceed SLO?)
└─ Action items (deploy freeze? incident prevention?)

Policy in Practice — Real scenarios

Scenario 1: Normal case

Monday: Budget 65%, green status
  - Deploy feature A (low-risk)
  - Unexpected incident causes 15min downtime
  - Budget now 52%

Tuesday: Budget 52%, still green
  - Deploy feature B (low-risk)
  - No issues
  - Budget now 48%

Wednesday: Budget 48%, yellow status (triggered)
  - Try to deploy feature C (high-risk)
  - Gets blocked by policy gate
  - Team removes high-risk changes, resubmits as low-risk
  - Deploy succeeds

Friday: Budget stabilizes at 42%

Scenario 2: Incident escalation

Thursday morning: Budget 25%, orange status
  - Several production deployments waiting
  - SRE on-call: pause deployments, freeze feature queue
  - Investigation into why burn rate high
  - Find: a service leaking connections, causes cascading failures
  - Deploy fix for leak

Thursday afternoon: Budget 30%, still orange
  - Can resume deployments but cautiously
  - Only critical bugfixes allowed
  - Feature queue held until budget > 50%

Scenario 3: Policy override request

Product manager: "We MUST deploy feature X today,
                  it's for a major customer"

SRE lead: "Budget is at 15%, policy says feature freeze."

Product manager: "Can we get exception?"

SRE lead: "Yes, but cost is explicit:
           - Acknowledge risk: 10% chance this breaks something
           - If incident happens, we might breach SLO
           - Requires VP sign-off"

VP approves, deployment proceeds with explicit risk acknowledgment

The benefit of policy: decisions are explicit and documented, not implicit.

Common Implementation Mistakes

Mistake 1: Policy too strict

Policy: Any budget < 50% → freeze features
Result: Team freezes features 70% of month
         Feature velocity drops to 0
         Product team unhappy

Fix: Thresholds should allow some feature work even at yellow. Only freeze high-risk changes.

Mistake 2: Policy never enforced

Policy: Budget < 30% → no feature deploys
Reality: Team deploys 5 features when budget at 20%
         Nobody enforces it
         Policy becomes meaningless

Fix: Automate enforcement in CD pipeline. Don't rely on manual compliance.

Mistake 3: Policy doesn't account for incident patterns

Team has policy: Feature freeze when budget < 30%
Monday morning: Incident causes budget to drop to 20%
Team: "Freeze features immediately"

But root cause: buggy code from 3 weeks ago
Team doesn't deploy fix because of freeze
More incidents happen
Budget further depleted

Fix: Policy should allow fixing root cause. The goal is preventing new failures, not preventing recovery from current ones.

Mistake 4: SLO so loose that freeze never triggers

SLO: 99% (very loose)
Error budget: 43,200 minutes per month
Reality: Rare to have incident > 1 hour
Result: Budget never depleted, policy never enforced

Fix: If budget never consumed, either:

SLO is too loose (raise it)
System is highly reliable (celebrate, but don't change SLO)

Policy communication — who needs to know?

SRE team

Must understand policy deeply
Owns monitoring and enforcement

Platform/Backend engineers

Must understand thresholds
Check policy before submitting deployment

Product managers

Should understand "feature freeze can happen"
Helps prioritization planning

Executives

Should know error budget is tied to deployment velocity
Explains why "velocity is limited by reliability"

Best practice: Quarterly business review should include "error budget consumed this quarter" and "deployment freeze days". This connects engineering reliability to business outcomes.

Policy evolution

Error budget policy should not be static. Review quarterly:

Q1 Review:
  - Was 30% threshold too strict? (features frozen too often)
  - Was burn rate alert useful? (was it accurate?)
  - Did we discover new failure modes?
  - Should we adjust thresholds for Q2?

Common evolution:

Year 1: Threshold 50/30/10 (conservative)
Year 2: Threshold 40/20/5 (more mature infrastructure, can take more risk)
Year 3: Threshold 60/40/20 (system highly reliable, can do more feature work)

Summary

Error budget policy is the connection between SLO and deployment velocity.

Key concepts:

□ Policy translates error budget (a number) into actionable rules
□ Thresholds trigger actions: green → normal, yellow → careful, red → freeze
□ Feature freeze ≠ code freeze; freeze high-risk changes, not all changes
□ Policy should be automated in CD pipeline, not manual
□ Policy should allow fixing root cause, not prevent recovery
□ Communicate policy clearly to engineers, product, and leadership
□ Review policy quarterly and adjust based on actual incidents

Error Budget Policy — Quản lý risk deployment ​

Tại sao cần policy chứ không chỉ tracking? ​

Internal Model: Policy Structure ​

Khi nào cần policy? ​

Anatomy của một error budget policy ​

Threshold design — picking the right numbers ​

Internal Model: What "deployment freeze" means ​

Type 1: Feature Freeze (most common) ​

Type 2: Code Freeze (stricter) ​

Type 3: Partial Freeze (middle ground) ​

Mental Model: Why freeze? ​

Relationship with deployment risk ​

Implementing Error Budget Policy ​

Step 1: Define thresholds and actions ​

Step 2: Automate threshold alerts ​

Step 3: Surface policy clearly to deployment team ​

Step 4: Weekly review ​

Policy in Practice — Real scenarios ​

Scenario 1: Normal case ​

Scenario 2: Incident escalation ​

Scenario 3: Policy override request ​

Common Implementation Mistakes ​

Mistake 1: Policy too strict ​

Mistake 2: Policy never enforced ​

Mistake 3: Policy doesn't account for incident patterns ​

Mistake 4: SLO so loose that freeze never triggers ​

Policy communication — who needs to know? ​

SRE team ​

Platform/Backend engineers ​

Product managers ​

Executives ​

Policy evolution ​

Summary ​

References ​