SLIs, SLOs, and SLAs: Measuring Reliability with Error Budgets
"99.9% uptime" sounds like a number everyone agrees on until you ask what counts as up, measured where, over which month. SLIs, SLOs, and SLAs answer those questions, and error budgets turn them into decisions about when to ship and when to stop. Here's how they fit together and how to alert on them without paging people for nothing.
"Is the Service Reliable?"
Your dashboards are green. CPU is fine, the pods are healthy, and the uptime check has passed every minute this month. Meanwhile, support has a stack of tickets saying checkout fails for some users, and sales wants to promise a big customer "four nines."
Everyone in that scenario has a different idea of reliability. The uptime check measures whether the homepage answers. Users care whether checkout works. The contract will care about whatever number gets written into it. SLIs, SLOs, and SLAs exist to make those the same conversation, and the error budget is what makes them useful day to day.
Three Terms, Three Audiences
A measurement of how the service is doing, from the user's point of view, expressed as a ratio of good events to total events. For example: the proportion of checkout requests that return a non-5xx response within 800ms. It's a number between 0% and 100%.
A target for an SLI over a time window: 99.9% of checkout requests are good, measured over a rolling 28 days. It's an internal engineering goal, and missing it should change what the team works on, not trigger refunds.
A contract with customers that says what happens, usually service credits, if reliability falls below a threshold. SLAs are written by lawyers and sales as much as engineers, and they're almost always looser than the internal SLO, so you find out about problems before they cost money.
Plenty of internal services need SLOs and have no SLA at all. The reverse, an SLA without an internal SLO, means you learn about breaches from your customers.
Choosing a Good SLI
A good SLI measures what users experience. Most bad SLIs measure what's convenient.
| Weak SLI | Why it misleads | Better SLI |
|---|---|---|
| Server CPU < 80% | Users don't experience CPU; a hot server can serve well and an idle one can be broken | Proportion of requests served successfully |
| Ping/uptime check passes | Checks one URL from one place every minute; misses partial failures | Ratio of good to total real user requests |
| Average latency | An average hides the slow tail that users actually complain about | Proportion of requests faster than a threshold (e.g. 95% under 300ms) |
| Error count | Grows with traffic; 500 errors is a lot at 10k requests and nothing at 100M | Errors as a fraction of requests |
The usual starting point is two SLIs per user-facing journey: availability (the share of requests that succeed) and latency (the share that are fast enough). Data pipelines add freshness (how old is the newest data) and correctness. Measure as close to the user as you can. Load balancer logs are a good default, because they include failures your application never saw, like requests that timed out before reaching it.
✅ Define 'good' carefully
Should a 429 rate-limit response count as a failure? A 404? A request that succeeded in 4 seconds? Write the definition down. Client errors caused by bad input usually don't count against you; errors caused by your own overload usually should.
The Error Budget
Here's the idea that makes SLOs worth the effort. A 99.9% SLO doesn't just say "be reliable." It says 0.1% of requests are allowed to fail. That allowance is the error budget.
| SLO | Allowed bad requests | As full downtime in 30 days |
|---|---|---|
| 99% | 1 in 100 | 7.2 hours |
| 99.5% | 1 in 200 | 3.6 hours |
| 99.9% | 1 in 1,000 | 43.2 minutes |
| 99.95% | 1 in 2,000 | 21.6 minutes |
| 99.99% | 1 in 10,000 | 4.3 minutes |
The budget changes the conversation between product and reliability work. While there's budget left, the team ships: risky deploys, migrations, and experiments are fine, because some failure is expected and paid for. When the budget runs low, the team slows down and works on reliability until it recovers. Nobody has to argue about whether the service is "reliable enough"; the budget answers it.
🔴 100% is the wrong target
Users reach you over Wi-Fi, mobile networks, and ISPs that are themselves less than 99.99% reliable, so they can't tell the difference between your 99.99% and a perfect 100%. Getting from 99.9% to 99.99% typically costs far more than the first 99.9% did: redundancy across regions, slower releases, more on-call load. Choose the lowest target your users will actually be happy with.
Each extra nine also shrinks your reaction time. At 99.99%, a single 5-minute outage spends more than the whole month's budget, so a human getting paged, opening a laptop, and finding the problem is already too slow. High targets require automation, like automatic rollbacks, not just faster people.
Watching the Budget Burn
Pick a target, set a background error rate, and add one incident. The line shows how much of the 30-day budget is left.
Model: steady traffic in 10-minute buckets over a 30-day window, a constant background error rate, and one incident on day 13 at midnight. Alerts use the multiwindow burn-rate thresholds from the Google SRE Workbook (without the short confirmation windows).
A few things to try:
- Same incident, different targets. A 30-minute incident with 20% of requests failing barely dents a 99% budget. At 99.9% it takes a big bite. At 99.99% the incident alone spends more than the whole month's budget.
- Turn the incident off and raise the background error rate. A slow, steady leak can exhaust the budget without any single outage. Notice that the ticket alert catches it, not the page.
- Make the incident short and total. 100% of requests failing for 10 minutes fires a page right away. Now try 1% failing for 3 hours at 99.9%: no page fires, yet it spends several percent of the month's budget. Stretch it to 4 hours and the 6-hour alert catches it.
Alerting on Burn Rate, Not on Errors
The traditional alert, "error rate above 1% for 5 minutes," is wrong in both directions. It pages for brief blips that don't threaten the SLO, and it stays quiet during a slow leak that will.
The SRE approach is to alert on burn rate: how fast you're spending the budget compared to the rate that would use exactly all of it by the end of the window. A burn rate of 1 means you'll end the month with zero budget left. A burn rate of 14.4 means you'll use up 30 days of budget in about 2 days.
| Severity | Long window | Burn rate | Budget spent when it fires |
|---|---|---|---|
| Page | 1 hour | 14.4× | 2% |
| Page | 6 hours | 6× | 5% |
| Ticket | 3 days | 1× | 10% |
These numbers come from the Google SRE Workbook. In practice each alert also has a short window (5 minutes for the 1-hour alert, 30 minutes for the 6-hour one, 6 hours for the 3-day one) that must also exceed the threshold. That second condition lets the alert stop firing soon after the problem is fixed, instead of staying red for the rest of the long window.
💡 About the simulator
The simulator uses steady traffic and checks only the long windows, so treat its timings as rough. Real traffic has daily cycles, and a 99.99% SLO on a low-traffic service has so few allowed failures that a handful of errors can trip it; that's a known weakness of request-based SLOs at small scale.
Making the Budget Mean Something
An SLO that nothing depends on is just another dashboard. The part that matters is agreeing in advance what happens when the budget runs out. A typical error budget policy says:
Budget healthy
Normal release cadence. The team spends its time on product work and accepts some risk.
Budget below a threshold (say 25%)
Extra review on risky changes. Postmortem action items get prioritized ahead of new features.
Budget exhausted
Feature releases pause except for reliability fixes and security patches, until the rolling window recovers. Product and engineering leadership signed off on this ahead of time, so nobody is negotiating it during an incident.
The policy also works the other way. If a team consistently ends the month with most of its budget unspent, the SLO may be stricter than users need, or the team is being too careful. Either way, that budget could be spent shipping faster.
Key Takeaways
- SLIs measure, SLOs target, SLAs promise. Keep the SLA looser than the SLO so you hear about problems before customers can claim credits.
- Measure what users experience. Good requests over total requests, as close to the user as you can get, not CPU or a single ping.
- The error budget is the point. 99.9% means 0.1% of requests may fail, and that allowance is what you spend on shipping.
- Don't aim for 100%. Each extra nine costs more and shrinks the time you have to react.
- Alert on burn rate across multiple windows. It catches fast outages and slow leaks while ignoring blips.
- Agree on the error budget policy in advance. What happens when the budget runs out is a decision for a calm day, not an incident.