SLOs and Error Budgets: A Practical Setup Guide for Engineering Teams

Most engineering teams know they should have service level objectives. Fewer have actually defined them in a way that drives decisions. Even fewer have error budget policies that anyone follows when the budget runs low. The gap between "we have SLOs" and "SLOs change how we work" is a setup and process problem, not a tooling problem.

This guide walks through the full setup: defining SLIs, writing SLO targets, calculating error budgets, and building alerting that fires at the right time rather than constantly or never.

Start With the Right Service Level Indicator

An SLI is the metric you actually measure. An SLO is the target you hold it to. Getting the SLI definition right matters more than the percentage you pick for the target.

Good SLIs measure what users experience:

• Availability — the proportion of requests that returned a successful response (typically anything outside 5xx)
• Latency — the proportion of requests that completed within a defined threshold (e.g., what fraction were under 300ms)
• Error rate — the fraction of requests that resulted in an application-level error
• Throughput — whether the system processed the expected volume without degradation

The most common mistake at this stage is measuring internal system metrics — CPU, memory, queue depth — instead of user-facing behavior. Internal metrics are useful for debugging. SLIs should describe the user's experience of the service.

Define your SLI as a ratio:

SLI = good_events / total_events

For an HTTP service: good_events = requests returning 2xx or 3xx, total_events = all requests.

Writing an SLO Target

An SLO is a target for your SLI over a rolling time window. A common starting structure:

99.5% of requests to the checkout API will return a non-5xx response,
measured over a rolling 28-day window.

The 28-day rolling window is standard practice because it avoids the cliff edge of calendar-month resets and gives a stable view of recent behavior.

On choosing the target percentage: start with what you can actually achieve today, then tighten it in future quarters. Setting an aspirational target that your system never meets makes the SLO meaningless as a signal. Look at recent error rates, add a small buffer, and set that as your initial target.

For most product-facing APIs, targets in the 99% to 99.9% range are appropriate. Above 99.9% requires significant investment in redundancy and should be reserved for services where downtime has direct financial or safety consequences.

Calculating the Error Budget

The error budget is the amount of unavailability your SLO permits. It is the flip side of the target:

Error budget = 1 - SLO target

For a 99.5% SLO over 28 days:

Error budget = 0.5% of all requests over 28 days

In time terms, if your service receives 1 million requests per day, 0.5% of 28 million requests is 140,000 allowed failures over the window.

The error budget is the key concept that makes SLOs operationally useful. When you have budget remaining, you have room to take risk: ship features faster, run migrations, experiment. When the budget is nearly exhausted, the calculus changes: freeze risky releases, focus on reliability work, and understand why you burned through budget faster than expected.

Multi-Window, Multi-Burn-Rate Alerting

A single alert that fires when you have consumed your entire error budget is too late. By the time the window's worth of budget is gone, users have been affected for hours or days.

Burn rate alerts detect that you are consuming budget faster than the window allows — before the budget runs out. The approach defined in Google's SRE Workbook uses multiple alert windows and burn rate thresholds:

Page-level alerts (high urgency, immediate response):

• If burn rate exceeds 14.4x over the last 1 hour AND over the last 6 hours, page immediately. At 14.4x burn rate on a 28-day window, you exhaust the entire budget in 2 days. This is a production incident.

Ticket-level alerts (lower urgency, next business day response):

• If burn rate exceeds 6x over the last 6 hours AND over the last 3 days, create a ticket. At 6x, you exhaust the budget in about 4.5 days.
• If burn rate exceeds 3x over the last 3 days AND over the last 30 days, investigate during normal hours.

The two-window requirement for each alert tier prevents false positives from short traffic spikes. A 1-hour spike at 14.4x that resolves quickly will not trigger the alert if the 6-hour window looks normal.

The Error Budget Policy: What Actually Changes Behavior

An SLO without a policy is a dashboard metric. The policy defines what the team does at each burn rate threshold.

A minimal policy structure:

• Budget above 50%: operate normally. Feature work proceeds. Risk is acceptable.
• Budget between 10% and 50%: notify the team. Slow down high-risk changes. Investigate any elevated error rates.
• Budget below 10%: freeze feature deployments. Dedicate engineering capacity to reliability. Conduct a review of the last week's incidents.
• Budget exhausted: stop all non-critical changes. Escalate. Conduct a postmortem before resuming normal deployment cadence.

The policy needs a named owner — typically an engineering lead or on-call rotation — and buy-in from product management. If product can override the policy whenever a deadline is at stake, the SLO means nothing.

Wiring This Into Your Tools

In Prometheus and Grafana, recording rules for SLI ratios and pre-computed burn rate windows make alert evaluation efficient. A typical setup:

1. Define a recording rule for your SLI ratio, grouped by service and endpoint.
2. Define recording rules for burn rate ratios at 1h, 6h, 3d windows.
3. Write alert rules that combine the short and long window conditions.
4. Route high-burn alerts to pager, medium-burn alerts to a Slack channel or ticket system.

Most commercial platforms — Datadog, New Relic, Grafana Cloud, Nobl9 — have SLO management UIs that handle the math. The concepts above apply regardless of which tool you use.

Start Small and Iterate

Define SLOs for one or two critical services first. A checkout endpoint. A user authentication flow. Services where an outage has direct user impact. Get the alerting working. Run one or two actual incident responses where the SLO data influences decisions. Then expand.

Teams that try to define SLOs for every service at once usually end up with unmaintained YAML that no one looks at.

If you want help setting up SLOs, error budgets, and the alerting infrastructure around them for a real system, Clixo works with product engineering teams on exactly this.