# SLO Burn Rate Alerting in Prometheus: Multi-Window Setup Guide

> How to implement SLO burn rate alerting in Prometheus using multi-window, multi-burn-rate rules — the method that reduces alert fatigue while catching real incidents.

- **Published:** 2025-06-21
- **Author:** Clixo
- **Reading time:** 6 min read
- **Tags:** prometheus, alerting, slo, burn-rate, sre, grafana
- **Canonical URL:** https://clixo.sh/blog/slo-burn-rate-alerting-prometheus

Standard threshold alerts fire when a metric crosses a number you picked somewhat arbitrarily. They miss gradual degradation and create false positives during traffic bursts. SLO burn rate alerting is a different model: it measures how fast you are consuming your error budget and fires based on trajectory, not on absolute values. The result is alerts that mean something when they fire and that stay quiet when the system is fine.

This guide walks through implementing multi-window, multi-burn-rate alerting in Prometheus, based on the approach described in Google's SRE Workbook.

## The Concept: Burn Rate

An error budget is the amount of unavailability an SLO permits. For a 99.5% availability SLO over a 28-day window, the error budget is 0.5% of all requests — the equivalent of about 3.6 hours of complete downtime spread across the window.

A burn rate of 1x means you are consuming the error budget exactly as fast as the window allows. At 1x, you will exhaust the budget precisely when the 28-day window closes. A burn rate of 14.4x means you will exhaust the entire 28-day budget in 2 days. That is a production incident.

Burn rate alerting fires based on burn rate, not on error rate. This matters because:

- A 5% error rate on a service with a 99% SLO is burning budget 5x faster than allowed. That is serious.
- A 0.3% error rate on a service with a 99.9% SLO is burning budget 3x faster than allowed. That also needs attention.

The same absolute error rate can be fine or catastrophic depending on your SLO target.

## Why Multi-Window Alerting

A single-window burn rate alert — "fire if burn rate exceeds 14.4x over the last hour" — has a precision problem. A short traffic spike with a bad error rate can push the 1-hour window above the threshold and fire the alert, even if the issue resolves within minutes. The alert fires, the on-call engineer wakes up, finds nothing wrong. Alert fatigue follows.

Multi-window alerting requires that both a short window AND a longer window show elevated burn rate simultaneously. The short window provides sensitivity — it detects problems quickly. The long window provides specificity — it confirms the problem is not a brief spike.

Both windows must exceed their thresholds for the alert to fire.

```mermaid
flowchart TD
  M[HTTP Metrics] --> RR[Recording Rules]
  RR --> BR["Burn Rate per Window"]
  BR --> C1{"1h and 6h above 14.4x?"}
  BR --> C2{"6h and 3d above 6x?"}
  BR --> C3{"3d above 3x?"}
  C1 -->|Yes| P["Critical: Page On-Call"]
  C2 -->|Yes| T["Warning: Create Ticket"]
  C3 -->|Yes| N["Info: Notify Engineering"]
```

## Implementing the Recording Rules

Before writing alert rules, create recording rules that pre-compute the metrics you need. This makes alert evaluation fast and avoids repeated computation.

Define your SLI first. For an HTTP service measuring availability:

```yaml
# prometheus/rules/sli.yml
groups:
  - name: sli.rules
    rules:
      - record: job:http_requests:rate5m
        expr: |
          sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_errors:rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))

      - record: job:http_error_ratio:rate5m
        expr: |
          job:http_errors:rate5m / job:http_requests:rate5m
```

Now define the burn rate recording rules for each window. The burn rate is the error ratio divided by the error budget (1 - SLO target). For a 99.5% SLO, the error budget ratio is 0.005:

```yaml
      - record: job:http_burn_rate:1h
        expr: |
          (
            sum by (job) (rate(http_requests_total{code=~"5.."}[1h]))
            /
            sum by (job) (rate(http_requests_total[1h]))
          ) / 0.005

      - record: job:http_burn_rate:6h
        expr: |
          (
            sum by (job) (rate(http_requests_total{code=~"5.."}[6h]))
            /
            sum by (job) (rate(http_requests_total[6h]))
          ) / 0.005

      - record: job:http_burn_rate:3d
        expr: |
          (
            sum by (job) (rate(http_requests_total{code=~"5.."}[3d]))
            /
            sum by (job) (rate(http_requests_total[3d]))
          ) / 0.005
```

Adjust `0.005` to match your SLO target. For 99.9%, use `0.001`. For 99%, use `0.01`.

## Writing the Alert Rules

Three alert tiers cover the important operating conditions:

```yaml
# prometheus/rules/slo-alerts.yml
groups:
  - name: slo.alerts
    rules:

      # Critical: high burn rate — page immediately
      # At 14.4x, budget exhausts in 2 days
      - alert: HighErrorBudgetBurnRate
        expr: |
          job:http_burn_rate:1h > 14.4
          and
          job:http_burn_rate:6h > 14.4
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error budget burn rate on {{ $labels.job }}"
          description: >
            Service {{ $labels.job }} is burning error budget at
            {{ $value | humanize }}x the allowed rate.
            At this rate, the 28-day budget will be exhausted in
            approximately {{ printf "%.1f" (48 | divf $value) }} hours.

      # Warning: elevated burn rate — ticket and investigate
      # At 6x, budget exhausts in ~4.5 days
      - alert: ElevatedErrorBudgetBurnRate
        expr: |
          job:http_burn_rate:6h > 6
          and
          job:http_burn_rate:3d > 6
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Elevated error budget burn rate on {{ $labels.job }}"
          description: >
            Service {{ $labels.job }} is burning error budget at
            {{ $value | humanize }}x the allowed rate.
            Investigate and address before budget is exhausted.

      # Low: slow burn — review at next standup
      # At 3x, budget exhausts in ~9 days
      - alert: SlowErrorBudgetBurn
        expr: |
          job:http_burn_rate:3d > 3
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "Slow error budget burn on {{ $labels.job }}"
          description: >
            Service {{ $labels.job }} has been burning error budget
            at {{ $value | humanize }}x sustained over 3 days.
            Review and address before next SLO window.
```

The `for` duration on each alert adds an additional confirmation window. The alert must be true for the full `for` duration before firing. This further reduces false positives from brief spikes.

## Routing Alerts to the Right Destinations

In Alertmanager, route by severity:

```yaml
# alertmanager/config.yml
route:
  receiver: default
  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical
    - match:
        severity: warning
      receiver: slack-oncall
    - match:
        severity: info
      receiver: slack-engineering
```

Critical alerts page the on-call engineer immediately. Warning alerts go to the on-call Slack channel for the next person to see. Info alerts go to the general engineering channel for visibility.

## Visualizing Error Budget Consumption in Grafana

Add a panel to your SRE dashboard showing:

- **Error budget remaining** — current budget consumed as a percentage
- **Burn rate over time** — a time series of burn rate at each window
- **Projected budget exhaustion** — if burn rate stays at current levels, when does the budget run out?

The projected exhaustion calculation:

```
projected_exhaustion_hours = 28 * 24 / current_burn_rate
```

This gives your on-call team an immediate sense of urgency without requiring mental arithmetic.

## Adapting This to Your SLO Mix

Most services need at least two SLOs — one for availability and one for latency. Repeat the recording rule and alert pattern for each SLI, using separate metric names and separate error budget values.

For latency SLOs, the "good events" denominator changes: instead of non-5xx responses, you are measuring requests that completed below your latency threshold:

```yaml
- record: job:http_fast_requests:rate1h
  expr: |
    sum by (job) (rate(http_request_duration_seconds_bucket{le="0.3"}[1h]))
    /
    sum by (job) (rate(http_request_duration_seconds_count[1h]))
```

The burn rate calculation and alert structure remain the same.

If you are setting up SLO-based alerting in Prometheus or Grafana and want to get it right without iterating through false positive fires, [Clixo can design and implement the setup for your services](https://clixo.sh/#contact).

---

Clixo · 1141 W Bryn Mawr Ave, Itasca, IL 60143, US · [hello@clixo.sh](mailto:hello@clixo.sh)
[Start a build](https://clixo.sh/#contact) · [All services](https://clixo.sh/services) · [Agent guide (llms.txt)](https://clixo.sh/llms.txt)