# The Four Golden Signals: A Monitoring Guide for Production Services

> How to implement the four golden signals monitoring framework — latency, traffic, errors, and saturation — to detect real problems before users do.

- **Published:** 2025-06-28
- **Author:** Clixo
- **Reading time:** 6 min read
- **Tags:** monitoring, golden-signals, metrics, alerting, sre, observability
- **Canonical URL:** https://clixo.sh/blog/four-golden-signals-monitoring-guide

Most teams that struggle with monitoring have the same underlying problem: they are measuring the wrong things. CPU utilization, memory usage, and disk I/O tell you about the machine. They do not tell you whether users can complete what they came to do. The four golden signals framework fixes this by centering monitoring on user-facing behavior rather than infrastructure state.

The four golden signals were defined in Google's Site Reliability Engineering book and have become one of the most widely applied frameworks for production monitoring. Here is what they are, how to implement each one, and how to build useful alerts from them.

```mermaid
flowchart LR
  T["Traffic (req/s)"] --> CTX["Context for other signals"]
  L["Latency (p99)"] --> ALERT["Alert engine"]
  E["Error rate"] --> ALERT
  S["Saturation (capacity %)"] --> ALERT
  CTX --> ALERT
  ALERT --> OC["On-call response"]
```

## The Four Golden Signals

### Latency

Latency is the time it takes to service a request. Track it separately for successful requests and failed requests — a fast error is not the same as fast service, and lumping them together can mask both problems.

Measure latency as a distribution, not an average. Averages hide the tail. If 95% of requests take 50ms and 5% take 5 seconds, the average might look acceptable while a meaningful fraction of users are experiencing a poor product. Track p50, p95, and p99. Alert on p99 for user-facing services — it represents the worst experience your real users are having.

In Prometheus, use a histogram metric:

```yaml
http_request_duration_seconds_bucket{le="0.1"} 945
http_request_duration_seconds_bucket{le="0.3"} 1203
http_request_duration_seconds_bucket{le="1.0"} 1247
http_request_duration_seconds_count 1251
```

Query p99:
```
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
```

Set latency SLOs based on what your users actually experience as acceptable. Work backwards from the user's perspective, not from what the system currently delivers.

### Traffic

Traffic is the demand being placed on your system. For HTTP services, this is requests per second. For streaming systems, it is records processed per second. For batch jobs, it is jobs per hour.

Traffic is not an alerting signal by itself — more traffic is usually good. What traffic provides is context for every other signal. An error rate of 2% at 100 req/s is very different from 2% at 10,000 req/s. A latency spike during a traffic surge has a different likely cause than a latency spike during normal traffic.

Track traffic segmented by endpoint and service so you can correlate changes. A sudden traffic increase on one endpoint that does not show on others may indicate a traffic source problem (bot traffic, misconfigured retry logic, a marketing campaign you were not told about) rather than an application problem.

### Errors

Errors are the rate of requests that fail. Define failure explicitly for your service — it is usually any 5xx HTTP response, but may also include 4xx responses in specific cases (429 rate limit errors, for example, may indicate a capacity problem rather than client error).

Measure error rate as a ratio, not a count:

```
error_rate = errors / total_requests
```

A count of 100 errors means different things at 1,000 requests per second versus 10 requests per second. The ratio is what matters for alerting.

Also distinguish between different error categories. A connection refused error from a downstream dependency has a different remediation path than an unhandled exception in your own code. Tag errors with their type — timeout, dependency_error, application_error — so that alerts and dashboards can be specific.

```
rate(http_requests_total{status=~"5.."}[5m])
/
rate(http_requests_total[5m])
```

Watch for partial failures. A service that returns 200 with an error payload embedded in the response body will not show up in HTTP error rate metrics. If your API has custom error handling, make sure your metrics capture application-level failures, not just HTTP status codes.

### Saturation

Saturation measures how full your service is. It answers the question: how much capacity do you have left? A service that is 90% saturated is much more fragile than one at 40% — small traffic increases can push it into failure.

Saturation is the most varied of the four signals because what "full" means depends on the service:

- For CPU-bound services: CPU utilization and run queue depth
- For memory-bound services: heap usage as a percentage of limit
- For I/O-bound services: disk throughput as a percentage of maximum, or IOPS utilization
- For database services: connection pool utilization, lock wait time
- For queue-based systems: queue depth and processing lag

The critical insight with saturation is that you want to alert on it before you hit the limit, not after. Alerting at 90% of capacity gives you time to respond. Alerting at 100% means you are already in an incident.

A particularly useful saturation signal for containerized environments is the ratio of current resource usage to the configured limit — whether a container is approaching its CPU throttle threshold or its memory limit.

## Building Alerts From the Four Golden Signals

The golden signals are most useful when they drive alerts. A minimal alert set for a production service:

**Latency alert:**
```yaml
- alert: HighRequestLatency
  expr: |
    histogram_quantile(0.99,
      rate(http_request_duration_seconds_bucket[5m])
    ) > 1.0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "p99 latency above 1s on {{ $labels.job }}"
```

**Error rate alert:**
```yaml
- alert: HighErrorRate
  expr: |
    rate(http_requests_total{status=~"5.."}[5m])
    /
    rate(http_requests_total[5m]) > 0.05
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Error rate above 5% on {{ $labels.job }}"
```

**Saturation alert (connection pool example):**
```yaml
- alert: DatabaseConnectionPoolSaturation
  expr: |
    db_connection_pool_used / db_connection_pool_size > 0.85
  for: 3m
  labels:
    severity: warning
  annotations:
    summary: "DB connection pool above 85% on {{ $labels.job }}"
```

Adjust thresholds based on your SLOs and what is normal for each service. The numbers above are starting points, not universal truths.

## Common Gaps

**Only tracking aggregated signals, not per-endpoint.** A spike in aggregate error rate could come from one problematic endpoint while the rest of the service is healthy. Always segment golden signals by the most granular unit that makes sense — endpoint, operation type, or customer tier.

**No baseline.** Alerting on absolute thresholds without knowing what is normal leads to either alert fatigue (too sensitive) or missed incidents (too conservative). Track signals over time to establish baselines before setting alert thresholds.

**Skipping traffic context in runbooks.** When an on-call engineer responds to an error rate alert, the first thing they should check is whether traffic changed. Build this into your runbooks and dashboards so that context is visible at a glance.

The four golden signals are not a complete observability solution — they are a monitoring layer. Traces and logs provide the depth to debug what the golden signals detect. Used together, they give a team both the early warning system and the diagnostic capability they need in production.

If you want help implementing the golden signals framework and wiring it into your alerting stack, [Clixo works with product teams on exactly this kind of production engineering work](https://clixo.sh/#contact).

---

Clixo · 1141 W Bryn Mawr Ave, Itasca, IL 60143, US · [hello@clixo.sh](mailto:hello@clixo.sh)
[Start a build](https://clixo.sh/#contact) · [All services](https://clixo.sh/services) · [Agent guide (llms.txt)](https://clixo.sh/llms.txt)