Observability Production Readiness Checklist for Engineering Teams

Shipping to production is not the hard part. Knowing what is happening in production after you ship — and being able to act on that knowledge fast enough to matter — is the hard part. Teams that skip observability setup treat production as a black box until something breaks, and then scramble to add visibility while users are affected.

This checklist captures what a system needs before it is production-ready from an observability standpoint. Use it before your first production deploy and revisit it as the system grows.

Logging

• All services emit structured logs in JSON. Log entries are machine-parseable with consistent field names across the entire system.
• Every log entry includes a timestamp in UTC, a log level, the service name, and a message. No exceptions.
• Trace IDs and span IDs are included in every log entry emitted during request handling. This enables correlation between logs and distributed traces.
• Log levels are used correctly. Production is not flooded with debug output. Info-level logs represent significant business events, not routine operations.
• Sensitive data is not logged. No passwords, raw API keys, full card numbers, or complete authentication tokens appear in log output.
• Logs are shipped to a central aggregation system. No one is SSHing into individual hosts to read log files.
• A log retention policy is defined. You know how long each log level is retained and whether older logs are archived or deleted.
• You can query "all errors from service X in the last hour" in under 30 seconds.

Metrics

• The four golden signals are defined and dashboarded for each critical service: latency, traffic (throughput), errors (error rate), and saturation.
• Request latency is measured at meaningful percentiles: p50, p95, and p99. An average latency metric alone is not sufficient.
• Error rate is measured as a ratio (errors / total requests), not an absolute count. Absolute counts are misleading during traffic spikes or troughs.
• All HTTP routes, queue consumers, and scheduled jobs have their own metrics. Aggregate service-level metrics are not sufficient for debugging.
• Infrastructure metrics are collected: CPU, memory, disk I/O, network, and for containerized environments, pod restart counts.
• External dependency health is visible: database connection pool usage, queue depth, third-party API error rates.
• Dashboards have labels that explain what you are looking at. Panels have titles, units are specified (ms, req/s, percent), and axes are labeled.

Distributed Tracing

• All services are instrumented with a tracing library (OpenTelemetry recommended for vendor neutrality).
• Trace context propagates across all service boundaries, including synchronous HTTP/gRPC calls and asynchronous message queue operations.
• Custom spans exist for business-critical operations. Instrumentation is not limited to the auto-instrumented HTTP and database layers.
• Custom spans have meaningful attributes: entity IDs, amounts, counts, and other values that answer debugging questions.
• Sampling is configured. You are not sending 100% of traces to your backend at high throughput, and you are not dropping all traces from error cases.
• Traces for error cases are always retained. Your sampling policy preserves traces that include spans with error status.
• You can answer "where did this specific slow request spend its time?" using a trace. Walk through this manually before launch.

Alerting

• Alerts fire on symptoms (user-facing impact), not causes (internal metrics). An alert on error rate is a symptom. An alert on CPU at 80% is a cause.
• Every alert has a defined severity and a defined response procedure. What does the on-call engineer do when this alert fires?
• Alert thresholds are based on SLOs, not arbitrary numbers. If your SLO is 99.5% availability, the alert should be correlated with that budget.
• Alerts have been tested. You have verified that the alert fires when the condition is met, and does not fire when it is not.
• Alert fatigue is actively managed. If an alert fires more than a few times per week without resulting in meaningful action, it is either misconfigured or not worth having.
• Pager alerts are reserved for issues that require immediate human action. Informational alerts belong in a Slack channel or dashboard, not a pager.

Error Tracking

• Application exceptions are captured and routed to an error tracking tool (Sentry, Rollbar, or similar).
• Errors are grouped by fingerprint, not by individual occurrence. You can see "this error occurred 47 times in the last hour" rather than 47 separate entries.
• New error types trigger a notification. You know when your system throws a type of error it has never thrown before.
• Errors include the trace ID where applicable, so you can jump from an error report to the full distributed trace.
• Error resolution is tracked. When a bug is fixed, you verify that the error rate for that fingerprint drops to zero.

On-Call and Incident Process

• There is a defined on-call rotation with named coverage and escalation paths.
• The on-call engineer has access to all observability tools from their laptop and phone.
• A runbook or response guide exists for the most likely alert types. Not for every possible failure — for the ones that are predictable.
• Incident postmortems are conducted after significant events and result in concrete action items, not just documentation.
• Mean time to detection (MTTD) and mean time to resolution (MTTR) are tracked. Even informally. If you do not measure these, you cannot improve them.

Before You Check This Off

Walk through a simulated incident before going live. Pick a failure mode — a downstream dependency returning errors, a slow database query, a spike in 500s — and verify that your observability setup would allow an engineer who was not involved in building the system to identify the problem within a reasonable time.

If there are gaps, fill them before launch. Observability that is added under incident pressure is observability added at the worst possible time.

If your team needs help getting a system production-ready from an observability standpoint, Clixo builds and ships production-grade systems with this foundation included.