Structured Logging Best Practices for Production Systems

Most teams have logging. Few teams have logs they can actually use. When an incident fires at 2 a.m., the difference between resolving it in twenty minutes and resolving it in three hours often comes down to whether your logs tell a story or just record that something happened.

Structured logging is the practice of emitting log entries as machine-parseable records — most commonly JSON — with a consistent schema. This guide covers the practices that make that investment pay off in production.

Why Structured Logging Matters in Production

Unstructured logs look like this:

ERROR: Failed to process order for user 4821 - payment declined

Structured logs look like this:

{
  "timestamp": "2025-06-05T09:14:33.421Z",
  "level": "error",
  "service": "order-service",
  "trace_id": "7d3f1a2b9c4e8f0d",
  "span_id": "a3c7b1e2",
  "msg": "payment declined",
  "user_id": 4821,
  "order_id": "ord_9f2k1",
  "amount_cents": 4999,
  "currency": "usd",
  "payment_provider": "stripe",
  "error_code": "card_declined"
}

The unstructured version is readable to a human who already knows the context. The structured version is queryable by any engineer, filtered by user_id, grouped by error_code, and automatically correlated with the trace that shows what happened before and after. That is the difference.

Structured Logging Best Practices

1. Use a Consistent Schema Across All Services

Define a baseline set of fields that every log entry emits, regardless of which service writes it:

• timestamp — ISO 8601, always UTC
• level — one of debug, info, warn, error, fatal
• service — the service name, matching your deployment identifier
• trace_id and span_id — critical for correlating logs with distributed traces
• msg — a short, static string describing the event (not a formatted sentence)
• env — production, staging, or development

Beyond the baseline, each service adds domain-specific fields. The baseline is what makes logs queryable across your entire fleet.

2. Keep the msg Field Static

This is one of the most commonly violated rules. The msg field should be a short, human-readable constant string — not a formatted sentence.

Wrong:

"msg": "Failed to charge user 4821 for order ord_9f2k1: card declined"

Right:

"msg": "payment declined"

Dynamic values belong in their own fields. Static messages allow you to group all instances of an event type in your log query tool without writing complex regex patterns. Every log aggregation tool handles field-level filtering efficiently; string matching against interpolated messages is fragile and slow.

3. Log Events, Not States

Each log entry should represent something that happened, not a description of the current system state. "Order processing started" is an event. "Order is being processed" is a state description. The event form is easier to sequence, correlate, and count.

4. Choose the Right Log Level

Teams that log everything at info end up with the same problem as teams that log nothing — signal buried in noise.

• debug — fine-grained detail useful only during development or when actively diagnosing a known issue. Should be off in production by default, configurable per-service without a restart.
• info — significant business events: request received, job completed, user authenticated.
• warn — something unexpected happened but the system recovered or degraded gracefully.
• error — an operation failed and a human should investigate.
• fatal — the process cannot continue.

If your production logs are mostly info level output from routine operations, you are building a haystack.

5. Include Trace Context in Every Log Entry

Every log line emitted during the handling of a request should carry the trace_id from that request's distributed trace. This single field is what turns logs from isolated records into a correlated story.

Most tracing libraries — OpenTelemetry, Jaeger clients, Datadog's tracer — expose the current trace context and can inject it into your logging library's context automatically. If you are using a language with middleware patterns, inject the trace context at the entry point and propagate it through every layer.

6. Do Not Log Sensitive Data

Logs are often shipped to third-party log aggregation services, retained for extended periods, and accessible to a wide range of team members. Before a field goes into a log entry, ask whether it would be safe to put it in a spreadsheet and share it with your entire company. That is roughly the security model of most log pipelines.

Concretely: never log passwords, full credit card numbers, raw API keys, or complete authentication tokens. Log partial values (last four digits of a card, first eight characters of a token) or opaque identifiers when you need traceability without exposing the sensitive value.

7. Sample High-Volume Debug Logs

In high-throughput systems, emitting a structured log entry for every low-level operation — every cache lookup, every function entry — becomes expensive in both CPU and log ingestion cost. Use sampling for debug-level logs. A common pattern is to emit debug logs for a random percentage of requests, or to emit them only when the trace for that request has been flagged (for example, when it exceeded a latency threshold).

8. Test Your Log Output

Logs are code. They should be tested like code. Write tests that verify your service emits the expected fields in the expected format for key events. This prevents silent regressions where a refactor breaks the field name that your on-call alert queries for.

9. Define a Retention Policy Before You Ship

Log storage costs accumulate quickly. Before you go to production, decide:

• How long to retain each log level (debug logs may need only 24-48 hours; error logs may need 90 days)
• Whether to archive to cold storage after a retention window
• Who can query production logs and under what conditions

Defining this early prevents the common situation where teams are paying for months of verbose debug logs they never query.

Getting This Right From the Start

Retrofitting structured logging into an existing system is tedious work — every service, every log statement, every integration. Building it correctly from the start takes a fraction of the effort and pays dividends every time something goes wrong in production.

If you are building a new system or modernizing an existing one, Clixo can help you design and implement an observability foundation that works — from log schema design to trace instrumentation to alerting strategy.