Data Pipeline Monitoring and Alerting: A Production Checklist

A data pipeline that runs unmonitored is a liability. When it fails silently, downstream consumers — dashboards, reports, product features — continue operating on stale or missing data. By the time someone notices, the impact has already spread. The goal of pipeline monitoring is to know about failures before the business does.

This checklist covers what to monitor, how to alert, and the observability you need to debug problems quickly when they happen.

Why Pipeline Monitoring Is Different from Application Monitoring

Application monitoring detects errors and latency in user-facing systems. Pipeline monitoring adds a dimension that application monitoring misses: data correctness and freshness. A pipeline can succeed (exit code 0, no exceptions) and still produce wrong or stale data. Traditional uptime monitoring does not catch this. Pipeline observability requires checking not just that the pipeline ran, but what it produced.

Checklist: What to Monitor

Run Status and Failures

• Every pipeline run has a logged outcome — succeeded, failed, or timed out. No run should exit without a status record.
• Failures trigger an alert immediately — not on the next business day check-in. On-call engineers should know within minutes.
• Partial failures are not treated as successes — if a pipeline processes 10 tables and 2 fail, the run is not marked as succeeded with a warning. It is a failure.
• Retry behavior is logged — how many retries occurred, what error triggered each, whether the final retry succeeded.

Data Freshness

• Each destination table has a defined freshness SLA — for example, the orders table must be updated by 7 AM daily.
• Freshness is checked automatically — not by a human opening a dashboard. Query the max(updated_at) or equivalent column and alert if it exceeds the SLA window.
• Freshness alerts fire before the business impact — if the nightly load is supposed to finish by 7 AM and stakeholders need it for 9 AM reports, alert at 7:30 AM, not 9:01 AM.

Row Count and Volume

• Row counts are logged per run — rows extracted, rows transformed, rows loaded to destination.
• Significant deviations from historical volume trigger an alert — a load of zero rows is almost always wrong. A load that is 10x the historical average warrants investigation before the data is promoted.
• Extract count matches load count — rows should not silently disappear between extract and load.

Data Quality Metrics

• Quality check results are recorded — not just pass/fail, but which specific checks ran and which failed.
• Quality check failures fail the run — a check that logs a warning but allows the pipeline to continue is not a quality check.
• Trend data for quality metrics — null rates, duplicate rates, and referential integrity violation rates should be tracked over time so that gradual degradation is visible.

Schema Changes

• Schema changes in source systems are detected and alerted — not silently accepted or silently dropped.
• Breaking schema changes (renamed or dropped columns) fail the pipeline — non-breaking additions (new optional columns) can be logged without failing.

Pipeline Duration and SLA

• Runtime is logged for every run — baseline it over a few weeks to understand normal duration.
• Alert if a pipeline exceeds a duration threshold — a pipeline that normally runs in 10 minutes and is still running at 45 minutes may be hung, not just slow.
• SLA breach alerts fire proactively — if the pipeline has not finished within the expected window, alert before the missed deadline causes downstream impact.

Checklist: How to Alert

• Failure alerts go to a channel that is actively monitored — Slack, PagerDuty, or email to an on-call distribution list. Not just a log file.
• Alerts include actionable context — pipeline name, run ID, which task failed, the error message, a link to the run logs. Do not send "pipeline failed" with no further context.
• Alert severity is tiered — a pipeline that feeds a critical product feature warrants a page. A nightly analytics report for internal use warrants a Slack message.
• Alert routing is documented — who is responsible for which pipelines? On-call rotation should cover the pipelines, not just the application.
• Alerts are not noisy — a team that ignores alerts because too many are false positives is a team flying blind. Tune alert thresholds to eliminate false positives.

Checklist: Observability and Debugging

• Structured run logs are stored and queryable — not just printed to stdout. A pipeline_runs table with run ID, pipeline name, start time, end time, status, rows processed, and error message enables debugging without reading raw logs.
• Lineage is traceable — when a downstream table contains bad data, you can trace which pipeline run wrote it and what source data it came from.
• Historical run data is retained — at minimum 90 days. This enables trend analysis and post-incident investigation.
• A runbook exists for each pipeline — what does the pipeline do, what are common failure modes, who is responsible, what is the recovery procedure?

Tools Worth Knowing

• Orchestrator built-ins: Airflow, Prefect, and Dagster all provide run history, alerting, and basic observability out of the box. Use them.
• Elementary (dbt): Adds automated data quality monitoring and anomaly detection on top of dbt.
• Monte Carlo / Bigeye / Anomalo: Dedicated data observability platforms for teams that need warehouse-level monitoring across many tables.
• Datadog / Grafana: General observability tools that work for pipeline metrics if instrumented correctly.

A Practical Starting Point

If you are adding monitoring to an existing pipeline with no observability, start with these three:

1. Failure alert — fire a Slack message whenever a pipeline run fails.
2. Freshness check — query the destination table daily and alert if data is older than expected.
3. Structured run log — write a row to a pipeline_runs table for every run, including status and row counts.

These three changes catch the majority of real-world pipeline issues with minimal implementation effort. Build from there as the team's needs grow.

Pipelines that run without monitoring are technical debt that compounds quietly. If you are building a data platform and want monitoring designed in from the start, Clixo works with engineering teams on production data infrastructure.