# RAG Production Readiness Checklist: 25 Things to Verify Before Launch > A comprehensive RAG production readiness checklist covering ingestion, retrieval, evaluation, observability, security, and multi-tenancy before you go live. - **Published:** 2025-11-17 - **Author:** Clixo - **Reading time:** 5 min read - **Tags:** rag, production-readiness, checklist, ai-engineering - **Canonical URL:** https://clixo.sh/blog/rag-production-readiness-checklist You have a RAG prototype that produces reasonable answers in your test environment. Before you open it to users, there is a meaningful gap to close. Most post-launch incidents in RAG systems trace back to things that were overlooked in the pre-launch period — not because teams are careless, but because there is no canonical checklist to work from. This checklist covers the production readiness criteria that matter. Work through it before launch, not after. ```mermaid flowchart LR A["Raw Documents"] --> B["Ingestion Pipeline"] B --> C["Vector Index"] C --> D["Hybrid Retrieval"] D --> E["Reranker"] E --> F["LLM Generation"] F --> G["User Response"] H["Observability"] -.-> D H -.-> F ``` ## Ingestion Pipeline - **Document pre-processing is validated.** Raw documents (PDFs, HTML, Office files) are parsed, cleaned, and structured before chunking. Tables, headers, and footers are handled explicitly, not ignored. - **Chunking strategy is appropriate for document type.** Fixed-size chunking has been evaluated against your corpus. If semantic or hierarchical chunking was chosen, the rationale is based on evaluation data, not assumption. - **Every chunk carries metadata.** At minimum: source document ID, chunk index, section or page, document type, and ingestion timestamp. Multi-tenant deployments include tenant or organization ID. - **Ingestion failures are logged and alerted.** Silent ingestion failures produce wrong answers with no visible error. Failed documents go to a dead-letter queue and trigger an alert. - **Document updates trigger re-indexing.** A mechanism exists to detect when source documents change, delete stale chunks, and re-ingest. This is tested before launch, not after the first stale-answer complaint. - **Full re-indexing is scripted and tested.** The ability to re-index the full corpus from scratch is documented, scripted, and has been run at least once against production data. ## Retrieval - **Hybrid search is enabled.** Dense and sparse retrieval are combined. Pure vector search is not used as the sole retrieval method unless there is a measured reason. - **Metadata pre-filtering is used where applicable.** Filters are applied inside the ANN search, not post-retrieval. Multi-tenant queries are filtered by tenant at retrieval time. - **A reranker is in the retrieval pipeline.** Top-50 candidates are reranked with a cross-encoder before the top-5 are passed to the LLM. - **Retrieval latency is measured under load.** P50 and P99 retrieval latency have been benchmarked at the expected query volume. Latency budget is allocated between retrieval, reranking, and generation. - **Similarity score thresholds are set.** A minimum score threshold filters out low-confidence retrievals. Queries that return no confident matches return an explicit "I don't know" rather than a hallucinated answer. ## Evaluation - **A labeled evaluation set exists.** At minimum 50-100 query-answer pairs drawn from real or realistic user queries. - **Recall@5 and MRR are computed and baselined.** You know what your retrieval quality numbers are before launch, and they are acceptable. - **Adversarial queries are included in the evaluation set.** The set includes out-of-scope queries, ambiguous queries, queries with no correct answer in the corpus, and queries using synonyms or paraphrases. - **Evaluation runs are automated.** Running the evaluation set is a single command. It is run before every significant pipeline change. - **Answer quality is sampled manually.** Automated retrieval metrics do not capture generation quality. A manual review of sampled LLM responses is part of pre-launch validation. ## Observability - **Every query logs retrieved chunk IDs and scores.** When an answer is wrong, you can immediately identify whether the failure was retrieval or generation. - **LLM calls are logged with latency and token counts.** Cost and latency are visible per query. Token budget overruns are detectable. - **Alerts are configured for retrieval metric degradation.** If Recall@5 drops below threshold in production, an alert fires before users complain. - **A mechanism exists to collect user feedback.** Explicit thumbs up/down or implicit dwell-time signals are collected and routable to manual review. ## Security and Multi-Tenancy - **Data isolation is enforced at retrieval time.** Queries from tenant A cannot retrieve documents belonging to tenant B. This is tested with adversarial cross-tenant queries. - **PII handling in document ingestion is defined.** If source documents contain PII, there is a documented policy for whether PII is indexed, how it is handled, and how it is deleted on request. - **LLM prompt injection is considered.** If user queries are inserted into prompts, there is a guardrail or review process for adversarial prompt injection attempts. ## Operational Readiness - **Rollback procedure is documented.** If a retrieval pipeline change degrades quality, you can roll back to the previous index state. - **On-call ownership is clear.** When the RAG pipeline fails at 2am, someone is responsible for diagnosis and recovery, and they have access to the logs and metrics they need. A checklist is a starting point, not a substitute for judgment. The items above represent the baseline — the things that, if missing, will cause visible failures in the first weeks of production. There are always application-specific concerns on top of these. If your team is preparing a RAG system for production and wants an experienced review before launch, [reach out to Clixo](https://clixo.sh/#contact). We help engineering teams ship AI systems that hold up. --- Clixo · 1141 W Bryn Mawr Ave, Itasca, IL 60143, US · [hello@clixo.sh](mailto:hello@clixo.sh) [Start a build](https://clixo.sh/#contact) · [All services](https://clixo.sh/services) · [Agent guide (llms.txt)](https://clixo.sh/llms.txt)