# 10 Common RAG Pipeline Mistakes That Kill Production Quality > The most common RAG pipeline mistakes teams make in production — from bad chunking and missing reranking to invisible ingestion failures and no evaluation harness. - **Published:** 2025-11-09 - **Author:** Clixo - **Reading time:** 5 min read - **Tags:** rag, ai-engineering, production, common-mistakes - **Canonical URL:** https://clixo.sh/blog/common-rag-pipeline-mistakes-production RAG pipelines are easier to prototype than they are to get right. A working demo requires an embedding model, a vector database, and a prompt. A production system that actually answers questions accurately, degrades gracefully, and holds up when documents change is a different problem. Most teams discover this the hard way. Here are ten mistakes that recur across RAG deployments, along with what to do instead. ## 1. Blaming the LLM When Retrieval Is the Problem When an AI assistant returns a wrong or irrelevant answer, the instinct is to tweak the prompt or switch to a larger model. Most of the time, the problem is upstream: the retriever did not surface the right document. Before changing anything else, check what chunks the retriever actually returned for a failing query. Add logging that records retrieved chunk IDs and scores alongside every response. Retrieval quality, not generation quality, is the primary determinant of RAG answer quality. ## 2. Skipping the Evaluation Harness Teams that build RAG without an evaluation set cannot tell if a change made things better or worse. They make changes based on anecdote. They ship regressions without realizing it. Build a labeled query set before you tune anything. Even 50 manually reviewed query-answer pairs lets you compute Recall@5 and catch regressions. This is not optional infrastructure — it is how you know whether the system works. ## 3. Using Fixed-Size Chunking Without Checking Document Structure Fixed-size chunking is a fine default, but not if your documents contain tables, PDFs with multi-column layouts, or structured data like JSON. Raw PDF extraction followed by fixed-size splitting produces chunks that contain half a table header, some boilerplate footer text, and a fragment of actual content. The embedding of that chunk is noise. Pre-process documents before chunking. Extract structure, remove headers and footers, handle tables separately, and validate chunk content before indexing. ## 4. Not Handling Document Updates Most teams index their corpus once and move on. In production, documents change. Outdated chunks produce confident wrong answers, which is worse than no answer. Track a hash or last-modified timestamp for every source document. On update, delete all associated chunks from the vector index before inserting new ones. Run a periodic reconciliation job that compares source state to index state and queues stale chunks for re-ingestion. ## 5. Missing Metadata on Chunks Chunks without metadata are hard to filter, impossible to audit, and difficult to remove when the source document changes. Every chunk should carry at minimum: source document ID, chunk index within the document, section or page, and ingestion timestamp. Metadata is also how you implement multi-tenant isolation — ensuring that queries from one user or organization never retrieve content from another. This is not a nice-to-have; it is a security requirement for any multi-tenant deployment. ## 6. Ignoring Retrieval Noise Retrieval noise occurs when irrelevant chunks make it into the context window. The LLM sees ten chunks, two of which are relevant. The other eight pull the generation toward irrelevance or contradiction. Noise is a retrieval precision problem. Solutions include: tighter similarity score thresholds, metadata pre-filtering, and reranking. A cross-encoder reranker applied to the initial retrieval set is the most reliable way to reduce noise before chunks reach the LLM. ## 7. No Reranking Step Initial ANN retrieval optimizes for recall — it finds candidates that might be relevant. Reranking optimizes for precision — it identifies which candidates are actually relevant. Running retrieval without reranking is like running a search engine that returns 100 results and stops, trusting the user to find the right one. Add a cross-encoder reranker between retrieval and generation. Retrieve 20-50 candidates, rerank, pass the top 5 to the LLM. This alone is one of the highest-leverage improvements in any RAG pipeline. ```mermaid flowchart LR A["User Query"] --> B["Embed Query"] B --> C["ANN Retrieval"] C --> D["Cross-Encoder Reranker"] D --> E["Top 5 Chunks"] E --> F["LLM Generation"] F --> G["Answer"] ``` ## 8. Ignoring Ingestion Failures When document ingestion fails silently, the LLM still returns answers — just not ones grounded in the missing documents. Unlike retrieval failures, which produce wrong answers the user can sometimes recognize, ingestion failures are invisible: the knowledge simply does not exist in the index. Log every ingestion job. Track success and failure rates per document. Alert on ingestion failure rates above a threshold. Implement a dead-letter queue for failed documents so they can be retried without re-running the full pipeline. ## 9. Using One Embedding Model for Everything The embedding model you use matters for retrieval quality, and different models have meaningfully different strengths. A general-purpose embedding model trained on web text may underperform a domain-adapted model on legal, medical, or financial documents. Multilingual corpora need multilingual embedding models. Evaluate multiple embedding models against your labeled query set before committing. The right model is determined by your corpus and query distribution, not by benchmark leaderboard position. ## 10. Treating RAG as a One-Time Build RAG systems degrade over time. New documents are added in formats the chunker was not designed for. User query distributions shift. Embedding models are updated and older vectors become stale relative to new model outputs. Treat your RAG pipeline as a running system, not a completed project. Schedule periodic evaluations against your labeled set. Monitor retrieval latency, chunk count per query, and answer quality feedback. Set up alerts for evaluation metric drops. This is what separates production AI systems from prototypes that were never productionized. If you are building a RAG system that needs to work reliably beyond the demo stage, [start a conversation with Clixo](https://clixo.sh/#contact). We build production AI pipelines for engineering teams who need measurable results. --- Clixo · 1141 W Bryn Mawr Ave, Itasca, IL 60143, US · [hello@clixo.sh](mailto:hello@clixo.sh) [Start a build](https://clixo.sh/#contact) · [All services](https://clixo.sh/services) · [Agent guide (llms.txt)](https://clixo.sh/llms.txt)