RAG Retrieval Quality: 8 Best Practices That Actually Move the Needle

Most RAG systems start with reasonable retrieval and slowly accumulate technical debt that nobody measures. The LLM answers look plausible. Evaluation is informal. Then a stakeholder asks why the system returned a confident wrong answer, and the post-mortem traces back to a retrieval failure that nobody caught.

Retrieval quality is the primary determinant of RAG output quality. Improving it is not glamorous — it involves evaluation harnesses, metadata schemas, and a lot of manual labeling. Here are eight practices that consistently produce results.

1. Define Your Retrieval Quality Metrics Before Indexing Anything

You cannot improve what you do not measure. Before you index a single document, define:

• Recall@K: Does the correct document appear in the top K retrieved results? K=5 is a reasonable starting point.
• MRR (Mean Reciprocal Rank): How high in the result list does the correct chunk appear on average?
• Context relevance: Of the chunks retrieved, what fraction are actually relevant to the query?

Build a small labeled evaluation set — 50 to 200 query-answer pairs drawn from real user questions. This set becomes your regression test every time you change chunking, embedding models, or retrieval parameters.

2. Use Hybrid Search as the Default Retrieval Architecture

Dense vector search alone has a known weakness: it struggles with exact keyword matches. Model names, product codes, error strings, proper nouns, and technical identifiers all tend to get semantically smeared across many similar vectors. A query for "ERR_CERT_COMMON_NAME_INVALID" will not reliably surface the document containing that exact string using dense search alone.

Combining BM25 (sparse, keyword-based) with dense embedding retrieval using reciprocal rank fusion produces measurably better recall across most corpora. This is now a supported mode in all major vector databases and should be your baseline, not an optimization.

3. Apply Metadata Filtering at Query Time, Not Post-Retrieval

Every chunk should carry structured metadata: source document ID, section, document type, date, and any tenant or product identifiers relevant to your application. Filters applied inside the ANN search (pre-filtering) are dramatically faster and more accurate than filtering the top-K results after retrieval.

If you are filtering post-retrieval, you are paying the compute cost of retrieving 100 chunks to surface 5 useful ones. Most purpose-built vector databases support payload-indexed pre-filtering. Use it.

4. Separate Retrieval Chunks from Generation Context

A common RAG best practice that improves answer quality without touching retrieval: retrieve at small chunk granularity but pass larger parent chunks to the LLM. Small chunks improve retrieval precision because they are semantically focused. But a 128-token chunk rarely gives the LLM enough context to produce a complete, accurate answer.

The hierarchical or "parent document" retriever pattern solves this: retrieve small, generate from large. Implement it by storing a parent_id reference in chunk metadata and fetching the parent when a child chunk is selected.

5. Preprocess Queries Before Embedding

The query that a user types is often a poor retrieval signal as-is. Users ask short, ambiguous questions. They use informal language or abbreviations that the embedding model did not see during training.

Query preprocessing techniques that reliably improve recall:

• HyDE (Hypothetical Document Embeddings): Use the LLM to generate a hypothetical answer document, then embed that document for retrieval. This embeds the expected answer shape, not the question shape, and often surfaces better matches.
• Query expansion: Ask the LLM to generate 2-3 alternative phrasings of the query and retrieve against all of them, then deduplicate and rerank.
• Query decomposition: For complex multi-part questions, decompose into sub-queries, retrieve against each, and merge results.

6. Add a Reranker After Initial Retrieval

Initial retrieval is approximate. Reranking is precise. A cross-encoder reranker (a model that scores each query-chunk pair jointly, rather than comparing independent embeddings) consistently outperforms embedding similarity as a final ranking signal.

The typical pattern: retrieve top-20 or top-50 chunks with your vector index, then rerank with a cross-encoder and pass the top-5 to the LLM. The retrieval step favors recall; the reranking step favors precision. Cohere Rerank and open-source cross-encoders from the sentence-transformers library are common choices.

7. Handle Document Updates Proactively

Most RAG implementations treat indexing as a one-time event. In production, documents change. Prices update. Policies change. Outdated chunks in your index are worse than missing chunks — they produce confident, wrong answers.

Implement a document versioning strategy:

• Track a hash or last-modified timestamp per source document
• On update, delete all chunks from the old version before inserting new ones
• For large corpora, run a nightly reconciliation job that compares source hashes against index metadata

8. Test Adversarially, Not Just on Sunny-Day Queries

Retrieval evaluation sets built from ideal user queries overestimate production quality. Add adversarial cases:

• Queries that should return no result (and your system should say so)
• Queries that are ambiguous between two valid answers
• Queries using synonyms or domain-adjacent terms not present in your documents
• Very short queries and very long queries

A system that handles these gracefully is one that works in production. A system that only handles clean queries will fail the moment real users interact with it.

If your RAG pipeline needs a systematic retrieval quality audit or a full rebuild with proper evaluation infrastructure, start a conversation with Clixo. We build retrieval systems that are measured, maintainable, and production-ready.