Advanced RAG Techniques: Reranking, Query Expansion, and Late Chunking

Basic RAG is a solved problem. Embed your documents, store the vectors, embed the query, retrieve top-K, pass to the LLM. That pipeline works well enough to get a prototype live. It does not work well enough for production systems where users ask complex questions, corpora are large, and wrong answers have real cost.

The techniques covered here — reranking, query preprocessing, late chunking, and multi-hop retrieval — are the incremental improvements that separate retrieval quality benchmarks from retrieval systems that hold up under real user behavior.

Cross-Encoder Reranking: Precision After Recall

Initial ANN retrieval optimizes for recall: it fetches candidates that might be relevant. The embedding-based similarity score is a fast approximation, not a precise relevance judgment. Documents ranked 8th or 12th in the retrieval pass are often more relevant than those ranked 1st or 2nd.

Cross-encoder reranking fixes this. A cross-encoder is a model that takes a (query, document) pair as a single input and outputs a relevance score. Because it sees both texts together, it can model fine-grained interaction between query terms and document content — something that independent query and document embeddings cannot capture.

The pattern:

1. Retrieve top-50 candidates from the vector index (optimize for recall)
2. Pass all 50 (query, chunk) pairs through the cross-encoder
3. Re-sort by cross-encoder score
4. Pass top-5 to the LLM (optimize for precision)

This two-stage architecture is now the production standard for high-quality RAG systems. Cohere Rerank, Jina Reranker, and the cross-encoder family from sentence-transformers are practical options. Open-source rerankers run on GPU in under 100ms for batches of 50 pairs.

HyDE: Embedding the Answer Instead of the Question

Hypothetical Document Embeddings (HyDE) is a query preprocessing technique that counteracts a structural mismatch in RAG retrieval. User queries are typically short and phrased as questions. Your documents contain answers — typically phrased as declarative statements. These two forms can be far apart in embedding space even when they are semantically equivalent.

HyDE inverts the query: instead of embedding the question, ask the LLM to generate a hypothetical answer document (a paragraph of text that would answer the question), then embed that. The resulting embedding lives in the same region of the vector space as real answer documents, producing better retrieval.

The obvious concern is that the LLM might hallucinate in the hypothetical answer, pulling retrieval toward wrong documents. In practice, the technique is robust because you only use the hypothetical text for retrieval, not for generation. The retrieved documents ground the final answer.

HyDE is most useful when queries are very short or highly idiomatic and retrieval quality is measurably poor. It adds one LLM call to the retrieval path, which increases latency.

Query Expansion and Multi-Query Retrieval

A single query embedding represents one point in vector space. If that point happens to be slightly off from where your relevant documents cluster, you miss them. Query expansion addresses this by generating multiple alternative representations of the same query.

The implementation:

• Use an LLM to generate 3-5 alternative phrasings or sub-questions for the user query
• Embed and retrieve against all of them in parallel
• Deduplicate retrieved chunks by document ID
• Merge into a single candidate set and rerank

This improves recall at the cost of additional embedding and retrieval calls. The latency overhead is manageable if retrievals run in parallel. The improvement in recall can be significant for ambiguous queries or queries where users use different vocabulary than the documents.

For multi-part questions — "What are the refund policy and the return window?" — query decomposition into independent sub-queries, with separate retrieval for each, is more effective than trying to retrieve a single chunk that answers both.

Late Chunking: Context-Aware Chunk Embeddings

Standard chunking embeds each chunk independently. The embedding model sees only the 512 tokens in that chunk, without any awareness of what came before or after in the document. For documents where terms derive meaning from context — technical documentation with defined acronyms, legal texts with antecedent references — independent chunk embeddings lose information.

Late chunking reverses the order of operations:

1. Pass the entire document (or a long section) through an embedding model with a large context window
2. Collect the token-level output embeddings — one vector per token
3. Pool the token embeddings within each chunk boundary to produce chunk-level embeddings

Because the token embeddings are computed in full-document context, each chunk's final embedding carries cross-document signals. "SSL" in chunk 5 knows it was defined as "Secure Sockets Layer" in chunk 1.

Late chunking requires an embedding model that outputs token-level embeddings (not just a single pooled output), and that supports the full document length in its context window. JinaAI's long-context embedding models are commonly used for this. The compute cost is higher than independent chunk embedding, but the retrieval quality improvement on technical corpora is measurable.

Multi-Hop Retrieval for Complex Questions

Single-hop RAG — retrieve once, generate once — fails on questions that require connecting information from multiple documents. "Which of our API endpoints were deprecated after the v3.0 release, and what are their replacements?" requires finding the deprecation list and the migration guide, and synthesizing them.

Multi-hop retrieval patterns:

• Iterative retrieval: After the first retrieval, use the retrieved content plus the original query to form a new retrieval query, repeat for N hops, then generate from the accumulated context.
• ReAct-style agents: Let the LLM interleave retrieval and reasoning steps, issuing retrieval queries as tool calls and building up context progressively.
• Decompose-then-retrieve: Decompose the complex question into atomic sub-questions upfront, retrieve for each independently, then synthesize.

Multi-hop patterns increase latency and complexity significantly. Use them only when single-hop retrieval measurably fails on your target query types, not as a default architecture.

Knowing When to Use Each Technique

Advanced techniques are not free. Each adds latency, complexity, and new failure modes. Add them in measured steps, against your labeled evaluation set, and keep only the ones that produce meaningful metric improvements:

• Start with hybrid search and hierarchical chunking
• Add a reranker if Recall@5 is adequate but precision in the top-3 is poor
• Add HyDE or query expansion if Recall@5 is insufficient for short or ambiguous queries
• Add late chunking if you have long technical documents with dense cross-references
• Add multi-hop retrieval only for query types that verifiably require it

The best advanced RAG system is the simplest one that meets your quality bar.

If you need help designing a retrieval system that handles complex queries at production quality, talk to Clixo. We build AI systems that ship and hold up.