RAG Chunking Strategies That Actually Improve LLM Retrieval Accuracy
Compare RAG chunking strategies — fixed-size, semantic, hierarchical, and late chunking — and learn which improves LLM retrieval accuracy for your document types.
You built a RAG pipeline. The retrieval returns chunks. The model reads them and answers. But the answers are frequently incomplete, wrong, or miss information that is clearly in the documents. The usual cause is not the model and not the embeddings — it is how the documents were split before indexing. Chunking decisions that seem like implementation details turn out to be the primary driver of retrieval quality.
This guide compares the main chunking strategies with honest tradeoffs, so you can choose the one that fits your document structure and accuracy requirements.
Why Chunking Matters More Than People Expect
Retrieval-augmented generation works by finding chunks that are semantically relevant to the query and passing them to the model as context. If the relevant information is split across two chunks by an arbitrary boundary, the retrieval system may return only one of them — or neither, if the split degrades the semantic signal in both. The model then answers from incomplete context.
Chunking determines what the retrieval system can find. It also determines how much context you inject per query (which affects cost and latency). Getting chunking wrong is expensive in accuracy terms and often invisible in early testing, because developers test on queries where the answer fits neatly within a chunk.
Strategy 1: Fixed-Size Chunking
Split the document every N tokens or characters, with optional overlap between consecutive chunks.
How it works: A 3,000-token document split at 512 tokens with 50-token overlap produces six chunks, where each chunk shares 50 tokens with its neighbor.
When it works: Homogeneous, uniform content — transcripts, logs, long-form prose with no internal structure. When the document has no natural semantic units, fixed-size chunking is as good as anything else.
When it fails: Structured content. A contract clause split at token 511 loses its meaning. A code function split mid-definition is useless. A numbered list split between items retrieves half-lists. Fixed-size chunking is the baseline, not the solution.
Overlap tradeoff: Larger overlap reduces boundary failures but increases index size and retrieval noise. An overlap larger than 15-20% of chunk size usually hurts more than it helps.
Strategy 2: Semantic Chunking
Split based on semantic similarity between consecutive sentences or paragraphs. Group sentences that are topically related and split at topic boundaries.
How it works: Embed consecutive sentences, measure cosine similarity between adjacent pairs, and split where similarity drops below a threshold. Each chunk is a self-contained topical unit.
When it works: Long documents with internal structure but no explicit formatting markers — research reports, news articles, long blog posts. Semantic chunking tends to produce better-bounded chunks for these document types.
When it fails: Short documents where every sentence is related (the algorithm sees no good split points and produces one massive chunk). Technical documents with dense, highly interrelated content. The similarity threshold is sensitive and requires tuning per document type.
Practical note: Semantic chunking requires an embedding call per sentence to compute boundaries. For a large corpus, this adds to indexing time and cost.
Strategy 3: Hierarchical (Document-Structure-Aware) Chunking
Split along the document's natural structural boundaries — sections, subsections, paragraphs — and maintain parent-child relationships between chunks.
How it works: Parse the document structure (using Markdown headers, HTML tags, or document metadata). Create chunks at each structural level. Store the hierarchy — a subsection chunk knows its parent section.
When it works: Well-structured documents where structure carries semantic meaning. Legal documents with numbered sections, technical documentation with headers, reports with defined sections. For these document types, structural chunking produces chunks that are semantically coherent by construction.
Retrieval enhancement: The hierarchy enables a retrieval pattern called "small-to-big" — retrieve at the granular level (subsection) for precision, then expand to the parent (section) for context before passing to the model. This gives the model surrounding context without inflating the embedding unit.
When it fails: Poorly structured documents with inconsistent or absent formatting. PDFs with complex layouts where structure is not recoverable cleanly from extraction.
Strategy 4: Late Chunking
Chunk after embedding rather than before. Embed the full document (or large sections), then split the resulting embeddings into chunk-level representations.
How it works: Pass the full document through a long-context embedding model. Take the output embeddings for each token position. Pool them into chunk-level embeddings by averaging token embeddings within each chunk boundary. The chunk embeddings carry full-document context because they were computed with access to the whole text.
When it works: Documents where every chunk's meaning depends on its context within the whole. A pronoun in chunk 3 refers to an entity introduced in chunk 1 — fixed-size and semantic chunking lose that connection; late chunking preserves it.
Tradeoffs: Requires a long-context embedding model (most current models have context limits of 8K-32K tokens). Higher indexing cost. Still an evolving technique with limited production case studies. Worth evaluating if your documents have high cross-reference density and your retrieval quality problems persist after trying structural chunking.
Practical Guidance: Matching Strategy to Document Type
| Document type | Recommended strategy |
|---|---|
| Uniform prose, no structure | Fixed-size with 10-15% overlap |
| Articles, reports, blog posts | Semantic chunking |
| Technical docs, legal contracts, manuals | Hierarchical with small-to-big retrieval |
| Short documents under 1K tokens | No chunking — embed whole document |
| Dense, cross-referential documents | Late chunking (if long-context embedding is available) |
Start with the simplest strategy that fits your document type. Measure retrieval recall against a test set of queries before moving to a more complex approach.
Evaluating Chunking Quality
The right eval for chunking is not embedding similarity — it is retrieval recall. For a set of queries where you know which document section contains the answer, measure what fraction of the time the relevant section is in the top-k retrieved chunks. If relevant sections are being missed, chunking (or embedding model selection) is the place to investigate.
Do not evaluate chunking in isolation from retrieval — they interact. A chunking strategy that produces better chunks may still underperform if the embedding model is not well-calibrated to your domain.
If you are building a RAG system and retrieval accuracy is a core product requirement, Clixo builds retrieval pipelines with the right chunking architecture for your document types.