How to Choose the Right Chunking Strategy for Your RAG Pipeline

You built a RAG pipeline. You indexed your documents. The LLM still returns vague, off-target answers. Before you blame the model, look at your chunks. Chunking is where most RAG pipelines quietly fail — not in the retrieval algorithm, and not in the prompt.

This guide walks through the main chunking strategies, the trade-offs each one carries, and a decision framework for picking the right approach before you index a single token.

Why Chunking Strategy Decides RAG Retrieval Quality

Every document you index gets split into pieces before embedding. Those pieces are what the retriever actually scores against a user query. If a chunk is too large, the embedding averages over too many concepts and retrieval precision drops. If a chunk is too small, you retrieve a fragment with no surrounding context and the LLM cannot synthesize a useful answer.

The goal is chunks that are semantically coherent, match the granularity of the questions users actually ask, and carry enough context to be useful in isolation.

The Four Core Approaches

Fixed-size chunking splits documents by token count — for example, 512 tokens with a 50-token overlap. It is fast, deterministic, and easy to tune. For the majority of use cases involving homogeneous, well-structured documents, this works well enough. Start here. Use RecursiveCharacterTextSplitter or an equivalent and move on to harder problems. The overlap matters: without it, key sentences that straddle a boundary get cut in half.

Semantic chunking uses embedding similarity between consecutive sentences to find natural break points. Instead of counting tokens, it groups sentences together until the semantic distance between adjacent groups crosses a threshold. This produces chunks that contain complete thoughts rather than arbitrary slices. The cost is latency at index time and sensitivity to the threshold parameter — too tight, and every sentence becomes its own chunk.

Hierarchical chunking is the pattern most production systems converge on. You index at two levels: small chunks (100-200 tokens) for retrieval, and larger parent chunks (500-1000 tokens) that you pass to the LLM at generation time. When a small chunk scores well, you fetch its parent for context. This resolves the core tension between retrieval precision and generation quality without choosing one at the expense of the other.

Late chunking embeds the full document first using a long-context model, then partitions the resulting token-level embeddings into chunk-sized groups. Because each embedding was computed in the context of the whole document, the resulting chunk vectors carry cross-document signals that independent chunk-level embeddings lose. This is especially useful for technical documentation where terms have meaning only in relation to earlier definitions.

How to Choose: A Practical Decision Framework

Start with fixed-size. Move up the complexity curve only when you have a measured reason to.

Use fixed-size chunking when:

• Documents are uniform in structure (support tickets, emails, product descriptions)
• You need fast iteration and do not have retrieval quality baselines yet
• Your retrieval precision is acceptable and your bottleneck is elsewhere

Use semantic chunking when:

• Documents are narrative or mixed — whitepapers, transcripts, long-form articles
• Users ask conceptual questions that span paragraphs rather than exact-phrase lookups
• You have the index time budget to run sentence-level embedding passes

Use hierarchical chunking when:

• You have measured retrieval precision dropping below your target
• Users need synthesized answers rather than extracted snippets
• Your documents have a natural section/paragraph structure you can exploit

Use late chunking when:

• Documents are highly technical with dense cross-references
• You are using a long-context embedding model (8k+ context window)
• You have already tuned the other approaches and still see recall failures on definitional queries

Overlap and Metadata Are Not Optional

Whatever strategy you use, every chunk should carry metadata: the source document ID, the page or section, the chunk index within the document, and an approximate creation date for time-sensitive corpora. Retrieval without metadata is retrieval without any ability to filter, deduplicate, or audit.

Overlap between adjacent chunks exists to catch sentence-boundary artifacts. A 10-15% overlap (by token count) is enough for most fixed-size setups. More than that and you are storing redundant information that inflates your vector index without improving recall.

Evaluate Before You Scale

Before indexing your full corpus, run a small evaluation:

1. Take 20-50 representative user queries
2. Index a sample of your documents with each chunking strategy you are considering
3. Retrieve top-5 chunks per query
4. Manually label whether the correct answer appears in those 5 chunks (this is Recall@5)
5. Compare across strategies

This evaluation takes a few hours and will prevent you from indexing terabytes of documents with the wrong strategy.

Common Chunking Mistakes

• Ignoring document structure: PDFs with tables, headers, and lists need pre-processing before any chunking strategy applies cleanly. Raw PDF text extraction produces garbage that no chunker can fix.
• No overlap on fixed-size: Sentences that straddle chunk boundaries become half-sentences in your index.
• Chunk size mismatched to query type: If your users ask two-word lookups, 1000-token chunks will hurt you. If they ask multi-paragraph synthesis questions, 128-token chunks will frustrate the LLM.
• Skipping re-indexing: When your chunk strategy changes, re-index everything. Mixing chunks from different strategies in one index produces unpredictable retrieval behavior.

The Right Strategy Is the One You Measure

There is no universally correct chunk size or strategy. The right answer is the one that moves your Recall@5 and answer quality metrics in the right direction for your actual documents and your actual users. Build the evaluation harness first, then tune the strategy.

If you are building a production RAG system and want a team that has shipped these pipelines end-to-end, talk to Clixo. We design, build, and evaluate AI retrieval systems for product teams who need to ship.