# How to Choose the Right Chunking Strategy for Your RAG Pipeline

> A practical how-to guide to RAG chunking strategies — fixed-size, semantic, hierarchical, and late chunking — with trade-offs and when to use each.

- **Published:** 2025-11-01
- **Author:** Clixo
- **Reading time:** 6 min read
- **Tags:** rag, chunking, vector-search, ai-engineering
- **Canonical URL:** https://clixo.sh/blog/rag-chunking-strategies-how-to-guide

You built a RAG pipeline. You indexed your documents. The LLM still returns vague, off-target answers. Before you blame the model, look at your chunks. Chunking is where most RAG pipelines quietly fail — not in the retrieval algorithm, and not in the prompt.

This guide walks through the main chunking strategies, the trade-offs each one carries, and a decision framework for picking the right approach before you index a single token.

## Why Chunking Strategy Decides RAG Retrieval Quality

Every document you index gets split into pieces before embedding. Those pieces are what the retriever actually scores against a user query. If a chunk is too large, the embedding averages over too many concepts and retrieval precision drops. If a chunk is too small, you retrieve a fragment with no surrounding context and the LLM cannot synthesize a useful answer.

The goal is chunks that are semantically coherent, match the granularity of the questions users actually ask, and carry enough context to be useful in isolation.

### The Four Core Approaches

**Fixed-size chunking** splits documents by token count — for example, 512 tokens with a 50-token overlap. It is fast, deterministic, and easy to tune. For the majority of use cases involving homogeneous, well-structured documents, this works well enough. Start here. Use `RecursiveCharacterTextSplitter` or an equivalent and move on to harder problems. The overlap matters: without it, key sentences that straddle a boundary get cut in half.

**Semantic chunking** uses embedding similarity between consecutive sentences to find natural break points. Instead of counting tokens, it groups sentences together until the semantic distance between adjacent groups crosses a threshold. This produces chunks that contain complete thoughts rather than arbitrary slices. The cost is latency at index time and sensitivity to the threshold parameter — too tight, and every sentence becomes its own chunk.

**Hierarchical chunking** is the pattern most production systems converge on. You index at two levels: small chunks (100-200 tokens) for retrieval, and larger parent chunks (500-1000 tokens) that you pass to the LLM at generation time. When a small chunk scores well, you fetch its parent for context. This resolves the core tension between retrieval precision and generation quality without choosing one at the expense of the other.

**Late chunking** embeds the full document first using a long-context model, then partitions the resulting token-level embeddings into chunk-sized groups. Because each embedding was computed in the context of the whole document, the resulting chunk vectors carry cross-document signals that independent chunk-level embeddings lose. This is especially useful for technical documentation where terms have meaning only in relation to earlier definitions.

## How to Choose: A Practical Decision Framework

```mermaid
flowchart TD
  A["Document corpus"] --> B{"Uniform structure\nor short documents?"}
  B -- Yes --> C["Fixed-size chunking"]
  B -- No --> D{"Narrative or\nmixed content?"}
  D -- Yes --> E["Semantic chunking"]
  D -- No --> F{"Need retrieval precision\nplus rich context?"}
  F -- Yes --> G["Hierarchical chunking"]
  F -- No --> H{"Technical docs with\ndense cross-references?"}
  H -- Yes --> I["Late chunking"]
  H -- No --> C
```

Start with fixed-size. Move up the complexity curve only when you have a measured reason to.

Use **fixed-size chunking** when:
- Documents are uniform in structure (support tickets, emails, product descriptions)
- You need fast iteration and do not have retrieval quality baselines yet
- Your retrieval precision is acceptable and your bottleneck is elsewhere

Use **semantic chunking** when:
- Documents are narrative or mixed — whitepapers, transcripts, long-form articles
- Users ask conceptual questions that span paragraphs rather than exact-phrase lookups
- You have the index time budget to run sentence-level embedding passes

Use **hierarchical chunking** when:
- You have measured retrieval precision dropping below your target
- Users need synthesized answers rather than extracted snippets
- Your documents have a natural section/paragraph structure you can exploit

Use **late chunking** when:
- Documents are highly technical with dense cross-references
- You are using a long-context embedding model (8k+ context window)
- You have already tuned the other approaches and still see recall failures on definitional queries

## Overlap and Metadata Are Not Optional

Whatever strategy you use, every chunk should carry metadata: the source document ID, the page or section, the chunk index within the document, and an approximate creation date for time-sensitive corpora. Retrieval without metadata is retrieval without any ability to filter, deduplicate, or audit.

Overlap between adjacent chunks exists to catch sentence-boundary artifacts. A 10-15% overlap (by token count) is enough for most fixed-size setups. More than that and you are storing redundant information that inflates your vector index without improving recall.

## Evaluate Before You Scale

Before indexing your full corpus, run a small evaluation:

1. Take 20-50 representative user queries
2. Index a sample of your documents with each chunking strategy you are considering
3. Retrieve top-5 chunks per query
4. Manually label whether the correct answer appears in those 5 chunks (this is Recall@5)
5. Compare across strategies

This evaluation takes a few hours and will prevent you from indexing terabytes of documents with the wrong strategy.

## Common Chunking Mistakes

- **Ignoring document structure**: PDFs with tables, headers, and lists need pre-processing before any chunking strategy applies cleanly. Raw PDF text extraction produces garbage that no chunker can fix.
- **No overlap on fixed-size**: Sentences that straddle chunk boundaries become half-sentences in your index.
- **Chunk size mismatched to query type**: If your users ask two-word lookups, 1000-token chunks will hurt you. If they ask multi-paragraph synthesis questions, 128-token chunks will frustrate the LLM.
- **Skipping re-indexing**: When your chunk strategy changes, re-index everything. Mixing chunks from different strategies in one index produces unpredictable retrieval behavior.

## The Right Strategy Is the One You Measure

There is no universally correct chunk size or strategy. The right answer is the one that moves your Recall@5 and answer quality metrics in the right direction for your actual documents and your actual users. Build the evaluation harness first, then tune the strategy.

If you are building a production RAG system and want a team that has shipped these pipelines end-to-end, [talk to Clixo](https://clixo.sh/#contact). We design, build, and evaluate AI retrieval systems for product teams who need to ship.

---

Clixo · 1141 W Bryn Mawr Ave, Itasca, IL 60143, US · [hello@clixo.sh](mailto:hello@clixo.sh)
[Start a build](https://clixo.sh/#contact) · [All services](https://clixo.sh/services) · [Agent guide (llms.txt)](https://clixo.sh/llms.txt)