RAG Chunking Explained: How to Split Documents Without Losing Meaning

You ask your RAG system a question. It retrieves a passage that is technically relevant and completely useless—a sentence fragment that starts mid-thought, a table split in half, or a chunk that keeps referring to context living in a neighboring chunk you never see.

Most beginners blame the retriever. Then they blame the embeddings. Then they tweak similarity thresholds and hope.

The real culprit is usually how the document was split before it was ever embedded.

Why Your Retrieval Feels Broken (When the Problem Is the Chunks)

Here's the pattern I see constantly with early RAG builders: the pipeline runs, retrieval returns something, but the answer comes back wrong or incomplete. The natural instinct is to suspect the search layer. After all, retrieval is where "finding" happens, so a bad find must mean a bad search.

But chunking happens before embedding. The chunk is the unit that gets embedded, indexed, retrieved, and eventually fed to the language model. If that unit is broken, everything downstream inherits the break. A perfect retriever searching over poorly prepared data still returns poorly prepared data.

This creates the core tension of document chunking for RAG: your chunks need to be easy for vector search to find and give the LLM enough context to answer well. Those two goals pull in opposite directions.

Here's the anchor criterion I use to judge any chunking choice: a good chunk carries enough self-contained meaning to support the answer your retrieval is meant to find. If someone handed you that chunk with no other context, could you make sense of it? If not, the chunk is the problem.

Notice the careful wording. Some questions are pure fact lookup: "What is the penalty for X?" Those benefit from a chunk that answers on its own. But other questions require synthesis across multiple passages—comparing sections, tracing an argument, summarizing a policy. For those, a chunk that works as a clean evidence fragment, designed to be combined with neighboring context, is the right unit. The test is whether the chunk preserves enough meaning for your retrieval task, not whether every chunk must answer every question alone.

If you need a refresher on where chunking sits in the pipeline or how embeddings make text searchable, those are worth revisiting—but the short version is this: ingestion splits documents, embeddings make chunks findable, retrieval picks candidates, and the LLM answers from what it receives. Chunking shapes every stage after it.

Three Levers, Not One Decision

Beginners tend to ask "What chunk size should I use?" as if chunking were a single dial. It's not. Chunking is three separate design decisions, and conflating them makes failures hard to diagnose.

Boundary choice: where may a chunk break? At sentence boundaries? Paragraph boundaries? Section headers? Around tables and code blocks? This decision determines whether your chunks contain complete thoughts or arbitrary slices.

Retrieval-unit size: how much content belongs in one chunk? This controls how precisely retrieval can match a query to a passage. Smaller units isolate specific facts. Larger units capture broader context but dilute relevance.

Cross-boundary context: how do you recover meaning that spans chunks? Overlap is one answer. Parent-section expansion—retrieving a chunk plus its enclosing section—is another. Metadata that tells the model where a chunk came from is a third.

When retrieval returns garbage, ask which lever caused it. A fragment that starts mid-sentence is a boundary failure. A chunk that contains the right topic but not the specific answer is a size failure. A chunk that references "the above table" when the table lives elsewhere is a context-recovery failure.

What Chunking Actually Does to Your Retrieval

Think of the LLM's context window as a working desk with limited surface area. Every token you place on that desk takes space from something else. When retrieval hands over a chunk, the model has to work with whatever is on the desk—no more, no less.

Chunks too small lose context. A sentence fragment like "the penalty increases to" retrieves fine but answers nothing. The model gets a piece of evidence with no referent.

Chunks too large dilute relevance. A 2,000-token chunk about an entire policy document might contain the one sentence about penalties, but it's buried under hundreds of irrelevant tokens. Vector search has a harder time matching the query to the right region, and even when it does, the model must sift through noise to find the signal.

There is no single correct chunk size. Anyone who gives you a magic number—"use 512 tokens"—is selling certainty the problem doesn't offer. The right choice depends on your document type and the questions your users actually ask.

Chunking is a means to an end. The real test is downstream answer quality, not whether your chunks look tidy.

Chunking Strategies as Building Blocks

Most chunking guides present strategies as competing boxes: pick fixed-size or semantic or structure-aware. That framing is misleading. These approaches solve different parts of the problem and often compose.

Structure gives you hard boundaries: sections, headers, tables, code blocks. Recursive splitting enforces a size limit while respecting natural breaks. Semantic logic refines ambiguous boundaries by detecting topic shifts. Model-based approaches add judgment where structure alone isn't enough.

Think of them as layers, not rivals.

Fixed-size chunking

The simplest approach: split by character or token count, often with overlap between chunks. Cheap, predictable, and easy to implement.

The problem: it cuts through meaning arbitrarily. A sentence can split mid-phrase. A paragraph can break in half. Fixed-size chunking works only when your text is uniform enough that arbitrary boundaries rarely land in bad places—think logs or consistently formatted records.

Recursive splitting

Instead of counting tokens and slicing blindly, recursive splitting works through a hierarchy of separators: paragraphs first, then sentences, then phrases. It respects natural boundaries while still enforcing a size limit.

This is my default recommendation for prose. It's simple, respects sentence and paragraph structure, and handles most documents reasonably well. It won't produce perfect semantic units, but it avoids the worst failures of fixed-size slicing.

Semantic chunking

Semantic chunking tries to split at meaning boundaries rather than structural ones. Some approaches embed consecutive sentences and merge segments that are semantically similar; others look for topic shifts.

The payoff is more coherent chunks that preserve context. The cost is more compute, more tuning, and more moving parts. Semantic chunking shines on technical, academic, or narrative documents where continuity matters—but it's not magic, and it can struggle on documents with unusual formatting.

Structure-aware chunking

Structure-aware chunking respects the document's actual organization: sections, headers, tables, code blocks, page boundaries. Tables and code blocks get treated as atomic units rather than sliced mid-structure.

For formatted or technical documents, this is often the right call. A table split across two chunks loses its column headers. A function split mid-body loses its signature. Research on code retrieval has shown that structure-aware chunking—splitting at syntactic boundaries like functions and classes—beats naive line-based splitting for code tasks.

LLM-based and agentic chunking

The newest approaches use a language model to decide where boundaries should fall. These can produce the most meaningful chunks, and some implementations show measurable gains in downstream RAG correctness. But they're expensive, slower, and less battle-tested than the alternatives.

My take: these are worth watching, not worth starting with. Get the fundamentals working first.

Strategy	Best for	Cost	Watch out for
Fixed-size	Uniform text, logs	Low	Cuts through meaning
Recursive	Prose, general documents	Low	May split at awkward spots
Semantic	Academic, narrative docs	Medium	More tuning, more compute
Structure-aware	Tables, code, formatted docs	Medium	Needs clean structure detection
LLM-based	Complex, nuanced documents	High	Slow, expensive, newer

How to Choose: A Decision Framework

Work through three questions in order.

First, where does your document have natural boundaries? The degree of structure is the best starting point for choosing an approach.

• Free-form prose: recursive splitting with modest overlap.
• Structured documents with clear sections: structure-aware chunking that respects headers and section boundaries.
• Tables, code blocks, or figures: treat these as atomic units. Never split them mid-structure.
• Uniform text like logs: fixed-size chunking is fine here.

Second, what do your users actually ask? Fact-lookup questions—"What is the penalty for X?"—benefit from precise, smaller chunks that isolate the relevant passage. Summary or comparison questions need larger context to capture the full picture. If your questions routinely require evidence from multiple sections, plan for cross-boundary context recovery from the start, not as an afterthought.

Third, what can you afford to maintain? Compute cost, latency, your embedding model's context limits, and how much tuning you can sustain all matter. A sophisticated chunking strategy you can't maintain is worse than a simple one you can.

When to start simple: fixed-size or recursive chunking with overlap is a perfectly good first pass. Optimize only after you see real retrieval failures with real queries. Most teams I've watched waste time optimizing chunking before they've measured what actually breaks.

Common Chunking Mistakes and How to Fix Them

Splitting mid-sentence or mid-paragraph. This loses the referent—pronouns, "as discussed above," conditional clauses. Fix: use recursive splitting that respects sentence boundaries, and add overlap when context spans chunk boundaries.

Cutting tables, code blocks, or figures in half. Structured elements are atomic. A table without its headers is noise. A function without its signature is a fragment. Fix: detect structured elements and keep them whole, even if that means a chunk exceeds your usual size target.

Ignoring cross-boundary context. When context spans chunk boundaries—a paragraph continues, a list keeps going, a section references an earlier definition—overlap preserves the thread. A modest overlap is usually enough to handle this without bloating context. But don't treat a percentage as a rule. Inspect your actual failures: if retrieval keeps returning chunks that reference material just outside their boundaries, increase overlap or add parent-section context.

Chasing a magic chunk size. There is no universal number. The right size depends on your documents, your queries, and your embedding model. Fix: test against real queries and adjust based on what retrieval actually returns.

Not adding metadata. Section titles, document source, dates, and document type help retrieval and enable filtering. A chunk that knows it came from "Section 4.2: Penalties" carries more useful context than the same text alone.

Evaluating with made-up queries. Test with real questions your users would ask. Retrieve, inspect what comes back, and ask: does this chunk carry enough meaning to support the answer? If not, that's your signal to adjust.

Common mistake: Treating chunking as a one-time setup decision. It's a tuning loop. You adjust, retrieve, inspect, and adjust again.

A Practical Path Forward

Here's where I'd start: recursive splitting with modest overlap for most prose documents. Structure-aware handling for formatted content—keep tables and code blocks whole. Add metadata as you go.

Then run a quick experiment. Take a handful of real questions your system should answer. Retrieve with your current chunks. Inspect what comes back. For each failure, classify it: is this a boundary problem, a size problem, or a cross-boundary context problem? Change one variable at a time and re-test.

The loop is short: collect real queries, retrieve, inspect the returned units, classify the failure, change one lever, retest. Run that loop until your retrievals consistently return chunks that carry the meaning your answers need.

Keep the anchor criterion in your pocket: a good chunk preserves enough self-contained meaning to support the answer your retrieval is meant to find. Apply that test to any chunking choice, and you'll catch most problems before they reach your users.