Chunking: RAG's First Do-or-Die Step

Chunk too small and you lose context; too big and the key point gets diluted. This step decides everything.

Chunking is the first step of building a knowledge base, the least glamorous, and the easiest to half-ignore. But it sets the smallest unit retrieval can return — get it wrong and every downstream stage is damage control.

Symptoms you'll recognize:

• It answers the main clause correctly but misses the exception right next to it.
• Retrieved chunks read like half-sentences; you can't tell what the surrounding context was.
• One chunk packs five topics and the model grabs the wrong one.

Why this step decides everything

Because the chunk you create is the smallest unit retrieval can return. If no chunk in the index can fully answer a question, no amount of retrieval quality can surface the answer.

Look at the three strategies in the diagram.

Too small: "You can return items within seven days, except fresh produce" gets split into four chunks. The user asks whether returns are allowed; retrieval hits "returnable within seven days," and "except fresh produce" sits in another chunk that never gets recalled. The model answers "yes, you can." Every step looks correct; the answer is wrong.

These errors are especially nasty: confident, cited, and only appearing under specific conditions — tests rarely catch them.

Too big: An entire chapter becomes one chunk. Compressing a passage into a single embedding averages all its topics — returns, exchanges, invoices, complaints — into one coordinate where none stands out. The result: every question recalls a little, none accurately. And stuffed into context, its middle is likely ignored.

By structure: One chunk says one thing, with its conditions and exceptions together. That's the goal.

How to chunk

Three approaches, most to least reliable.

1. Follow the document's own structure (first choice). Use heading levels, clause numbers, Markdown sections. The author already marked where each "complete meaning" begins and ends — just use it. It covers most structured documents: manuals, policy clauses, API docs, help centers. If your document has clear headings, you barely need another strategy. Pinecone's chunking guide ranks semantic and structural chunking above fixed sizes. pinecone-chunking

2. Cut at semantic boundaries. For unstructured documents (meeting notes, long emails), split at paragraph and topic changes. A model can help find where topics shift.

3. Fixed length (last resort). Slice every 500 characters. Simplest to implement, easiest to saw a complete sentence in half.

If you must use fixed length, add overlap — neighboring chunks share a slice (say 10% to 20%). Even if a cut lands mid-sentence, one chunk stays whole. Overlap is the mandatory patch for fixed-length chunking, costing only a little extra storage. LangChain's recursive splitter bakes this in by default — preserving separators, configurable overlap — for exactly this reason. langchain-splitters

On size, a starting point: 200 to 500 tokens per chunk works for most cases. But it truly depends — legal clauses can be short, tutorials often need more room. Don't trust any "standard value"; test on your own documents.

Attach the heading before indexing

Nearly free, clearly effective, and most teams don't know it.

The move: prepend each chunk's source path before storing it. "Return Policy > Chapter 3 Returns > 3.2 Non-applicable cases," then the body text.

Why it works: cut chunks are often "orphaned." The body says "this clause doesn't apply to the following," but which clause? That information lives in the heading — which was cut away. Embedded in isolation, the text doesn't know its own topic, so retrieval can't match it.

With the path prepended, the chunk carries its own topic, the embedding lands far more accurately, and citing sources becomes easier.

Similarly, store metadata per chunk: source document, section, effective date, version, visibility scope. Retrieval can then filter — current valid version only, or only what this user may see. The "stale version got recalled" trap is exactly what metadata plugs.

Special content types

A few content types that break under naive chunking.

Tables. Splitting by row loses the header; storing the whole table may be too large. Practical fixes: turn each row into a natural-language sentence ("Product A has a 12-month warranty"), or at least keep the header with every chunk. Slicing a table like plain text basically destroys it.

Q&A pairs. If your material is already an FAQ, great — one Q&A pair is a natural chunk, and the question is naturally close to user queries. This is the best-performing corpus; prioritize cleaning it up.

Long procedures and steps. "Step five" retrieved alone is meaningless. Either keep the whole procedure in one chunk, or tag every chunk with the procedure name and total step count.

Code and config. Cut at syntax boundaries — one function, one config block — not by line count.

One pragmatic closing check: after chunking, randomly pull twenty chunks and read them. Ask: does this chunk alone make sense? Is the information complete? That five-minute audit beats tweaking any parameter.