What is chunking in RAG?
Your assistant quotes a refund window and leaves off the two conditions attached to it, because the cut landed between them.
Chunking
Text splittingDocument splitting
Chunking is the practice of splitting documents into smaller passages before storing them for retrieval. A search can then return the relevant part rather than a whole file, because each passage is stored and retrieved on its own.
Retrieval works on whole chunks. Search finds a passage, and that passage goes to the model as the evidence. Anything left outside the boundary is invisible, however relevant it was.
Size is the first decision. Small passages match a question precisely and carry little context. Large ones carry the surrounding argument and dilute the match, because one paragraph about your topic sits among five that are not.
Overlap is the second. Repeating a sentence or two at each boundary keeps a thought from being cut in half. It costs storage and it produces near-duplicate results, so most teams end up trimming it back.
Retrieval quality is mostly a chunking problem
Teams blame the model for wrong answers when the fault is upstream. If the passage handed over is incomplete, a strong model produces a fluent answer built on half a rule. Nothing errors, and the answer reads well.
Structure beats arithmetic here. Cutting every 500 characters ignores what the document is. Follow the headings, the clauses or the table rows instead, and each passage stays a thing that means something on its own.
The awkward material decides your approach. Tables lose their meaning when a row is separated from its header. Contracts break when a clause is split from its definitions. Scanned pages arrive with no structure at all.
- 01Store the source and the page with every chunk, so an answer can be traced back.
- 02Keep headings in the chunk text. A passage that names its own section retrieves better.
- 03Re-chunking means re-embedding everything, so treat the first choice as expensive to change.
- ParsePDF into text and structure.
- SplitOn meaning, not character count.
- EmbedEach passage becomes numbers.
- RetrieveTop matches for the question.
- AnswerModel sees only what came back.
The fourth station gets all the attention. The second one decides what it is able to return.
Common questions
01What chunk size should we start with?
Start from the document's own structure rather than a number. A section, a clause or a table is the right unit, and the size follows from that. Where the material has no structure, try a few hundred words with a sentence of overlap. Then measure it against real questions.
02Does a bigger context window remove the need for chunking?
No. A large window lets you pass more passages, but retrieval still has to choose which ones, and the choice is made on chunks. Feeding whole documents also buries the relevant line among thousands of irrelevant ones, which costs accuracy as well as money.
03How do you chunk a table or a spreadsheet?
Keep the header with the rows and keep related rows together. A row of figures separated from its column names cannot be interpreted by anything, human or model. For wide tables, repeating the header in each chunk is worth the duplication.
04How do you tell if chunking is the problem?
Log what retrieval returned alongside every answer, then read the failures. If the correct text is missing from the retrieved passages, the fault is chunking or search. If the text was there and the answer still went wrong, look at the prompt or the model.
Systems that had to read messy documents
Related
- retrieval-augmented generation →The pattern chunking exists to serve.
- embedding →What each passage turns into before it is stored.
- vector database →Where the passages live and how they are searched.
- why your RAG returns wrong answers →The failure list, in order of how often we see it.
- RAG development →How we build retrieval that holds up on real documents.
