Inaccurate RAG Answers? Check How Your Articles Are Chunked

The basic idea of RAG is to retrieve information first, then have a model answer using it. When retrieved content loses its context, generation can continue reasoning from a mistaken premise. Anthropic's Contextual Retrieval article describes adding local context to chunks and combining lexical and semantic retrieval. Read the reference ↗

Suppose you save a hardware review that includes the sentence, “Under this configuration, it uses about 8GB.” If that sentence becomes a standalone chunk, the model may retrieve it when a reader asks how much VRAM they need, without knowing which precision, input length, and concurrency level “this configuration” refers to.

You can attach the article title, section heading, publication date, and necessary conditions to each chunk. For example, explain that a passage comes from a particular single-user inference experiment using a specified quantization method, and that its figures apply only to that setup. This context must come from the original text; the model should not guess missing details.

Consider the different forms questions can take, too. Model numbers, error codes, and function names often need exact matching. “Why does memory use grow the longer it runs?” is closer to a semantic question. Hybrid retrieval is worth testing, but you still need to verify its effectiveness using your own questions.

Try it

Prepare ten questions that readers would actually ask. Compare two knowledge bases: one stores only the chunk text, while the other adds titles and experimental conditions. First check whether retrieval finds the right information, then check whether the answer is correct. Recording those steps separately helps you avoid mistaking a retrieval problem for a model problem.

Do not present a vendor's percentage improvement on its own dataset as a result your knowledge base can also achieve.