Why Longer AI Conversations Use More VRAM: Understanding the KV Cache

The KV cache stores the keys and values needed for attention calculations, reducing repeated computation when generating subsequent tokens. Hugging Face's cache strategy documentation discusses memory and speed tradeoffs between caching approaches. NVIDIA's inference article also identifies the cache as an important part of resource budgeting. Hugging Face documentation ↗, NVIDIA technical article ↗

For a typical full-attention cache, a simplified estimate is:

Cache size in bytes ≈ 2 × number of layers × number of KV heads × dimensions per head × token count × concurrent sequences × bytes per element.

The 2 accounts for keys and values. Suppose an example model has 32 layers, 8 KV heads, and 128 dimensions per head. It stores 4,096 tokens for a single sequence using 2-byte values. The calculated cache size is approximately 512MiB.

Holding those assumptions constant:

• Increasing the context to 8,192 tokens gives approximately 1GiB.
• Storing four 4,096-token sequences concurrently gives approximately 2GiB.

This is an illustrative order-of-magnitude calculation created for this article. It excludes overhead such as cache allocation and buffers. Sliding windows, different attention architectures, and cache quantization can also change the result, so the estimate cannot be applied to every model.

Try it

Keep the model and quantization method fixed, gradually increase input length, and record peak memory and time to first token. Then vary concurrency. Change only one condition at a time to make it easier to identify which requirement drives memory usage upward.

Measure speed when testing ways to save VRAM, too. Moving the cache to another memory location or quantizing it further may introduce additional computation and data-transfer costs.