How Much Memory Does a 7B Model Need? Start with the Weights

Start with a simplified estimate of weight size:

Weight size in bytes ≈ number of parameters × bits per parameter ÷ 8.

Taking 7B to mean approximately seven billion parameters gives these theoretical figures:

These are arithmetic estimates, not measured VRAM usage for a particular model. A real implementation also includes quantization metadata, unquantized layers, working buffers, activations, and the KV cache. Conversion between GB and GiB also produces differences in the numbers.

Hugging Face's documentation describes loading lower-precision weights with bitsandbytes, along with relevant compute types and implementation requirements. The actual benefit depends on the model and how it runs. Read the documentation ↗

If you are choosing hardware, first fix three conditions: the model and quantization version, the longest input, and the number of requests handled concurrently. Then measure peak memory, time to first token, and generation speed under those conditions. Parameter count alone misses how you actually intend to use the model.

Try it

List the models you plan to run and calculate the theoretical weight size of each. Add four columns to your test sheet: input length, concurrency, peak memory, and speed. Leave entries blank for models you have not run; do not record theoretical estimates as measurements.

If you only process one article a day, comparing the total cost of a remote API with local hardware should also include equipment purchases, idle time, electricity, and maintenance.