Choosing an MoE model: active parameters are not a memory budget

A recent NVIDIA engineering article explains a common model-selection trap. Build a memory worksheet from total weights, runtime overhead and your actual workload instead of assuming the active-parameter count tells you what will fit.

Three-month free access · All articles are open through January 10, 2027 (Japan time).

Read the two parameter counts separately

Sophia Abbassi’s NVIDIA article of 2026-09-15 ↗ discusses dense and mixture-of-experts models. An MoE model uses only selected experts for a token, while its stored weights have a separate memory cost. This distinction is a useful starting point for choosing a model you can actually run.

The following worksheet is our own suggested exercise. It contains illustrative arithmetic, not a hardware benchmark, a purchase recommendation or a report that we deployed the cited model.

Translucent file slices pass through an inspection gate to a compute chip and separate memory modules, illustrating a local inference path. — AI-generated concept illustrationLook at the whole memory budget

Begin with the whole checkpoint

Write down the total parameter count, stored precision and exact checkpoint revision. Keep the active parameter count in another column. An active count helps you ask about per-token work; it does not erase the inactive weights from a resident model.

The Nemotron 3.5 Lightning BF16 model card ↗ is a concrete reference: it identifies 30B total and 3B active parameters and describes a hybrid architecture. It is an older model reference checked on 2026-10-11, not a claim that the model launched this month. Use the specific card and runtime recipe rather than transferring requirements from a different architecture.

As an invented arithmetic example, 30 billion weights stored at two bytes each take 60 billion bytes, or about 55.9 GiB. That is an idealized weight-only calculation. It excludes metadata, buffers and everything needed to execute the workload. Replacing 30 billion with the active count would answer the wrong storage question.

Add the workload instead of a magic margin

Create separate worksheet lines for checkpoint storage, runtime working memory, cached context or recurrent state, and temporary buffers. If a quantity is unknown, write “not measured” rather than selecting a comfortable-looking number. The distinction helps you see which assumptions prevent a confident decision.

Record the input length you intend to use, the maximum output and the number of simultaneous requests. A personal assistant with one short conversation and a batch job with many long documents are different workloads. The same model name does not make their memory use or response time interchangeable.

Before testing, decide what failure you will record: inability to load, an out-of-memory error during a longer input, malformed output, or a response that misses your quality rubric. A model that loads has passed only the first of those checks.

EXPLAINED IN SLICES

Look at the whole memory budget

Weights
What is actually loaded
KV cache
Context and concurrent requests
Runtime
Buffers and other allocations

Editorial diagramThese are categories, not measured proportions. Active parameter count describes computation; it does not by itself specify the memory you need.

Compare under a useful task

Choose a short invented technical note and ask for a structured extraction of its assumptions and unresolved questions. Save your expected fields before running any model. If you have compatible hardware and choose to test later, keep the note, output limit and concurrency fixed across candidates.

Record startup behavior, peak memory, first-output latency and whether the answer passes your rubric. Change one condition at a time when investigating a failure. A smaller file or a higher token rate is only useful if the output meets the purpose of the task.

Make the decision inspectable

Finish with the worksheet, raw outputs and a sentence that separates known measurements from estimates. Note any architecture-specific requirement you still need to verify. This is more useful than declaring every MoE model faster or treating a vendor’s comparison as your own result. The practical skill is learning which number belongs to storage, which belongs to computation, and which still needs an experiment.

Understand it. Then try it.

Read something else