anik-jha/Nemotron-3.5-Bolt-74B-A9B-16GB-GGUF

Nemotron 3.5 Bolt 74B-A9B on a 16 GB GPU (expert-pruned GGUF)

A coding-focused build: experts were chosen on a mix that was 70% code, coding-agent traces and function calls, and only coding and tool calls were benchmarked.

Run Nemotron 3.5 Bolt on a 16 GB card. NVIDIA's compact Bolt build is sized for 24 GB. These two GGUFs remove routed experts from it so the whole model sits on a 16 GB or a 12 GB GPU, with room left for a long context window, and with no re-quantization: every expert that remains is byte for byte NVIDIA's quantization-aware-trained 2-bit weights.

	NVIDIA Bolt 24GB	This repo: 16 GB	This repo: 12 GB
File	20.5 GB	12.5 GB	8.4 GB
GPU memory allocated at 32K context (measured at server start)	20.7 GB	12.7 GB	8.6 GB
GPU memory allocated at 128K context (measured at server start)	21.6 GB	13.5 GB	9.5 GB
Peak GPU memory during a 124K-token request (measured)	21.6 GB	13.6 GB	9.6 GB
HumanEval+ (pass@1)	0.890	0.835 (94% kept)	0.579 (65% kept)
MBPP+ (pass@1)	0.743	0.688 (93% kept)	0.571 (77% kept)
Tool calls, BFCL simple (valid / correct)	100% / 94.3%	99.5% / 90.8%	99.3% / 89.0%
Tool calls, BFCL multiple (valid / correct)	99.0% / 92.0%	99.0% / 93.0%	97.5% / 89.0%
Decode / prefill speed (tok/s, DGX Spark)	40.1 / 1,145	41.4 / 1,398	42.3 / 1,567

Memory figures are for text; the vision projector adds about 1.7 GB (see Run it). Sizes are in GB the way GPUs count them (1 GB = 1,024 MB), so they compare directly with your card's 16 GB. The Hugging Face file list counts in decimal, so it shows the 16 GB build's file as 13.4 GB.

The 16 GB build keeps 93 to 94% of the 24GB build's coding scores and its tool calling, and a real 124K-token request peaked at 13.6 GB of GPU memory, inside a 16 GB card. Long-context quality was not benchmarked. The 12 GB build is for cards that cannot fit the 16 GB one: a 124K-token request peaked at 9.6 GB and it still calls tools reliably, but it loses about a third of its HumanEval+ score.

Files

File	Use
Nemotron-3.5-Bolt-74B-A9B-16GB-REAP50-IQ2_XXS.gguf	16 GB GPUs
Nemotron-3.5-Bolt-74B-A9B-12GB-REAP25-IQ2_XXS.gguf	12 GB GPUs. Aggressive: keeps 65% of HumanEval+ and 77% of MBPP+
mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf	Vision projector, NVIDIA's file unchanged. Add --mmproj <file> for image input
selections/*.json	The exact experts kept in every layer
eval/	Scores with paired CIs, per-problem results, tool-call outputs, and provenance (source revision and hash, llama.cpp commit, calibration size, expert-score stability, tensor verification)

74B-A9B names the parent model. REAP50/REAP25 mean 50%/25% of each layer's routed experts were kept.

Run it

Any llama.cpp build that runs NVIDIA's official Bolt 24GB GGUF should run these: same architecture, same format, fewer experts. Evaluated with llama.cpp at commit 547208d, plus a small local patch that returns malformed chat output as plain text instead of an HTTP error; it never triggered in any run behind this card.

llama-server -m Nemotron-3.5-Bolt-74B-A9B-16GB-REAP50-IQ2_XXS.gguf -ngl 99 -c 32768 \
  --parallel 1 --jinja --reasoning off --reasoning-format deepseek

These are the settings the memory and long-context numbers on this card were measured with: one slot, whole model on the GPU, reasoning off. The code and tool-call benchmarks ran with four parallel slots and the same reasoning settings; speed comes from llama-bench. Leave out --reasoning off to let the model think first (not evaluated). Raise -c to 131072 for 128K context; llama.cpp then allocates 13.5 GB for the 16 GB build. LM Studio and Ollama have not been tested yet.

For image input add --mmproj mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf. The projector adds about 1.7 GB of GPU memory: the 16 GB build then allocates 14.3 GB at 32K context and 15.2 GB at 128K, which leaves less than 1 GB free on a 16 GB card: too tight if the card also drives a display. For long context with images, add --no-mmproj-offload to keep the projector on the CPU: GPU memory stayed at 13.6 GB in our test, but each image took about 40 s to encode on the DGX Spark's CPU. The 12 GB build with the projector allocates 10.3 GB at 32K and 11.1 GB at 128K.

How it was made

Nemotron 3.5 Bolt routes each token to 9 of 240 to 496 experts per layer. Most of the 24GB file is those experts. We measured how much each expert contributes on 3.1 million tokens of calibration text (2,047 conversations: code, coding-agent trajectories, function calling, and general chat), scored experts with REAP (router weight times expert output size, averaged over the tokens routed to it), and kept the top half (16 GB build) or top quarter (12 GB build) of every layer. Layers keep the same fraction, so NVIDIA's own per-layer expert counts are scaled, not flattened.

The pruning is a byte copy. Kept experts, the router rows for them, and every non-expert tensor are identical to NVIDIA's file; this was checked tensor by tensor for every build. Expert scores were stable across the calibration set: the expert rankings from its two halves correlate at a median of 0.99 across layers (0.95 in the least stable layer). REAP was chosen over routing mass (router weight alone) by evaluation: routing mass scored lower on both builds.

Method background: Half the Experts, All the Code, accepted at the COLM 2026 Workshop on Efficient Reasoning.

Evaluation details

• Code: EvalPlus HumanEval+ (164 problems) and MBPP+ (378), greedy decoding, reasoning off, served by llama-server. Against Bolt 24GB, the 16 GB build's MBPP+ drop is -5.6 points (95% CI -9.3 to -1.9) and its HumanEval+ drop is -5.5 points (95% CI -11.0 to 0.0, not significant at p < 0.05). Paired bootstrap CIs and McNemar tests are in eval/summary.json.
• Tool calls: BFCL v3 simple (400) and multiple (200) through llama-server's OpenAI tools interface. "Valid" means one well-formed call to a provided function; "correct" uses a simplified version of BFCL's argument checker, so compare these numbers with each other, not with the official leaderboard.
• Degenerate outputs: 1 of 542 code generations for each build.
• Long context: one passphrase-retrieval check at 124K tokens (passphrase placed 60% of the way in) passed for both builds; it is a smoke test, not a long-context benchmark.
• Memory: GPU memory of the llama-server process (nvidia-smi), model fully offloaded, one sequence, measured on an NVIDIA DGX Spark. llama.cpp's own breakdown for the 16 GB build at 32K context: 12,207 MB weights + 420 MB KV and state + 148 MB compute. These are the allocations llama.cpp makes at startup (it reserves the full KV cache and compute buffers up front), read once the server was ready and before any request. Peak memory during one 123,598-token request, sampled every 0.25 s, was 22,165 MB for Bolt 24GB, 13,925 MB for the 16 GB build and 9,806 MB for the 12 GB build, each 90 MB above its startup allocation. Memory on an actual 12 or 16 GB discrete GPU was not measured. On a desktop GPU that also drives a display, leave some headroom.
• Speed: llama-bench, 512-token prefill and 128-token decode, DGX Spark. Decode speed depends on the experts active per token, which pruning does not change; expect your own GPU's numbers.

Limitations

• Calibration is code-heavy. Coding (HumanEval+, MBPP+) and single tool calls (BFCL simple and multiple) were measured. Multi-step agent workflows, general knowledge, multilingual use, long-context quality and long-form reasoning were not, and may degrade more.
• Evaluated with reasoning off. Reasoning-on behavior was not measured.
• Vision: smoke-tested only. On six generated images (reading text, counting, colour, a bar chart, code, a sign), both builds answered 6/6, the same as Bolt 24GB; vision quality was not benchmarked.

License and attribution

Unofficial community derivative of NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB; not affiliated with or endorsed by NVIDIA. Distributed under the OpenMDW License Agreement, version 1.1, the license of the original model; the full text is in LICENSE. All metadata from NVIDIA's GGUF is retained in these files.

Pruned with moep at commit 400603b.