Skip to the content.

Build: DeepSeek-V4.1-Flash-1017 on eight V100-SXM2-32GB with every routed expert at exactly the precision of NVIDIA’s NVFP4 pack. The most-used experts are on the cards; the rest are in pinned host RAM and are read over PCIe. Repository: https://github.com/ProprietaryLegal/Volta-Full-NVFP4-DeepSeek-V4.1-Flash-8xV100.

This is a reference build: it exists to give an all-NVFP4 output to check other work against, and it accepts being slow for that. Every figure below is MEASURED on 2026-11-09 unless the row says ESTIMATE, at a 1,522 MHz clock ceiling on the eight-card server, or on one card of a four-card test machine at 150 W where the row says “one card”. Source of every row: the repository’s README and docs/measurements.md.

What worked

Item What it is Result Changes answers? Code
Exact NVFP4 to MXFP4 fold NVIDIA’s pack stores one FP8 scale per 16 values. Because the pack was converted losslessly from DeepSeek’s MXFP4 experts, those scales fold back to one power-of-two scale per 32 values 44,696,862,720 values bit-equal to the NVFP4 formula in FP32, relative error 0 exact build_packs.py, accept_p1.py
Scale range on Volta The kernel rebases the scales into FP16. All 61,440 expert records had to fit all accepted; widest exponent span 8 against a kernel limit of 29 - the same folder
The QPN expert layer 4-bit weights, FP16 activations, FP16 tensor cores: the v100-skinny kernel as packaged by 1Cat-vLLM #742 one card: worst relative error 4.78e-4 against an FP32 reference; CUDA graphs replay bit-equal to eager; 131 microseconds per layer at 1 row, 280 to 348 at 6 rows - engine/
Two tiers, bit-equal The experts on the cards and the experts in host RAM give the same bits as all experts on one card, in decode and in prompt reading bit-equal for natural and forced host routes exact the same
Zero-copy host experts The host-RAM experts are packed once into a page-aligned arena, registered with CUDA and read in place by the kernel works; its cost is the row “Reading experts from host RAM” below exact the same
Per-stage card reserve One reserve for both stages ran out of memory on a 100K prompt. Reserves from sampled peaks (4.5 GiB on the first stage, 7.5 on the second) fixed it a 498,099-token prompt read in 274.9 and 326.6 s without an out-of-memory error - serve-nvfp4-pp2.yaml
Decode flat with context - 30.2 to 33.7 tok/s from 46 to 303,004 tokens of context - -

What did not work or is not good

Item Result What follows
Speed 28.2 and 30.2 tok/s on a legal-style short set, 18.8 and 20.1 on a general set Use it as a reference, not as a server
Reading experts from host RAM one host-RAM expert per token at 1 row adds about 975 microseconds to a layer; at 6 rows each costs about 1.0 to 1.4 ms per layer (about 3.5 to 4.9 GB/s over PCIe). Other miss paths on one card: copy-engine DMA 743.7 microseconds per expert, a gather on the card 1,014.8, an AVX2 host kernel 2,953.7 ESTIMATE from those: no miss path comes within about 10 ms per round of all experts on the cards. This number is why the later builds keep every expert in card memory
General text about half the legal-style speed (174.64 against 87.58 ms per round) The most-used set was chosen from routing counts that are not published, and general text misses it more often. With no experts in host RAM the gap goes away
Prompt reading 1,419 to 1,595 tok/s Every expert not on the cards is staged from host RAM while reading
MXFP4 Engram tables from NVIDIA’s pack built and checked (rows bit-equal), but the 4-bit GGUF table decodes 4 to 5 times more rows per second Not the default
Memory margin highest card memory seen 32,086 MiB against a limit of about 32,494 About 0.4 GiB of margin on the first stage
First boots three failed fast: a host mapping requested on card 0 for every worker; staging buffers kept per layer instead of per card; one shared reserve for both stages Each fixed at its cause

What this build teaches

PCIe is the wall. A V100 reads its own memory at up to 900 GB/s. The same card reads host memory at 3.5 to 4.9 GB/s. Any design that leaves weights a decode round needs on the far side of PCIe pays about a millisecond per expert per layer, and a round touches several experts in each of 40 layers. The measured answer was to stop trying to make the miss path fast and to make the miss not happen: first by choosing which experts get native precision (Volta Max uPLIft), then by adding a third board (Volta 12X AmPLIfy).