Skip to the content.

How a change to a model server was measured and decided on the Volta builds. It is written to be reused: nothing in it is specific to the V100 except the numbers in the examples. This is the long form of part 5 of the front page.

The method exists because of one fact about this work: a change that makes a large language model server faster can also make it quietly wrong, and the people (and coding agents) making the change want it to have worked. Every rule below removes one way of fooling yourself. Most were added after the failure they prevent.

0. Before any of this: launch first, read upstream first

The rules below apply where a result is claimed. They are not a licence to test forever, and the founder said so more than once during this work.

1. Three labels, and one word that is never a pass

Every number carries one of three labels.

An estimate is never written as a result, and a plan is never written as done. A configuration that has not passed its own rule says “quality not established” next to every speed it reports.

A check has three outcomes: PASS, FAIL and INCONCLUSIVE. “Could not check” and “found nothing wrong” are different results. INCONCLUSIVE is never a pass; it is the start of the next test, and the report names the missing input.

2. The unit is the round, not tokens per second

With a speculative drafter, throughput is (1 + accepted drafts per round) / round time. The first factor depends on the text: code accepts three or four of five drafts, prose one. Tokens per second therefore moves with the prompt set, and a change can look like a win because the test text changed.

The engine’s own speed is the time of one verify round, decode only (first to last streamed token, divided by the rounds). Every decode change is decided on milliseconds per round. Prompt-reading changes are decided on time to first token at fixed prompt lengths. Source: Volta sPLIt, docs/method.md.

3. A rule is written before its data

Before a test runs, a file states: the arms, how each is measured, the figure that decides, the noise floor and how it is computed, the outcome that means ACCEPT, the outcome that means “no measurable change”, and what makes the test INCONCLUSIVE. The file has a time stamp earlier than the first measurement, and it is not changed afterwards.

For a quality comparison the same idea is a preregistration: the items, the scoring code and the verdict rules are fixed and hashed before any output is captured (example). Anything counted after scoring is labelled as outside the preregistered rules.

4. Mirrored arms, a measured noise floor

5. The fenced window on a machine that also serves

The eight- and twelve-card servers were in daily use. A test on them is a window:

  1. The serving build is stopped. The window’s script checks that the cards are empty.
  2. The reference arm boots in the same window. A speed or an output is compared with a reference measured minutes earlier on the same cards at the same power limit, never with last week’s number.
  3. Test arms boot one at a time, one change per boot. Each must log its switch as active on every worker; an OFF arm must log none.
  4. Each arm is judged by the rule. A boot that fails is not booted twice in the same window.
  5. The serving build is restored, and the restore is itself checked.

On the twelve-card build the last step became a two-phase rule: every check (no expert left in host memory, the memory limit active on all workers, seven long prompts token for token as the accepted capture, a memory rehearsal with one grown chat per slot) runs on a test port and writes a record of exactly which code and configuration passed. The serving boot on the real port sends no request and starts only if the record matches what is installed. A client can never reach a server that is still being judged, and a client’s request can never fail a check.

After a card error (an Xid line) or an unplanned reset, the rule is: stop nothing, start nothing, send nothing, report, wait.

The DGX Spark builds use the same idea; its harness is published as dsv41-window-harness.

6. Switch off means unchanged

Every new code path sits behind a switch that defaults off. Before any speed test:

Only then is a window spent on it.

7. Two judges, and what each can show

A change either keeps the same arithmetic or it does not. The two cases need different judges.

The identity judge. For a change that claims to compute the same thing: the same prompts, greedy decoding, compared token id by token id with a reference from the same window.

The near-tie gate. For a change that reorders floating-point sums (a fused kernel, a different split): outputs may differ in their last bits. The gate, fixed in advance, accepts a changed token only where the model’s top two candidates were within a small logit margin, and reports those separately. A confident token that moves is a FAIL. Several of the fastest kernels in these builds failed here and were not installed (list).

The closeness judge. For a change that is meant to change the arithmetic (another quantization, another chunk cut): there is no identity to check, so the build is compared with a reference build on many short contexts.

Published example with its limits stated: Volta Max uPLIft, “Accuracy”.

8. What “identical” cannot mean on this engine

Three findings limit every token-exact test. All three hold on the unmodified engine.

  1. A cache hit is not a cold read. At near-tied tokens (top-two logit gaps of 0.125 to 0.47 in the cases seen) a greedy reply can differ between a cold read of a prompt and a read that resumes from the prefix cache, and between hits at different positions. A token-exact test of cache reuse compares a hit with a hit at the same position in the same boot.
  2. A different chunk cut rounds differently. Large sums across the four cards of a board go through NCCL in half precision, in an order that depends on the buffer’s size; and the dense multiply library picks its split count from a chunk’s row count. A 64-row shift of the cut moved long-prompt answers as far as halving the chunk.
  3. Two boots can differ. Before one fix, sampled output varied by boot because an all-reduce’s order did.

So the acceptance test for “the same answers” is: same prompts, same cache state, same chunk cut, same boot or the same window’s reference. Anything else is a closeness question.

9. What went wrong with the method itself

These are the most useful part of this page.

10. The independent review

After the twelve-card build was serving, a second model was given the whole record, read-only, with one question: what did we measure wrong, and what are we missing? Its findings are the equality judge, the token cache, the overstated cause and the sum-versus-rate items above, plus a correction to the order of work. The owner of the build re-ran or re-read each finding before acting on it.

Two things made the review work. It was read-only, so it could not “fix” what it found. And it was handed the code and the receipts, not the authors’ conclusions.

11. A checklist

Before the test:

During:

After: