Origin. PLI Labs research archive, file
eight-card research/topk-full-20261110.md. Published 2026-11-18 in the Volta V100 research repository. Edited for publication. Machine names, private file paths, network addresses, session names and the operator’s name were replaced with generic terms (the V100 server,[private path],[address],the owner). Links to files that are not published were turned into plain text. No figure and no finding was changed. Dates inside the note are its own; later results are in the build pages of this repository.
Labels: MEASURED = from the receipt of a run, READ = from code or a log, INFERENCE = not measured.
Receipt: logs/topk/topk-full-full-20261110T042244Z.json (pass 2 reported). Bench: stage/topk/bench_topk_full.py.
Answer
- MEASURED. Top-k cost per decode round on the engine’s path (
index_topk_parallelat the capacity width), 1M-token capacity server:
| Live tokens | 1K | 4K | 16K | 64K | 128K | 256K | 512K | 1M |
|---|---|---|---|---|---|---|---|---|
| Stage 0, 3 layers (ms) | 0.032 | 0.092 | 0.232 | 0.231 | 0.233 | 0.260 | 0.312 | 0.472 |
| Stage 1, 5 layers (ms) | 0.119 | 0.231 | 0.387 | 0.387 | 0.434 | 0.521 | 0.793 | 1.560 |
| Total per round (ms) | 0.152 | 0.323 | 0.618 | 0.618 | 0.667 | 0.780 | 1.106 | 2.031 |
| Share of a 40 / 65 ms round | 0.4 / 0.2 % | 0.8 / 0.5 % | 1.5 / 1.0 % | 1.5 / 1.0 % | 1.7 / 1.0 % | 2.0 / 1.2 % | 2.8 / 1.7 % | 5.1 / 3.1 % |
- MEASURED. Launches sized by capacity cost almost nothing. A grid of the live segments only (the “live grid” arm) saves 0.003 to 0.015 ms per round at every live length. The 512K-capacity server gives the same totals within 0.011 ms (0.145 to 1.095 ms per round, 1K to 512K). The dead blocks past the live end are not the cost. Verdict: under the card’s 0.3 ms bar. Close the card.
- MEASURED. The larger, separate saving is the choice of kernel at short live length. The one-kernel
index_topkbeats the six-launch path at 1K, 4K and 16K by 0.108, 0.185 and 0.149 ms per round. From 64K up it loses (1.87 ms against 0.62 ms at 64K, 33.9 ms against 2.03 ms at 1M). That saving is also under 0.3 ms. - INFERENCE. The flat 77 us per call from 16K to 128K is the time one block takes to walk one 8,192-score segment (k7 write alone is 27 us of it). The cost is set by the segment size SEG and by the live length, not by capacity.
Microseconds per call (MEASURED, 1M capacity, 6 rows, median of 400 samples, each the mean of 10 calls)
| Stage | Live tokens | n per row (max) | nseg cap / live | cap, engine path: median (min-max) | k2 hist p1 | k4 hist p2 | k6 count | k7 write | k1+k3+k5 | single | live grid |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1,024 | 512 | 65 / 1 | 10.8 (10.6-11.2) | 1.9 | 1.9 | 2.0 | 1.9 | 4.6 | 1.5 | 8.6 |
| 0 | 4,096 | 2,048 | 65 / 1 | 30.6 (30.5-30.9) | 5.6 | 5.6 | 4.6 | 8.6 | 6.5 | 12.1 | 30.0 |
| 0 | 16,384 | 8,192 | 65 / 1 | 77.3 (77.2-77.7) | 15.5 | 14.6 | 13.6 | 27.4 | 6.5 | 38.0 | 77.6 |
| 0 | 65,536 | 32,768 | 65 / 4 | 77.1 (77.0-77.5) | 15.5 | 14.6 | 13.5 | 27.3 | 6.5 | 141.0 | 75.2 |
| 0 | 131,072 | 65,536 | 65 / 8 | 77.5 (77.4-77.8) | 15.6 | 14.8 | 13.8 | 27.3 | 6.5 | 295.1 | 75.9 |
| 0 | 262,144 | 131,072 | 65 / 16 | 86.5 (86.3-86.9) | 19.7 | 20.1 | 14.0 | 27.4 | 6.5 | 586.1 | 85.0 |
| 0 | 524,288 | 262,144 | 65 / 32 | 104.1 (103.7-104.6) | 25.5 | 27.1 | 14.7 | 31.3 | 6.5 | 1169.5 | 102.8 |
| 0 | 1,048,576 | 524,288 | 65 / 64 | 157.3 (151.2-160.1) | 39.4 | 43.4 | 20.2 | 48.8 | 6.8 | 2345.0 | 156.5 |
| 1 | 1,024 | 1,024 | 129 / 1 | 23.9 (23.7-24.3) | 4.1 | 4.1 | 3.1 | 5.5 | 6.5 | 7.8 | 22.2 |
| 1 | 4,096 | 4,096 | 129 / 1 | 46.3 (46.2-46.7) | 9.0 | 8.7 | 7.7 | 15.0 | 6.5 | 20.4 | 46.0 |
| 1 | 16,384 | 16,384 | 129 / 2 | 77.3 (77.2-77.6) | 15.5 | 14.6 | 13.6 | 27.4 | 6.5 | 71.1 | 75.5 |
| 1 | 65,536 | 65,536 | 129 / 8 | 77.3 (77.2-77.7) | 15.6 | 14.7 | 13.7 | 27.3 | 6.5 | 289.9 | 75.7 |
| 1 | 131,072 | 131,072 | 129 / 16 | 86.8 (86.6-87.3) | 19.7 | 20.1 | 14.2 | 27.5 | 6.5 | 576.6 | 85.2 |
| 1 | 262,144 | 262,144 | 129 / 32 | 104.1 (103.7-104.7) | 25.5 | 27.0 | 14.6 | 31.5 | 6.5 | 1149.4 | 102.7 |
| 1 | 524,288 | 524,288 | 129 / 64 | 158.6 (155.3-160.8) | 39.2 | 43.4 | 20.5 | 48.7 | 6.7 | 2309.6 | 157.3 |
| 1 | 1,048,576 | 1,048,576 | 129 / 128 | 311.9 (307.6-316.6) | 78.4 | 86.2 | 40.3 | 102.3 | 7.2 | 5378.2 | 310.6 |
| 0, short path (width 16,384) | 1K / 4K / 16K | 512 / 2,048 / 8,192 | - | - | - | - | - | - | - | 1.5 / 12.2 / 37.6 | - |
| 1, short path (width 16,384) | 1K / 4K / 16K | 1,024 / 4,096 / 16,384 | - | - | - | - | - | - | - | 7.6 / 19.5 / 67.2 | - |
k1 = hist.zero_() memset, k3 and k5 = _seg_select (1.4 to 2.9 us each). The sum of k1 to k7 timed alone is within
2.6 us of the whole call. Short-path ranges are in the receipt (min-max within 1.0 us). 512K-capacity rows are in the
receipt (the engine-path arm within 2.2 us of the 1M row at the same live length, every arm within 5.2 us).
Method
- READ. Engine path (
attention_ops.py:962-1030): a decode call (tokens x 8 heads per rank <= 64) scores the live keys, reduces them, then callsindex_topk_parallelwhenkeys.shape[0] > 16384, elseindex_topk.keys.shape[0]= max_len // ratio + 1 (kv_store.py:208): 524,289 (stage 0, ratio 2) and 1,048,577 (stage 1, ratio 1) at max-model-len 1,048,576, so nseg = 65 and 129. The scores areout[:, :width]of a buffer whose row stride is the bank width, roundup8(rows) (kv_store.py:123,attention_ops.py:935-939). The bench uses those widths and strides. - READ and MEASURED. Rows per call: 6. The boot log
run/state/p6-boot5.loghasnum_speculative_tokens: 5andcudagraph_capture_sizes [1..6], so a decode round verifies 1 + 5 tokens. The decode branch admits up to 8 tokens (DECODE_ROWS = 8). The six-row figure is for one request. - READ. Indexer layers: 8 per round, 3 on stage 0 (layers 2, 8, 14) and 5 on stage 1 (20, 24, 28, 32, 36), from
index_source_layer_idsandmodel.py:592. The drafter’s attention has no indexer. - MEASURED. Inputs: the 6 rows are positions L-6 to L-1. In each row the min(n, 600) largest live values are distinct
fp16 values in [2.5, 8) at random positions, every other live value is below 2.0, and every column past n holds
- Correctness: each arm’s output from a graph replay must equal torch.topk’s ascending indices row by row (or arange(n), -1 padded, when n <= 512). The checker was first shown to reject k = 511, the head of top-513, k = 256 against 512, a one-index mutation and a call that reads 64 columns past the live end. All 306 timed arms passed.
- MEASURED. Timing: each arm captured in a CUDA graph of 10 back-to-back calls (the engine runs these kernels inside
the FULL decode graph). 3 s card warm-up, then 30 warm-up replays and 20 batches of 20 replays per arm, with
torch.cuda.synchronize()around each batch. A CUDA event pair around each replay gives 400 samples of the mean of 10 calls. Batch wall time agrees within 1.00 to 1.33 times the event median. The sweep ran twice in one process, and pass 1 is within 5 % of pass 2 except for 8 of 306 arms (the first configurations, up to 11 %). - MEASURED. One test card, Tesla V100-SXM2-32GB-LS, 80 SMs, power limit 150 W, stock clocks
(max SM 1,440 MHz). Peak device memory reserved 69 MB. Nothing pinned or registered. Engine module
index_topk.pysha256 c491ab97806ed4cb, identical to the V100 server’s engine/src copy. - MEASURED. The first run (
topk-full-full-20261110T042023Z.*, one call per graph, no card warm-up) is superseded. It counted the host gap before each graph launch (1K calls of 23 to 30 us against 1.5 to 24 us now) and the first configurations ran slow. Its checks also passed.
Not measured
- The “grid of min(segments, cap) blocks” kernel does not exist. The live-grid arm stands in for it by passing only
the live columns (
scores[:, :n_max]). A real fixed-shape kernel would also loop, so its cost is at or above the live-grid figure (INFERENCE). - the V100 server’s production cards: clocks and power state there may differ from the 150 W stock-clock test card. The shares of the 40 to 65 ms round assume the per-call times carry over (INFERENCE).
- The prefill top-k calls (
index_topkinselect_compressed_positionsandindex_fuse) are not part of a decode round and were not timed. The scorer and the reduce kernels before the top-k were not timed. - Interaction with neighbouring kernels in the real decode graph (L2 state, overlap) was not measured. Each arm ran alone on the card.