Origin. PLI Labs research archive, file
kernel-research/KERNEL-RESEARCH-CLAUDE-NOTES.md. Published 2026-11-18 in the Volta V100 research repository. Edited for publication. Machine names, private file paths, network addresses, session names and the operator’s name were replaced with generic terms (the V100 server,[private path],[address],the owner). Links to files that are not published were turned into plain text. No figure and no finding was changed. Dates inside the note are its own; later results are in the build pages of this repository.
Terse and binding for kernel lanes on the V100 server (8x V100-SXM2, sm_70, PP2 x TP4, 1Cat vLLM fork, v41k / sm70_turbomind ops); rule 12 covers NVIDIA’s DGX Spark (GB10, sm_121). Each rule names its source. L-sources are local: PROF1103/REPORT.md [L1], TOK1103/REPORT.md [L2], LANES.md by UTC stamp [L3], code in V41PKG/wt/combo1103 [L4]. The full report is KERNEL-RESEARCH/REPORT.md.
-
Size it before you build it. Compute the speed-of-light time: bytes ÷ ~720 GB/s achievable per V100 (900 GB/s nameplate) for decode, or FLOPs ÷ 125 TFLOPS FP16 for compute-bound work. Convert to ms per round against the 31.1 ms baseline. A saving under ~0.3 ms per round (~1 %) won’t be measurable; don’t spend a GPU window on it unless it unblocks something larger. Source: roofline (Williams et al., https://doi.org/10.1145/1498765.1498785); 718.6 GB/s best observed GEMV and the round breakdown [L1].
-
Remove work before speeding it up. Look for kernels sized by capacity (max context, full store width, padded vocab, max verify rows) instead of live data. The live-prefix index reduce saved ~0.85 ms per round (first reported as 1.3 against an outlier baseline; REVIEW1103 F6), the campaign’s biggest single kernel win. Source: [L2]; IDX-A/B 98.0 → 102.9 tok/s [L3 2026-11-03T02:24Z].
-
Every launch must pay for itself. The decode path already runs 2,089 kernels per round with ~2 ms of bubbles per stage. Never add chains of small torch ops per layer: lever 2 as ~50 ops per layer cost +6.9 ms. Fuse into a neighbour, and value a fusion by the kernel time and gaps it removes (~0.3-1 µs per graphed gap), not by its microbenchmark. Source: [L1]; [L2] item 3; lever 2 78.2 vs 99.1 tok/s [L3 01:29Z]; CUDA graphs on V100 (https://developer.nvidia.com/blog/cuda-graphs/).
-
Default-off knob, untouched old path. New kernels go behind a
VLLM_V41_*knob that defaults to 0. Prove the knob-off path is unchanged (AST/knob_off_check, not only an equality test) before landing. Source: V41PKG landing conditions [L3 2026-11-02 00:32Z]; every lever this campaign followed it [L3]. -
Declare the numerics contract up front. If you keep the reduction order (same warps/split; split N, not K), the result must be bitwise identical. Otherwise state a tolerance and pass the end-to-end gate: greedy identical, or flips only at traced near-ties. Autotuned or “suggested” configs are numerics changes too. Source:
gguf_dmv13_x.cuhheader [L4]; wo_a N-split [L3 2026-11-01 15:54Z]; DMVX-CFG 2/8 identical, dropped [L3 03:28Z]; lever 1 trace proof [L3 03:08Z]; batch invariance (https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/). -
Test like an adversary. Cover held-out shapes: M = 1 … k+1 verify rows, both PP stages, every layer role, ragged K, size 0/1/max. Write into fresh, NaN-poisoned output buffers and never reuse reference outputs. Run
compute-sanitizer(memcheck, racecheck, synccheck). Mutation tests must be killed. This applies doubly to AI-written kernels. Source: Sakana’s walk-back (https://techcrunch.com/2025/02/21/sakana-walks-back-claims-that-its-ai-can-dramatically-speed-up-model-training/); KernelBench-Verified (https://arxiv.org/abs/2607.16241); Kernel Contracts (https://arxiv.org/abs/2604.22032); KFUS1 mutants [L3 2026-11-02T00:28Z]. - Benchmark protocol.
- Microbenchmarks run with a cold L2, CUDA-graph replay, warm-up and clocks locked at 1522 MHz, and report % of speed of light.
- Nsight Compute’s defaults (base clocks, cache flush) are not production timings.
- Only the end-to-end ABBA decides. Use mirror order, an A/A noise pair and a quiet host CPU (no nvcc builds during arms), and discard or flag the first arm after a deploy.
Source: Triton
do_bench(https://github.com/triton-lang/triton/blob/main/python/triton/testing.py); NCU defaults (https://forums.developer.nvidia.com/t/is-it-acceptable-to-measure-kernel-performance-using-nsight-compute/338212); DMVX bench 0.03 vs end-to-end 0.45 ms [L3 03:24Z]; ASYNC-B1 confound [L3 01:58Z]; first-boot divergence [L3 03:00Z]. -
The sm_70 ISA is fixed. You have
mma.sync.aligned.m8n8k4FP16 with f32 accumulate (HMMA.884), the WMMA 16x16x16 API,dp4a,lop3and warp shuffles. You do not havecp.async,ldmatrix, TMA,wgmma, clusters, BF16, FP8/FP4 MMA or INT8 MMA. Pipelines are register-staged. Never port code that needs the missing pieces; port the idea. Source: Volta tuning guide (https://docs.nvidia.com/cuda/volta-tuning-guide/index.html); Ampere/Hopper tuning guides for what came later;x_mmain [L4]. - The sm_70 resource budget.
- ≤ 96 KB shared per SM (dynamic shared above 48 KB needs an opt-in); 64K registers per SM; ≤ 255 registers per thread.
- Check
-Xptxas -v: 0 spills, and registers low enough for the blocks per SM you planned. - Keep hot loops under ~12 KiB of instructions.
- Launch no empty blocks, and size grids for 80 SMs.
- Put read-only tables in L1-cached global memory, not per-CTA shared copies.
- Set the L1/shared carveout on purpose (
dmv13_xrequests max-shared so blocks per SM aren’t capped).
Source: Volta tuning guide (above); KMOE1 i-cache/regs/empty-block findings [L3 2026-11-01T18:49Z]; v41k codebook table and
dmv13_xcarveout notes [L4]. -
Pin the toolchain. Stay on CUDA 12.x: CUDA 13.0 removed Volta offline compilation. PyTorch keeps sm_70 only in cu126 builds, through 2.15. Triton officially supports CC 8.0+, so compile-check every Triton kernel for sm_70, verify it on the GPU, and prefer CUDA C++ for hot paths. Source: https://docs.nvidia.com/cuda/cuda-features-archive/index.html ; https://dev-discuss.pytorch.org/t/dropping-volta-support-from-cuda-12-8-binaries-for-release-2-11/3290 ; https://github.com/triton-lang/triton
-
Communication kernels have hard rules. Our custom all-reduce path needs size % 16 B == 0 (pad rows) and a fully connected TP group. Use one-shot for small messages and reduce only live data. Cross-GPU flags use
st.release.sys/ld.acquire.syswith per-CTA epochs, and the reduce order is fixed by rank (deterministic, but not NCCL-bitwise). Source: TOK1103index_reduce_live[L2]; vLLM one-shot vs two-shot (https://github.com/vllm-project/vllm/files/13574639/Latency-optimal.allreduce.and.cuda.graph.optimization.pdf). - DGX Spark (GB10) is sm_121a, not sm_100a.
- Build with
-gencode=arch=compute_121a,code=sm_121a; the shorthand reportedly drops the “a”. - There is no
tcgen05/TMEM/2-SM MMA: FP4/FP8 run through warp-level block-scaledmma.sync. - Limits are 99 KB shared per block and 48 warps per SM.
- At 273 GB/s, bytes per token decide decode speed. FA4, DeepGEMM-SM100 and FlashMLA-SM100 won’t run as-is.
Source: https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html ; https://docs.nvidia.com/dgx/dgx-spark/hardware.html ; community reports (https://forums.developer.nvidia.com/t/dearest-cutlass-team-when-the-hell-are-you-going-to-properly-fix-tcgen05-fp4-support-for-dgx-spark-gb10-sm121/359598); not confirmed in NVIDIA ISA docs.
- Build with