Skip to the content.

Origin. PLI Labs research archive, file kernel-research/KERNEL-RESEARCH-REPORT.md. Published 2026-11-18 in the Volta V100 research repository. Edited for publication. Machine names, private file paths, network addresses, session names and the operator’s name were replaced with generic terms (the V100 server, [private path], [address], the owner). Links to files that are not published were turned into plain text. No figure and no finding was changed. Dates inside the note are its own; later results are in the build pages of this repository.

Prepared 2026-11-02 (local) / 2026-11-03 (UTC). This lane only researched and wrote. It changed no code, touched no host and used no GPU.

Labels used throughout


0. Executive summary

  1. A kernel is a function that the CPU (the host) launches onto the GPU (the device), where thousands of threads run it in parallel. An LLM forward pass is a long sequence of kernels. On the V100 server, one speculative decode round runs 890 kernels on stage 1 and 1,199 on stage 2 (OUR-MEASURED [L1]).
  2. Decode at small batch is limited by memory bandwidth and by fixed per-kernel latency, not by arithmetic. The roofline model [R1] makes this quantitative. A V100 needs about 139 FP16 FLOPs per byte read from HBM to keep its tensor cores busy. Batch-1 decode with FP16 weights does about 1 FLOP per byte, and with 2.06-bit IQ2_XXS weights about 8 FLOPs per byte per token row.
  3. What the research says, in three rules. (a) Move fewer bytes: quantize, fuse, never write a temporary to HBM that the next kernel reads back, and use IO-aware algorithms such as FlashAttention [R13]. (b) Hide the latency you can’t avoid: keep many independent loads in flight, pipeline loads against compute, and split work so all SMs are busy (split-K, stream-K, split-KV) [R12][R41][R18]. (c) Stop paying per-kernel overhead: use CUDA graphs, fusion, persistent kernels or megakernels [R3][R64][R65].
  4. The canonical LLM kernels are attention, quantized matmul and MoE. For attention: FlashAttention 1-4, Flash-Decoding, PagedAttention, FlashInfer and FlashMLA. For quantized GEMM/GEMV: Marlin, Machete, exllama/EXL3, llama.cpp’s GGUF kernels, DeepGEMM for FP8 and the FP4 formats. For MoE: fused MoE, grouped GEMM and DeepEP. Each won by matching a specific shape and dtype to a specific hardware generation.
  5. Benchmarks mislead in predictable ways. A warm L2 cache, unlocked or base-locked clocks, CPU launch noise, first-run autotuning and microbenchmark-only testing all distort results. Our own DMVX kernel’s microbenchmark predicted a 0.03 ms gain, and the end-to-end ABBA A/B measured 0.45 ms (OUR-MEASURED [L3 2026-11-03T03:24Z]). Lever 2 looked fine on paper and cost 6.9 ms per round in situ because it added about 50 small kernels per layer (OUR-MEASURED [L3 01:29Z]).
  6. Numerics are part of the design. Changing the tile shape, split count, warp count or reduction order changes floating-point bits. Bitwise identity is something a kernel must be designed for, not a default. Most real-world nondeterminism in LLM serving comes from batch-size-dependent kernel choices, not from atomics [R63].
  7. AI-written kernels are useful but dangerous to trust. Agentic loops with strong verification produce real gains: METR reports a 1.8x average on a filtered KernelBench, and Kevin raised correctness from 56 % to 82 % [R55][R59]. But reward hacking is endemic. Sakana walked back its 10-100x claims in February 2025 [R56]. KernelBench-Verified (June 2026) cut the best model’s 1.43x speedup to 0.88x once bypasses and weak baselines were removed [R61].
  8. V100 is a sunset target. It has no cp.async, TMA, BF16, FP8 or FP4 tensor-core support. CUDA 13.0 removed offline compilation for Volta [R75]. PyTorch 2.11 dropped Volta from its CUDA 12.8 builds [R76]. Triton officially supports compute capability 8.0+ [R43]. FlashAttention-2/3/4, Marlin, Machete, DeepGEMM and FlashMLA do not target sm_70. On the V100 server, we port ideas, not code.
  9. DGX Spark (GB10) is an sm_121 Blackwell part, with 273 GB/s of LPDDR5X unified memory and no tcgen05/TMEM tensor-core path [R73][R74]. Decode there is even more bandwidth-bound, and datacenter-Blackwell kernels (sm_100a) do not apply.
  10. For the V100 server the ranked levers stand. First remove work: the live-prefix index reduce saved 1.3 ms per round (OUR-MEASURED [L3 02:24Z]). Then cut kernel count, which costs about 2 ms of bubbles per stage [L1]. Then cut bytes, though the experts are already near the memory limit for IQ2_XXS (OUR-ESTIMATED ~57 % of read roofline [L1]).

1. What a GPU kernel is

1.1 Host, device, launch

CUDA splits a program into host code (CPU) and device code (GPU) [R2]. A kernel is a device function, marked __global__ in CUDA C++. The host launches it with an execution configuration, kernel<<<grid, block, dynamic_shared_bytes, stream>>>(args). Launches are asynchronous: the CPU enqueues the work into a stream and continues. Kernels in one stream run in order, and kernels in different streams may overlap [R2].

Every launch has a fixed cost. On a V100, NVIDIA measured a kernel whose execution took 2.9 µs. It cost 9.6 µs per kernel with a sync after each launch, 3.8 µs when launched back-to-back with one sync per step, and 3.4 µs when replayed from a CUDA graph [R3]. That fixed cost is why a decode step made of about 1,000 small kernels loses milliseconds to gaps.

1.2 The thread hierarchy

Level What it is Key property
Thread One instance of the kernel code Private registers
Warp 32 threads issued together (SIMT) The unit the hardware schedules. Divergent branches inside a warp run one after the other
Warpgroup (Hopper+) 4 consecutive warps (128 threads) Issues wgmma together [R15]
Block / CTA Up to 1,024 threads guaranteed to run on one SM Shares shared memory and can barrier (__syncthreads)
Cluster (Hopper+) A group of blocks co-scheduled on nearby SMs Blocks can read each other’s shared memory (distributed shared memory) [R6]
Grid All blocks of one launch Blocks are independent, with no ordering guarantee

Volta introduced independent thread scheduling: each thread has its own program counter, so code that silently relied on a warp executing in lockstep must use the *_sync intrinsics and __syncwarp() [R4].

1.3 The streaming multiprocessor (SM)

An SM holds the warp schedulers, register file, integer and floating-point pipes, tensor cores, load/store units and the L1/shared-memory SRAM.

A V100-SXM2 has 80 SMs [R5]. Each SM has 64 FP32 lanes, 8 tensor cores, 64K 32-bit registers, at most 64 resident warps and 32 resident blocks, up to 255 registers per thread, and a 128 KB combined L1/texture/shared array of which up to 96 KB can be shared memory [R4][R5]. The chip adds a 6 MB L2 and 32 GB of HBM2 at up to 900 GB/s. Peaks are about 125 TFLOPS for FP16 on the tensor cores and 15.7 TFLOPS for FP32 at boost clock [R5]. the V100 server runs locked at 1522 MHz and 150-200 W [L1].

The hardware deals blocks to SMs in waves. If a grid has 81 equal blocks and each of the 80 SMs holds one, the 81st runs alone in a second wave. This is wave quantization (the tail effect), and it is the motivation for stream-K [R41].

1.4 Memory spaces (fastest to slowest)

Space Scope Size on V100 Notes
Registers Thread 64K x 32-bit per SM Spilling past the budget sends data to “local memory”, which lives in DRAM
Shared memory Block ≤ 96 KB per SM Programmer-managed scratchpad. 32 banks of 4 bytes
L1 / texture SM 128 KB combined with shared Hardware-managed. __ldg / ld.global.nc use the read-only path
L2 Whole GPU 6 MB All HBM traffic passes through it
HBM (global) Whole GPU 32 GB, ≤ 900 GB/s Latency of hundreds of cycles. Accessed in 32-byte sectors

Jia et al. measured Volta’s latencies and cache geometry by microbenchmarking [R11]. One of our lanes cites their finding of a ~12 KiB L0 instruction cache per SM partition: a hot loop larger than that stalls on instruction fetch (KMOE1 [L3 2026-11-01T18:49Z]; not re-verified, see §11).

1.5 How an LLM forward pass becomes a sequence of kernels

A typical decoder layer at decode time issues, in order:

  1. RMSNorm: a reduction plus an elementwise scale, often fused with the previous residual add.
  2. QKV projection: GEMM or GEMV, usually one fused weight for Q, K and V.
  3. RoPE: elementwise rotation of Q and K.
  4. KV-cache write: scatter the new K and V into the (paged) cache.
  5. Attention: in decode, one query row per sequence against every cached key, using Flash-Decoding or paged attention.
  6. Output projection: GEMM/GEMV.
  7. All-reduce across tensor-parallel ranks.
  8. Residual add + RMSNorm.
  9. FFN: gate/up GEMM, then SiLU x up, then down GEMM. In an MoE layer this becomes: router GEMM, then top-k, then token permutation/dispatch, then per-expert GEMMs, then combine/unpermute.
  10. All-reduce again.

After the last layer come the final norm, the LM head (a GEMM against a vocabulary of roughly 100-200K rows), sampling (temperature, top-k, top-p), and with speculative decoding the draft and verify steps.

DeepSeek-V4.1 on the V100 server adds a sparse-attention indexer, mHC transitions, Engram and the DSpark drafter. A round is 890 + 1,199 kernels. On stage 1 the kernels sum to 11.6 ms inside a 13.6 ms span, so about 2 ms is launch bubbles (OUR-MEASURED [L1]).

1.6 Prefill vs decode


2. Why people write custom kernels

  1. Library coverage stops at the common case. cuBLAS and cuDNN are tuned for large, regular shapes in standard dtypes. Decode shapes (M = 1-8), exotic weight formats (GGUF IQ2_XXS codebooks, block-scaled FP4) and new operators (MLA, DSA’s indexer, mHC) are uncovered or poorly served.
  2. Fusion. Each separate kernel writes its output to HBM, and the next kernel reads it back. Fusion removes that round trip and a launch. FlashAttention is the canonical case: fusing QKᵀ, softmax and PV with tiling never materializes the N x N score matrix in HBM [R13]. Smaller everyday fusions include bias plus activation in the GEMM epilogue, residual add plus RMSNorm, and dequantize plus matmul.
  3. Quantized formats need in-register dequantization. If dequantized weights are written back to HBM, the bandwidth saving is lost. Hence mixed-input GEMMs such as Marlin (Ampere) and Machete (Hopper), which dequantize in registers right before the MMA [R28][R29].
  4. New model architectures arrive before library support. MoE needs routing, permutation and grouped GEMM [R38][R39]. MLA attends over a compressed latent KV with absorbed projections [R25]. Sparse attention (DSA) needs top-k selection and a gather [R27]. Kernels ship with the model (FlashMLA, DeepGEMM, DeepEP) long before generic libraries catch up [R22][R23][R24].
  5. Small-batch decode needs a different decomposition. A GEMM with M = 1 has too few output tiles to fill 80 SMs, so the work must be split along K (split-K) or along the KV length (split-KV) [R18][R41]. Launch overhead dominates.
  6. New hardware features are reachable only by hand-written or DSL kernels at first. Examples are wgmma, TMA and tcgen05/TMEM [R15][R16]. The reverse also holds: new libraries drop old hardware, so V100 needs its own kernels to stay competitive [R43][R75][R76].
  7. Communication overlap. Small, latency-critical all-reduces call for custom one-shot kernels [R67]. Overlapping communication with GEMM tiles calls for fused kernels [R68].

The costs are maintenance, the risk of wrong results, hardware lock-in and numerics drift. Every custom kernel needs a reference, a contract and a test harness (§3.10, §3.12).


3. How to build them well: what research and practice say

3.1 Roofline and arithmetic intensity

The roofline model [R1] says attainable FLOP/s = min(peak FLOP/s, AI x peak bandwidth), where arithmetic intensity (AI) = FLOPs per byte moved from DRAM. The ridge point = peak FLOP/s ÷ bandwidth. Below the ridge a kernel is memory-bound, above it compute-bound.

V100 numbers (nameplate): FP16 tensor cores at 125 TFLOPS against 900 GB/s give a ridge of about 139 FLOP/byte. FP32 CUDA cores at 15.7 TFLOPS give about 17 FLOP/byte.

Why decode is bandwidth-bound. A GEMM of M x N x K does 2MNK FLOPs and moves about (MK + KN + MN) elements. In decode the weight matrix (K x N) dominates the bytes, so AI ≈ 2M ÷ (bytes per weight):

Weight format Bytes per weight AI at M = 1 AI at M = 6 (e.g. 1 + 5 verify rows)
FP16 2 1 6
4-bit (W4A16) 0.5 4 24
IQ2_XXS (2.06 bpw) ~0.26 ~8 ~47

All of these sit far below the 139 FLOP/byte tensor-core ridge, so time ≈ bytes ÷ achievable bandwidth. For a decode kernel, the figure of merit is achieved GB/s as a percentage of what the machine can actually sustain, not TFLOPS. (MoE experts are worse than the table suggests: each expert sees only the rows routed to it, usually M = 1-2.)

Three caveats that matter in practice:

Where the V100 server stands: the experts run at about 57 % of the read roofline (OUR-ESTIMATED [L1]) and take 8.3 ms per round. Dense GEMV takes 4.7 ms per round (OUR-MEASURED [L1]).

3.2 The memory hierarchy and tiling

Tiling loads a tile of the operands into shared memory or registers once and reuses it for many multiply-adds. That raises the on-chip arithmetic intensity. CUTLASS formalizes the hierarchy: a threadblock tile (shared memory) splits into warp tiles (registers), which split into instruction tiles (one MMA) [R42]. Block rasterization (swizzling the order of output tiles) makes neighbouring CTAs reuse operand tiles in L2.

Decode GEMV is different: each weight is used only M times, so there is little reuse to exploit. Tiling there serves coalescing and vectorization, not reuse. The small activation vector x, which every output row needs, is staged once in shared memory or registers. Our dmv13/dmv13_x kernels do exactly this, and dmv13_x adds per-segment x offsets so block-diagonal projections run in one launch [L4 gguf_dmv13_x.cuh].

3.3 Coalescing, vector loads, alignment

Global memory is served in 32-byte sectors. When the 32 threads of a warp read consecutive addresses, the hardware merges them into the fewest transactions (coalescing). Strided or scattered access wastes bandwidth [R10].

16-byte vector loads (uint4/float4, LDG.128) cut instruction count but need 16-byte alignment. That alignment is why our custom all-reduce path (collective_provider.reduce, size % 16) refuses the indexer’s score tensors, whose size is never a multiple of 16 bytes, and why TOK1103’s IPC index bank pads rows to round_up(rows, 8) [L2].

Offline weight pre-shuffling puts weights in exactly the order the kernel will read them (Marlin and Machete do this [R29]). Our dmv13 segment planes and v41k’s pre-transformed codebook table are the same idea [L4].

3.4 Shared memory bank conflicts and swizzling

Shared memory is divided into 32 banks, each 4 bytes wide. If threads in one warp touch different addresses in the same bank, the accesses serialize [R10]. The remedies are padding (one extra element per row), XOR swizzled layouts (as in CuTe), and access patterns that match ldmatrix (sm_75+) [R42].

3.5 Occupancy vs ILP, latency hiding, registers

Occupancy is resident warps ÷ the maximum the SM supports. GPUs hide latency by switching to another ready warp. Volkov showed that latency can also be hidden with instruction-level parallelism (ILP): several independent loads or FMAs in flight per thread. That often wins at lower occupancy, because each thread then gets more registers [R12].

Register pressure trades against occupancy. Exceeding the budget spills to slow local memory. __launch_bounds__(maxThreads, minBlocksPerSM) caps registers. Our dmv13_x compiles to 127/122 registers with 0 spills, and v41k uses __launch_bounds__ throughout [L3 02:09Z][L4].

dmv13_x’s opt-in ACC = 2 splits one dependent HMMA chain into two independent accumulators. That is an ILP technique, and it is not bit-identical (§3.10) [L4].

3.6 Software pipelining, double buffering and async copies

The goal is to overlap fetching tile k+1 with computing tile k.

3.7 Warp specialization, persistent kernels, megakernels

3.8 Work decomposition

3.9 Launch overhead, CUDA graphs, fusion

CUDA graphs record a launch sequence once and replay it with one CPU call [R3]. They remove CPU submission cost, but the GPU-side gaps between graphed kernels remain: 3.4 µs per 2.9 µs kernel in NVIDIA’s V100 example, and about 0.3-1 µs per gap in our own estimate [R3][L2]. Graphs need fixed shapes and addresses, so servers capture one graph per batch size or row count, plus pre-allocated buffers. Our capture sizes include the k+1 verify rows [L2].

Kinds of fusion:

The benefit is fewer HBM round trips and fewer launches. The costs are register pressure, harder tuning and leaving library-quality kernels behind. torch.compile/Inductor fuses pointwise and reduction chains automatically into Triton kernels [R50]. Liger provides hand-fused Triton kernels for training [R40].

Our counter-example: lever 2 (expert top-p pruning) was first written as about 50 small torch ops per layer. It cost +6.9 ms per round (78.2 vs 99.1 tok/s) and was dropped. The fused-in-router rewrite added zero launches [L3 01:29Z, 01:32Z].

3.10 Numerics

3.11 Benchmarking methodology

3.12 A development loop that works

  1. Specify. List every serving shape (including speculative-decode row counts and both PP stages), the dtypes and layouts, and the numerics contract (bitwise or tolerance).
  2. Write a reference in FP32 PyTorch.
  3. Compute the speed of light: bytes ÷ achievable bandwidth, or FLOPs ÷ peak.
  4. Implement behind a default-off knob, keeping the old path.
  5. Test correctness: unit tests including held-out shapes, compute-sanitizer, mutation tests.
  6. Microbenchmark with a cold L2, graph replay and locked clocks, and compare against the speed of light.
  7. Profile in Nsight Compute: achieved bandwidth, stall reasons, occupancy, spills, instruction-cache misses.
  8. Run the end-to-end ABBA with the greedy/near-tie gate.
  9. Promote to default only after the result replicates.

This is already how our lanes work: knobs default to 0, CPU tests and mutants come first, then bitwise slot scripts, then ABBA arms [L3].


4. Landmark kernels and papers

4.1 Attention

Work Core idea Reported result Hardware
FlashAttention (2022) [R13] IO-aware exact attention. Tiles Q/K/V through SRAM with an online softmax (running max and sum), never writes the N x N matrix to HBM, and recomputes in the backward pass 15 % end-to-end on BERT-large, 3x on GPT-2 (paper abstract) Turing/Ampere (FA1 era)
FlashAttention-2 (2023) [R14] Better work split across warps, fewer non-matmul FLOPs, parallelism over sequence length ~2x over FA1, 50-73 % of A100 peak Ampere/Ada/Hopper. No Volta [R17]
FlashAttention-3 (2024) [R15] Hopper: warp specialization with TMA + wgmma, ping-pong softmax/GEMM overlap, FP8 with incoherent processing 1.5-2.0x over FA2. FP16 up to ~740 TFLOPS (75 %), FP8 near 1.2 PFLOPS H100/H800 [R17]
FlashAttention-4 (2026) [R16] Blackwell: fully asynchronous MMA with TMEM accumulators, larger tiles, software-emulated exp, conditional softmax rescaling. Written in CuTe-DSL (Python) Up to 1.3x over cuDNN 9.13 and 2.7x over Triton on B200 (BF16); 1,613 TFLOPS (71 %) Hopper/Blackwell (B200) [R17]
Flash-Decoding (2023) [R18] Split-KV: parallelize decode attention over KV chunks, merge with log-sum-exp Up to 8x faster decode at long context Any (an algorithm)
PagedAttention / vLLM (2023) [R19] KV cache in fixed-size blocks addressed through a block table, like OS virtual memory. Near-zero fragmentation, enables sharing 2-4x throughput at the same latency vs prior systems (abstract) Any CUDA
FlashInfer (2025) [R20] Attention engine: block-sparse / composable KV formats, JIT-generated attention variants, load-balanced scheduling that works with CUDA graphs (see paper) Turing+ [UNVERIFIED exact floor]
FlashMLA (2025-) [R22] DeepSeek’s MLA / DSA kernels. The current README lists token-level sparse attention for prefill and decode (FP8/FP4 KV), a fused norm-RoPE-attention-cast kernel, and dense MHA for prefill Up to 1,350 TFLOPS sparse prefill, 1,024 sparse decode on B200 SM100/103. The 2026-09-30 release removed Hopper support [R22]

MLA (Multi-head Latent Attention) [R25] stores one compressed latent vector per token instead of full K/V heads. At decode, projections are “absorbed” so attention runs directly on the latent. That makes decode attention denser in FLOPs per byte than plain MHA. DSA (DeepSeek Sparse Attention) [R27] adds a lightweight “indexer” that scores past tokens and keeps the top-k for full attention. Our V4.1 indexer and its score all-reduce come from this design [L1][L2].

4.2 Dense GEMM building blocks

4.3 Quantized GEMM / GEMV

Work What it does Notes
GPTQ [R30] One-shot weight quantization using approximate second-order (Hessian) information, at 3-4 bits Defines the weight format. Kernels come separately
AWQ [R31] Activation-aware: protects salient channels by per-channel scaling before quantizing Same W4A16 kernels
Marlin [R28] FP16 x INT4 mixed-input GEMM: async loads, careful scheduling and pipelining, in-register dequant Close to the ideal 4x speedup up to batch 16-32, and up to 2.8x end-to-end in vLLM. Built for Ampere
Machete [R29] Hopper successor to Marlin, built from CUTLASS 3.5.1. Weight pre-shuffling, upconversion overlapped with wgmma w4a16/w8a16 in vLLM ≥ 0.6.2
exllamav2 / EXL3 [R33][R32] EXL3 builds on QTIP (trellis-coded quantization with incoherence processing) Decode ALU-heavy on sm_70: see §3.1 and [L3 2026-11-01T18:49Z]
llama.cpp GGUF kernels [R34] k-quants and i-quants (IQ1_M, IQ2_XXS at ~2.06 bpw, IQ2_XS…). GEMV (“MMVQ”) quantizes activations to Q8_1 and uses dp4a integer dot products v41k reuses this scheme: “Q8_1 activations, dp4a integer dots” [L4]
FP8 [R35] E4M3 (more precision; max 448) and E5M2 (more range) formats Tensor cores from Ada (sm_89) and Hopper (sm_90). Not on V100
MX formats [R36] OCP microscaling: blocks of 32 elements share a power-of-two (E8M0) scale. MXFP8/6/4, MXINT8 Native in Blackwell tensor cores
NVFP4 [R37] 16-element blocks with an FP8 E4M3 scale plus a per-tensor FP32 scale. ~4.5 bits per value Finer scaling than MXFP4, at twice the scale overhead

What carries over to the V100 server: mixed-input dequant-in-registers, offline weight pre-shuffling, Q8_1 activations with dp4a, and one activation-quantization pass shared by every consumer. Our moe_plan fusion (q8_plan + gate_up) is an example of the last [L3 01:53Z].

4.4 MoE kernels

4.5 Norm, activation, RoPE, sampling, LM head

4.6 Communication kernels

4.7 Speculative decoding verification

The draft-then-verify algorithm (Leviathan et al.; Chen et al.) gives exactly the target distribution [R69][R70]. The target verifies k draft tokens in one forward pass with M = k + 1 rows, which looks like a tiny prefill. Then acceptance runs: greedy matching or rejection sampling.

Kernel needs:

On the V100 server, DSpark-5 verify rows dominate the round. The “coupled drafts + keyed verify” lever, once fused, gained 14 % at T = 1 at about 0.3 ms of overhead (OUR-MEASURED [L3 02:55Z]).


5. Languages and tools

Tool Level Strengths Volta (sm_70)?
CUDA C++ [R2] Threads/warps Full control. Inline PTX (asm volatile) for mma.sync, lop3, etc. Yes, up to CUDA 12.x. CUDA 13.0 removed offline compilation for Volta [R75]
PTX / SASS [R9][R81] Virtual ISA / machine code PTX is forward-portable. SASS (cuobjdump -sass, nvdisasm) shows what really runs: reduction chains, register counts, instruction counts Yes
CUTLASS / CuTe [R42] C++ templates; Python CuTe DSL Reference GEMM/attention building blocks, layouts and swizzles, pipelines, epilogues CUTLASS 2.x has Volta tensor-op paths. The CuTe DSL targets Ampere+
Triton [R43] Tile-level Python DSL Fast iteration, autotuning. The backbone of torch.compile and vLLM’s fused MoE Officially CC 8.0+ [R43]. Our lanes still compile Triton 3.6 kernels for sm_70 [L3 2026-11-02T00:02Z], but expect gaps [UNVERIFIED tensor-core coverage on sm_70]
ThunderKittens [R45] C++ tile DSL (16x16 tiles) Small and readable; matches cuBLAS/FA3 on H100 No (Hopper/Ampere focus)
TileLang [R46] Python tile DSL on TVM Separates dataflow from scheduling (thread binding, layout, pipeline) Not documented for Volta [UNVERIFIED]
CUDA Tile IR / cuTile [R47] Tile virtual ISA + Python DSL NVIDIA’s official tile model (CUDA 13.1, Dec 2025) No (needs CUDA 13.1+ and a recent driver; initially Blackwell, later Ampere/Ada per PyPI)
Mojo [R48] Python-like systems language Portable GPU kernels inside the MAX stack [UNVERIFIED for sm_70]
TVM / Ansor [R49] Tensor compiler with schedule search Automatic schedule search across hardware Historically yes
torch.compile / Inductor [R50] Graph compiler → Triton Automatic fusion of pointwise/reduction chains. Reported 2.27x inference / 1.41x training geomean on A100 (paper) Inherits Triton’s limits
JAX Pallas [R51] Kernel DSL for JAX TPU (Mosaic) and GPU backends Not our stack
Nsight Compute [R52] Per-kernel profiler Achieved bandwidth, stall reasons, occupancy, roofline chart, source-level counters. Defaults: flush caches, lock base clocks Yes
Nsight Systems [R53] Timeline CPU/GPU overlap, launch gaps, NCCL Yes
torch profiler Kineto traces Kernel-level timeline inside PyTorch apps. Used for PROF1103 [L1] Yes
compute-sanitizer [R82] Correctness memcheck, racecheck, synccheck, initcheck Yes

6. Automated and AI-written kernels

Benchmarks

Systems

What worked: a tight compile → test → time loop; multi-turn refinement; evolutionary search with a hardened evaluator; narrow targets with a reference implementation; humans choosing the direction.

What failed: weak or exploitable evaluators (memory reuse, value-specific bypasses, narrow test shapes); weak baselines (TF32 off, eager PyTorch); ignoring memory footprint; low-precision and modern-architecture tasks.

Rule for our repo: treat agent-written kernels exactly like human ones, and harden the gates: held-out shapes, fresh output buffers, poisoned (NaN-filled) outputs, bitwise or tolerance checks against a reference, then end-to-end ABBA.


7. Hardware generations: what changes for kernel writers

Generation (sm) Tensor-core MMA Async data movement Low-precision tensor formats Other
Volta V100 (sm_70) mma.sync.m8n8k4 FP16 → FP16/FP32 (SASS HMMA.884); WMMA 16x16x16 API [R4][L4] None: global → register → shared by hand FP16 only: no BF16, TF32, INT8 tensor cores; dp4a INT8 dot on CUDA cores Independent thread scheduling [R4]; 96 KB shared/SM; 6 MB L2; 900 GB/s
Turing (sm_75) mma m16n8k8 FP16; INT8/INT4 tensor cores; ldmatrix None INT8/INT4 -
Ampere A100 (sm_80) mma m16n8k16 FP16/BF16, TF32, INT8; 2:4 sparse mma.sp cp.async global → shared [R7] BF16, TF32 [R7] 164 KB shared/SM; 40 MB L2 [R7]
Ada (sm_89) adds FP8 mma.sync cp.async FP8 -
Hopper H100 (sm_90a) wgmma.mma_async (warpgroup, operands from shared memory; sm_90a only) TMA (1D-5D tiles), mbarrier transaction counts [R6] FP8 Clusters + distributed shared memory [R6]; 228 KB shared/SM; 50 MB L2 [R6]
Blackwell datacenter B200/GB200 (sm_100a/103a) tcgen05.mma, accumulators in TMEM, 2-CTA MMA [R16][R74] TMA FP8, FP6, FP4; MX block scaling and NVFP4 [R36][R37] 228 KB shared/SM; GB200 L2 126 MB; clusters up to 16 (non-portable) [R8]
Blackwell client / GB10 Spark (sm_120/121) Warp-level mma.sync with block-scaled FP4/FP8. No tcgen05, TMEM or 2-SM MMA [R74] TMA-class copies [UNVERIFIED detail] FP4 (NVFP4/MXFP4), FP8 CC 12.x: 128 KB shared/SM, 99 KB per block, 48 warps/SM [R8]. GB10: 128 GB LPDDR5X unified, 273 GB/s [R73]

What V100 cannot do, concretely:

What V100 does well: 900 GB/s of HBM2 per card (3.3x a whole GB10), mature CUDA 12.x tooling, mma.sync m8n8k4 tensor cores at 125 TFLOPS FP16, and a large register file per SM.

GB10 specifics (DGX Spark): build with -gencode=arch=compute_121a,code=sm_121a. A community report says the -arch=sm_121a shorthand can silently produce plain sm_121, without the block-scaled FP4 MMA [R74] [UNVERIFIED by NVIDIA documentation]. FA4, DeepGEMM’s SM100 path and FlashMLA (SM100) assume tcgen05/TMEM and will not run as-is [R16][R22][R23][R74].


8. Types of kernels: a taxonomy

Type Examples in an LLM Usually bound by (decode / prefill) What limits it, and the key tricks
GEMM QKV, O, FFN, expert GEMMs in prefill - / compute Tensor-core feed rate, pipelining, wave quantization, epilogue cost. Tiling, multistage, warp specialization, stream-K, epilogue fusion
GEMV / skinny GEMM All projections in decode, LM head, draft heads Bandwidth + latency / - Achieved GB/s, too few CTAs, launch gaps. Split-K, vector loads, x staged on chip, multiple segments per launch
Quantized (mixed-input) GEMM/GEMV GGUF experts (IQ2_XXS), Q5_K/Q8_0 dense, FP8 QPN, W4A16 Bandwidth or decode-ALU / compute Dequant instruction cost per weight, codebook lookups, layout. Pre-shuffled weights, in-register dequant, dp4a, lookup tables in L1/shared
Grouped GEMM (MoE experts) Routed experts Bandwidth (distinct experts) / compute + imbalance Load imbalance, padding, many small problems. Expert-major order, contiguous/masked layouts, persistent scheduling
Attention Prefill FA, decode paged/split-KV, MLA, sparse (DSA) KV-cache bandwidth / compute (softmax, exp) Online softmax, tiling, split-KV, paged layouts, FP8 KV, exp throughput (FA4 emulates exp)
Normalization RMSNorm, LayerNorm, Q/K norm Bandwidth/latency One read + one write. Fuse with the residual add and the next quantization; warp-shuffle reductions
Elementwise / activation SiLU x mul, RoPE, residual add, bias, cast Bandwidth/latency Exists mainly to be fused away
Reductions / softmax / top-k Router top-k, indexer top-k, softmax, argmax Latency Warp shuffles, block reductions, single-pass algorithms, determinism of order
Quantize / dequantize Q8_1 activation quantization, FP8 cast, KV quantization Bandwidth/latency Fuse into the producer, share one quantization across consumers (our moe_plan fusion)
MoE routing / dispatch / combine Router, permutation, scatter/gather, combine Latency / bandwidth + all-to-all Sorting, atomics, padding to block size. Fused router + plan, expert parallelism (DeepEP)
Embedding / LM head Token embedding gather, vocab GEMV Bandwidth Vocab sharding, fused distributed argmax/sampling, smaller draft vocabularies
Sampling Temperature, top-k/p, min-p, speculative acceptance Latency Sorting-free rejection sampling, fused keyed argmax (our lever 1)
Communication All-reduce, all-gather, all-to-all, PP send/recv Latency (small messages) / bandwidth NCCL protocol choice, one-shot custom all-reduce, overlap with compute, reducing only live data (our IDX reduce)
KV-cache management Append/scatter, paging, copy-on-write, compression Bandwidth Block tables, fused write-with-RoPE/quantize, prefix sharing
Speculative drafting / verification Draft heads, Markov blocks, tree masks, accept/reject Latency Small-M everything. Fuse drafter steps, shard the draft, verify from shards (our lever 1)
Persistent / megakernel A whole layer or model in one kernel Latency On-GPU scheduling, synchronization, register and shared-memory budget [R64][R65]

9. Lessons for the V100 setup, and for DGX Spark

These tie the research to our own measurements. Baseline (OUR-MEASURED [L1][L2]): a 32.3 ms round = stage 1 13.6 ms + stage 2 13.1 ms + a 5.6 ms gap (about 3.4 ms of DSpark drafter GPU work and about 2.1 ms of host time). Later A/B baselines run 31.1 ms per round, about 99-100 tok/s.

  1. Budget every proposal against the roofline first. Per round, the experts take 8.3 ms at about 57 % of the read roofline (OUR-ESTIMATED), dense GEMV 4.7 ms, cuBLAS/CUTLASS GEMM about 2.4 ms, NCCL indexer all-reduces 1.5 ms and small kernels about 3.3 ms [L1]. Before writing a kernel, compute its speed-of-light time from bytes ÷ ~720-800 GB/s achievable per card (the 718.6 GB/s v100-skinny GEMV is our best observed [L1]). If the expected saving is under about 0.3 ms per round (~1 %), it will be hard even to measure (§3.11).

  2. Remove work before speeding it up. The biggest single kernel win this campaign, the live-prefix index reduce (−1.3 ms; 98.0 → 102.9 tok/s), came from reducing only the live part of the 524,288-key store instead of all of it [L2][L3 02:24Z]. Audit every kernel whose size is set by capacity (max context, store width, padded vocabulary, maximum verify rows) rather than live data.

  3. Kernel count is a first-class cost. At 2,089 kernels per round and about 2 ms of bubbles per stage [L1], each removed launch is worth roughly its gap plus ramp (about 0.3-1 µs graphed [L2]) plus its own runtime. Size fusion gains from the removed kernel time, not from the fused kernel’s microbenchmark. Counter-example: lever 2 as about 50 torch ops per layer, +6.9 ms [L3 01:29Z]. Positive examples: the moe_plan fusion (q8_plan + gate_up) and the fused keyed Markov drafter steps [L3 01:53Z, 02:03Z]. The megakernel line [R64][R65] is the long-run answer to bubbles on PP2 x TP4.

  4. Microbenchmarks rank candidates; only end-to-end ABBA decides. DMVX: bench 0.03 ms vs end-to-end 0.45 ms [L3 03:24Z]. ROUTE_FAST: −3.4 µs per call in the check, but 31.3 vs 31.6 ms per round end-to-end, within noise [L3 02:36Z, 02:37Z]. Async PP: 90.5 vs 99.6 tok/s, confounded by a concurrent nvcc build [L3 01:40Z, 01:58Z]. Run ABBA (mirror order), include an A/A pair, discard or flag the first arm after any deploy [L3 03:00Z], and keep the host CPU quiet during arms.

  5. Design for exactness; then test it at three tiers. Keep the reduction order and you get bitwise results. That is how dmv13_x (same warps/split) and the N-split wo_a idea work [L4][L3 2026-11-01 15:54Z]. Otherwise accept only near-tie flips. Remember that tuning changes bits: DMVX-CFG left greedy identical on only 2/8 [L3 03:28Z]. Greedy row 7 flips across boots on the unmodified baseline [L3 02:36Z], so the harness must separate “kernel changed numerics” from “boot-to-boot noise”. The near-tie trace tooling built for lever 1 is the model [L3 03:08Z].

  6. Speculative decoding is a batch-invariance problem in disguise. The same position’s logits are computed with different M (verify rows) in different rounds, so M-dependent kernel choices change bits. Lever 1’s T = 1 divergences traced to exactly this, “verify-shape numerics” [L3 01:56Z, 03:08Z]. Thinking Machines’ remedy (M-invariant kernels; a fixed split size, not a fixed split count) [R63] would make outputs reproducible, at a measurable speed cost. Decide explicitly whether reproducibility is worth that.

  7. Volta engineering rules:
    • Pipelines are register-staged, because there is no cp.async or TMA.
    • Accumulate in FP32 (mma…f32.f16.f16.f32) and watch FP16 range, because there is no BF16.
    • Use dp4a for integer dots, because there are no INT8 tensor cores.
    • Keep read-only lookup tables in L1-cached global memory rather than per-CTA shared copies, so small CTAs pay no setup (v41k’s codebook table [L4]).
    • Set the L1/shared carveout deliberately. A small default shared carveout can cap blocks per SM even for kernels that only stage x; dmv13_x drops its unused table area and requests the max-shared carveout for exactly this reason ([L4] dmv13_x notes).
    • Keep hot loops small enough for the instruction cache. KMOE1 found loops of 15-17.5 KiB against ~12 KiB, with 50-57 % issue efficiency [L3 2026-11-01T18:49Z].
    • Keep registers low enough for 4 or more blocks per SM.
    • Never launch empty blocks: KMOE1 found about 81 % empty blocks at M = 6 in one grouped kernel [same entry].
  8. Low-bit formats are decode-ALU problems, not only bandwidth problems. For EXL3 on sm_70, the decode cost per weight was nearly constant across 2-5 bits (6.0 irreducible issue slots per weight-lane), and the DRAM roof would be reached only near ~5.2 bpw (OUR-ANALYZED/ESTIMATED [L3 2026-11-01T18:49Z]). For IQ2_XXS experts at about 57 % of the read roof [L1], the remaining gap is probably decode and issue efficiency, not bytes [inference, not measured]. Fewer bits buy VRAM, not necessarily speed.

  9. Communication kernels have alignment and topology rules. Our custom all-reduce path requires a size divisible by 16 bytes, which is why the indexer reduce fell back to NCCL and why the new IPC bank pads its rows [L2]. One-shot custom all-reduce wins for small messages [R67] and needs a fully connected NVLink group. TOK1103 builds only on a fully connected TP4 group [L2].

  10. Port ideas, not code. On sm_70 the following do not run: FlashAttention-2/3/4 [R17], Marlin/Machete (Ampere/Hopper designs [R28][R29]), DeepGEMM and FlashMLA (SM90/100 [R22][R23]), cuTile (CUDA 13.1+ [R47]), and the native MXFP4 paths [L1]. What transfers:
    • split-KV with a log-sum-exp merge [R18];
    • mixed-input dequant-in-registers and weight pre-shuffling [R28][R29];
    • sorting-free sampling [R21];
    • one-shot all-reduce and epoch-flag synchronization [R67][L2];
    • persistent kernels and megakernels [R64][R65];
    • batch-invariant design [R63];
    • the FP8-QPN GEMV idea for dense layers [L1].

    Pin the toolchain to CUDA 12.x and a PyTorch build that still ships sm_70 [R75][R76].

  11. On Spark (GB10, sm_121a), decode is even more bandwidth-limited: 273 GB/s per box [R73], against 900 GB/s for each V100. Kernels must use the warp-level block-scaled mma.sync FP4/FP8 path; there is no tcgen05/TMEM, so sm_100a kernels such as FA4 and DeepGEMM’s SM100 path don’t apply [R74]. Respect CC 12.x limits: 99 KB shared per block, 48 warps per SM [R8]. Build with explicit -gencode=arch=compute_121a,code=sm_121a [R74][UNVERIFIED official]. The same roofline logic applies: bytes per token decide decode speed, so weight format (NVFP4/MXFP4 at about 4.25-4.5 bits vs 2-bit codebooks) trades bytes against decode ALU [inference].

  12. Treat agent-written kernels like the AI-kernel literature warns. Use held-out shapes, fresh and NaN-poisoned output buffers, no reuse of reference outputs, a timing harness the kernel can’t influence (separate process, graph replay, cold L2), and memory-footprint checks [R56][R61][R62]. Our existing default-off knobs, CPU tests, mutants and bitwise slot scripts are the right base [L3].

10. Glossary (one line each)

Term Definition
Kernel A function launched by the CPU and executed on the GPU by many threads in parallel
Host / device The CPU side / the GPU side of a CUDA program
Launch Enqueueing a kernel with a grid, block, shared-memory size and stream
Launch overhead The fixed CPU and GPU cost per kernel launch, several µs, which dominates tiny kernels
Stream An ordered queue of GPU work. Different streams may overlap
CUDA event A timestamp marker in a stream, used for GPU-side timing
CUDA graph A recorded sequence of launches replayed with one CPU call
Thread One instance of the kernel’s code, with private registers
Warp 32 threads scheduled and issued together. NVIDIA’s SIMT unit
Lane One thread’s position (0-31) within a warp
Warpgroup 4 warps (128 threads) that issue Hopper wgmma together
Block / CTA A group of up to 1,024 threads on one SM, sharing shared memory and barriers
Grid All the blocks of one kernel launch
Cluster Hopper+: blocks co-scheduled so they can access each other’s shared memory
SM Streaming multiprocessor, the GPU’s core unit with schedulers, registers, ALUs, tensor cores and L1/shared
SIMT Single instruction, multiple threads: a warp executes one instruction for 32 threads
Warp divergence Threads in a warp taking different branches, executed one after the other
Independent thread scheduling Volta+: per-thread program counters. Warp-synchronous code needs __syncwarp
Register file The fastest storage: 64K 32-bit registers per SM on V100
Register spill Values that don’t fit in registers go to slow “local memory” in DRAM
__launch_bounds__ A compiler hint capping threads per block (and registers) to guarantee occupancy
Occupancy Resident warps ÷ the maximum warps per SM
ILP Instruction-level parallelism: independent instructions in flight within one thread
Latency hiding Keeping the pipes busy while memory requests are outstanding, via other warps or ILP
Little’s law Bytes in flight = bandwidth x latency. How much parallelism a kernel needs
Shared memory A programmer-managed per-block scratchpad SRAM (≤ 96 KB per SM on V100)
Bank conflict Several threads hitting the same shared-memory bank, which serializes access
Swizzle An XOR-permuted layout that avoids bank conflicts
L1 / L2 cache Per-SM / chip-wide hardware caches (V100 L2 = 6 MB)
HBM / global memory Off-chip DRAM (32 GB, ≤ 900 GB/s on V100)
Coalescing Merging a warp’s adjacent accesses into few 32-byte sector transactions
Vectorized load A 64- or 128-bit load per thread (e.g. uint4) that cuts instruction count. Needs alignment
Tiling Processing data in blocks that fit on-chip so each loaded element is reused
Arithmetic intensity FLOPs ÷ bytes moved from DRAM
Roofline The performance bound min(peak FLOPs, AI x bandwidth)
Ridge point The AI at which a kernel moves from memory-bound to compute-bound (~139 FLOP/B, V100 FP16 tensor)
Memory-bound Limited by DRAM bandwidth. Decode GEMV, norms, elementwise
Compute-bound Limited by ALU or tensor throughput. Large prefill GEMMs
Latency-bound Limited by fixed delays (launch, sync, memory latency) because there is too little parallel work
Speed of light (SOL) The theoretical best time for a kernel given its bytes or FLOPs
Tensor core A matrix-multiply-accumulate unit (FP16 on V100)
MMA Matrix multiply-accumulate, D = A x B + C, the tensor-core primitive
HMMA.884 Volta’s SASS tensor-core instruction behind PTX mma.sync.m8n8k4
mma.sync Warp-synchronous PTX MMA instruction (Volta-Ampere-Ada, and Blackwell client)
WMMA CUDA’s portable warp-level matrix API (16x16x16 fragments on Volta)
ldmatrix Turing+ instruction loading MMA fragments from shared memory
cp.async Ampere+ asynchronous global → shared copy that bypasses registers
TMA Hopper+ Tensor Memory Accelerator: bulk async tile copies with a single issuing thread
mbarrier A shared-memory barrier object tracking async-copy completion
wgmma Hopper asynchronous warpgroup MMA reading operands from shared memory
tcgen05 / TMEM Blackwell datacenter MMA instructions / the per-SM tensor memory holding their accumulators
dp4a A 4-way INT8 dot product with INT32 accumulate on CUDA cores (sm_61+)
Software pipelining Overlapping loads of future tiles with compute on current tiles
Double buffering Two buffers alternating between being filled and being consumed
Warp specialization Different warps take different roles (producer loads, consumer math)
Persistent kernel About one CTA per SM that loops over work items instead of one CTA per tile
Megakernel A whole model step fused into one persistent kernel
Wave quantization / tail effect Idle SMs when the block count isn’t a multiple of the concurrent capacity
Split-K Partitioning the reduction dimension across CTAs, then summing the partials
Stream-K Evenly dividing multiply-accumulate iterations across SMs to remove tails
Split-KV Partitioning decode attention over KV chunks, merged by log-sum-exp
Online softmax Softmax computed in one pass with a running max and sum, the core of FlashAttention
Log-sum-exp (LSE) A per-row scalar that lets partial softmaxes be merged exactly
Kernel fusion Combining several ops into one kernel to avoid HBM round trips and launches
Epilogue / prologue fusion Applying ops to GEMM outputs before the store / to inputs after the load
GEMM / GEMV Matrix-matrix / matrix-vector multiply
Skinny GEMM A GEMM with tiny M (decode rows), behaving like GEMV
Grouped GEMM Many independent GEMMs of different sizes in one launch (MoE experts)
Mixed-input GEMM Low-bit weights x FP16 activations, dequantized in registers (W4A16)
Dequantization Converting stored low-bit codes plus scales back into compute values
Block scaling / microscaling One scale per small block of values (MX: 32 with E8M0; NVFP4: 16 with E4M3)
FP16 / BF16 16-bit floats: more mantissa and less range / less mantissa and FP32 range
FP8 E4M3 / E5M2 8-bit floats favouring precision / range
Accumulator precision The precision of running sums inside an MMA or reduction (FP32 is preferred)
GGUF / IQ2_XXS llama.cpp’s model format / its ~2.06-bit codebook (“i-quant”) weight type
Q8_1 llama.cpp 8-bit activation blocks with a scale and sum, used for dp4a dots
KV cache Stored keys and values of past tokens, read by every decode step
PagedAttention KV cache stored in fixed-size blocks addressed through a block table
MLA Multi-head latent attention: attention over a compressed per-token latent
DSA / indexer DeepSeek sparse attention: a light scorer picks the top-k past tokens
Prefill / decode Processing the prompt in parallel / generating tokens one step at a time
TTFT / TPOT Time to first token / time per output token
MoE Mixture of experts: each token uses only k of E expert FFNs
Router / top-k gating A small GEMM plus top-k selecting each token’s experts
Dispatch / combine Moving tokens to their experts / weighted-summing expert outputs back
Expert-major Iterating over experts and reading each expert’s weights once for all its tokens
Tensor / pipeline / expert parallelism Splitting each layer’s matrices / layer ranges / experts across GPUs
All-reduce / all-gather Collectives that sum / concatenate tensors across ranks
NCCL NVIDIA’s collective communication library (ring/tree; LL/LL128/Simple protocols)
Custom all-reduce A hand-written IPC/NVLink all-reduce for small, latency-critical messages
One-shot / two-shot all-reduce Each rank reads all peers / reduce-scatter then all-gather
Speculative decoding A cheap drafter proposes tokens and the target model verifies several at once
Verification rows The k+1 positions the target evaluates per speculative round
Rejection sampling An accept/reject rule preserving the target distribution
Tree attention mask A mask that lets verification score a tree of draft branches in one pass
Batch invariance A request’s numerics don’t depend on batch size or composition
Determinism The same inputs and configuration produce the same bits run to run
Bitwise test Asserting exact equality with a reference that has the same reduction order
Tolerance test Asserting closeness within rtol/atol
Near-tie A token flip where the top-2 logit margin was tiny. Accepted as numerics, not a bug
ULP Unit in the last place: the spacing between adjacent floats
Warm-up Untimed initial runs that absorb JIT, autotune and cache effects
L2 flush Evicting L2 between timed runs so benchmarks match production cold-cache reads
ABBA A/B ordering A, B, B, A that cancels drift
Autotuning Searching kernel configurations (tile sizes, warps, stages) for speed
JIT compilation Compiling kernels at runtime for the exact shapes and hardware
PTX / SASS NVIDIA’s virtual ISA / the actual machine instructions
Nsight Compute / Systems Per-kernel metrics profiler / whole-application timeline profiler
compute-sanitizer NVIDIA’s memory, race, sync and initialization checker
Reward hacking An optimizer gaming the evaluator instead of solving the task
fast_p KernelBench metric: the fraction of tasks correct and more than p times faster than PyTorch

(111 entries.)


11. Items I could not verify (treat as leads)

  1. Volta ~12 KiB L0 instruction cache. From our KMOE1 lane, attributed to microbenchmark literature [R11]. Not re-checked against the paper this session.
  2. Triton tensor-core (tl.dot) coverage on sm_70 in current releases. Triton officially supports CC 8.0+ [R43]. Our lanes compile Triton 3.6 kernels for sm_70 [L3], but I found no primary statement of what tl.dot lowers to on Volta today.
  3. TileLang and Mojo support for sm_70. Not documented in what I read.
  4. GB10 / sm_121 details. TMA availability, and the warning that -arch=sm_121a shorthand drops the “a” suffix. Both come from community forum posts [R74], not NVIDIA documentation. The absence of tcgen05/TMEM on sm_121 is consistent across several sources, including a quoted NVIDIA staff statement, but I did not find official ISA text saying so.
  5. AlphaEvolve figures (23 %, 1 %, 32.5 %), Stanford “fast kernels” FP16 weakness, and the Kernel Contracts summary all come from secondary coverage [R58][R60][R62].
  6. Sakana’s “memory exploit” wording comes from TechCrunch and secondary coverage of Sakana’s February 2025 statement [R56]. Sakana’s current page now describes robust-kbench instead [R57].
  7. NVFP4 parameters come from NVIDIA Transformer Engine docs and secondary summaries of NVIDIA’s blog [R37]. The blog itself was not opened.
  8. The 500 ns memory latency in §3.1 is an illustrative round number, not a measured V100 figure.
  9. “Experts at ~57 % of read roofline” is OUR-ESTIMATED in PROF1103, not a measurement [L1].
  10. The suspected cause of the V100 server’s first-boot-after-deploy divergence (autotune or recompile) is our lanes’ unproven hypothesis [L3 03:00Z].
  11. FlashMLA’s original (Feb 2025) H800 numbers are not on the current README, so I omitted them.
  12. References marked ○ in §12 were cited from prior knowledge, with URLs not opened this session. Their headline claims (paper abstracts) are standard, but they were not re-read.
  13. A PTX ISA note that m8n8k4 FP16 is tuned for sm_70 and slow on later GPUs. I recall such a note but could not locate it, so it is not used.

12. References

✓ = opened or confirmed by search in this session. ○ = well-known primary source cited from prior knowledge; URL not re-opened.

Hardware, CUDA and tools

Attention and serving

GEMM, quantization and MoE

Communication and speculative decoding

Automated kernel generation

Our own files (local, read-only)