Skip to the content.

Origin. PLI Labs research archive, file kernel-research/UPSWEEP-ATTN-PRIOR-ART.md. Published 2026-11-18 in the Volta V100 research repository. Edited for publication. Machine names, private file paths, network addresses, session names and the operator’s name were replaced with generic terms (the V100 server, [private path], [address], the owner). Links to files that are not published were turned into plain text. No figure and no finding was changed. Dates inside the note are its own; later results are in the build pages of this repository.

UPSWEEP, 2026-11-01 (about 17:30Z). Research only. No the V100 server GPU, no builds; the V100 server reads were read-only.

Question (16:57Z): KATT1’s HMMA decode attention kernel (patch 0012) failed R6 numerics on three q std 2000 cases. How do existing sm70 tensor-core attention kernels stay accurate at extreme logit scales without losing speed, and what can KATT1 adopt, at what cost in microseconds?

Tags: VERIFIED = I or a helper read the code at the named commit. INFERRED = arithmetic or reasoning from verified code, not measured. UNVERIFIED = from memory or a title, code not read in this pass. External code was read through gh api at pinned short shas. Fetched copies are in [private path] and .../scratchpad/pa-B/src/. They were never executed.

Commits used:

Repo Commit Notes
NVIDIA/cutlass e13f5cd414 examples/41
facebookresearch/xformers 3e62181f37 Last commit before 3d947a67fa removed the cutlass FMHA backend; head b5410d3bb4 no longer has it.
Dao-AILab/flash-attention v1.0.9 6d48e14a6c, v2.0.0 4f285b3547, main 4616f5116d  
zhinianqin/flash-attention-v100 c2eda5e611 This is exactly the engine’s pin VLLM_FLASH_ATTN_SM70_COMMIT (the V100 server [private path] cmake/external_projects/vllm_flash_attn.cmake:39, engine 23cec6a).
ai-bond/flash-attention-v100 0ababf82e3  
ZRayZzz/flash-attention-v100 05eace2700  
xormal/V100-SM_70-Flash-Attn-2-v1 05c632aa8e  
InternLM/lmdeploy dd4db0da27  
ggml-org/llama.cpp dde879bc4f  
Ootomo wmma_extension a9e6b34e40  
cuMpSGEMM 42dd92ea1c  
1Cat #1277 899e9ed331 Local copies in scratchpad up/v41src/.
KATT1 worktree head 96db5a1 [private path]. The kernel file has an uncommitted “version 2” edit (mtime 17:17Z).

0. Bottom line

  1. The R6 failure is in the per-block score path, not in the P precision or the split merge.
    • All three failing cases are T16 v640 q std 2000 at split_keys 32, 64 and 128, and the only failing check is matches_emulation (kernel vs CPU emulation 4.5 ulps). VERIFIED (slot.json, below).
    • All three splits give identical numbers, and so does the no-P-split variant. So the error comes before the split merge and before P. INFERRED.
  2. No surveyed sm70 kernel does anything special for extreme logits.
    • Every tensor-core kernel (cutlass/xformers, all three FA-V100 forks, llama.cpp MMA, TurboMind prefill, 1Cat) chains QK through the tensor-core C operand. VERIFIED.
    • The two production decode stacks avoid the problem by not using tensor cores for Volta decode QK. TurboMind runs FP32 SIMT for all sm70 decode, including the 576-dim MLA decode. llama.cpp routes Volta decode to its VEC or TILE kernels. VERIFIED.
  3. The one direct prior art for V100 accumulator rounding is Ootomo’s TCEC mma_rn_sync.
    • Each k-block is computed with C=0 and added outside the tensor core with FP32 round-to-nearest adds. VERIFIED, tcec.hpp L672-694.
    • That is what KATT1’s uncommitted version 2 already does: DS41_HMMA_QK_CHAIN default 1, and DS41_HMMA_PV_FRESH default 1. VERIFIED, KATT1 worktree ds41_decode_hmma_sm70.cu L26-36, L254-282.
  4. The best cheap addition is TurboMind’s exp2 FFMA fold, at about 0 µs.
    • Keep the running max in raw units and compute p = exp2f(fmaf(S_raw, c, -mc)), with c = scale*log2e and mc = fl(m*c). TurboMind impl_simt.h L478, L495; FA-V100 c2eda5e611 softmax.h L72-79. VERIFIED.
    • This removes the per-element rounding of a * scale at |s| about 8.4e3, where the FP32 ulp is 2^-10. INFERRED.
    • Doing the rescale and the merge as exp2(mc_old - mc_new) keeps the offsets exactly consistent. No surveyed kernel does this. INFERRED.
  5. Do not copy the prior art’s shortcuts. VERIFIED in the listed code.
    • fast-math builds (TurboMind, xformers, all FA-V100 forks).
    • Q pre-scaled in fp16 (llama.cpp MMA, xormal, ZRayZzz).
    • fp16 raw scores (ZRayZzz; engine sm70_79t).
    • fp16 O partials (1Cat attn_split).
    • fp16 output accumulators with max offsets (llama.cpp).
    • LSE-form merges at large LSE.

    KATT1’s FP32 S, P hi/lo split and FP32 (m,l) merge already exceed every surveyed kernel.


1. What failed in R6

Sources: R6-owner/run-20261101T163248Z/slot-KATT1-hmma.out and the V100 server [...]/slot.json (read-only). VERIFIED.

T today 0012 (all ks) emulation diff #1277 comparator
2 0.00/0.000 0.00/0.000 0.00 0.00
6 0.85/0.004 1.28/0.005 0.001 2.63
12 1.33/0.005 0.53/0.005 0.125 2.67
16 0.64/0.002 2.75/0.004 4.5 9.63
18 0.50/0.002 0.50/0.002 1.0 0.93

Reading (INFERRED):

1.1 Error budget at q std 2000 (INFERRED arithmetic)

Setup.

Version 1 (one chain per warp output):

Version 2 (KATT1 worktree defaults: QK_CHAIN 1, PV_FRESH 1):

Output sensitivity.


2. Per-implementation survey

Columns: (a) where scale and log2e are applied; (b) QK accumulator on sm70 and how it chains; (c) running max; (d) P precision before P·V; (e) split combine; (f) attention sink; (g) large-logit or Volta numerics notes.

2.1 NVIDIA cutlass examples/41 (fused MHA), e13f5cd414. All VERIFIED (helper read).

2.2 xformers cutlass FMHA, 3e62181f37 (removed at 3d947a67fa). VERIFIED.

2.3 Dao-AILab flash-attention: no sm70 in any upstream version. VERIFIED.

2.4 zhinianqin/flash-attention-v100, c2eda5e611. This is the engine’s FA pin. VERIFIED.

2.5 ai-bond/flash-attention-v100, 0ababf82e3. VERIFIED.

2.6 ZRayZzz/flash-attention-v100, 05eace2700. VERIFIED.

2.7 xormal/V100-SM_70-Flash-Attn-2-v1, 05c632aa8e (8 stars). VERIFIED.

2.8 LMDeploy TurboMind, dd4db0da27. VERIFIED (helper read; I re-read the softmax, merge, reduce and sink lines and the 576 instantiation).

2.9 llama.cpp CUDA flash attention, dde879bc4f. VERIFIED (helper read; constants re-read).

Volta dispatch (fattn.cu L691-699):

Per kernel:

Kernel Scale and exp QK accumulation P and output accumulator
VEC Q in FP32, times scale in FP32 (fattn-vec.cuh L241-245); expf FP32 SIMT P FP32; float2 accumulator
TILE q × scale in FP32 → half2 → × 0.25 in half; KQ × 4 after the dot (fattn-tile.cuh L925, L932, L936, L631) each product rounded to fp16 (L546) P half; half2 accumulator
MMA Q rounded to fp16, then times scale in fp16 (fattn-mma-f16.cuh L1278, L1306) m8n8k4 f32, chained through C (mma.cuh L1522-1529) P half2; fp16-accumulating m8n8k4 for P·V (L1542-1549)

Shared:

2.10 1Cat #1277, 899e9ed331. I read all three paths. VERIFIED unless noted.

Served decode (attention_ops.py _attend_fused L925-941 with _sink_softmax L898-922; VLLM_V41_ATTN_FUSED=1):

_sparse_attention_fused (L371-414; VLLM_V41_FUSED_SPARSE, listed in #1277 HANDOFF as a dead end):

v41k.cu attn_split / attn_combine (benchmarks/deepseek_v41/v41k/v41k.cu L2033-2165; compiled but unused by the served path, per KATT1 report §7):

#1280 (open, DS-V4 fused sparse-MLA softmax): my earlier sweep (candidates.tsv U04) recorded it as “the same softmax for DS-V4”. Code not read in this pass. UNVERIFIED.

2.11 Our engine’s V100 attention (1Cat-vLLM port, the V100 server engine 23cec6a). VERIFIED (read-only grep).

2.12 Reference: KATT1 0012 (worktree, uncommitted version 2 over 329e92a). VERIFIED.


3. Techniques aimed directly at tensor-core accumulator rounding

  1. Ootomo/Yokota TCEC mma_rn_sync (wmma_extension a9e6b34e40). VERIFIED (helper read; I re-read L668-695).
    • mma_sync defaults to mma_rn_sync (tcec.hpp L898-917).
    • Each k-block’s main term is computed with zero_op(tmp); mma_op(tmp, a, b, tmp) and then added with ordinary FP32 adds (frag_d.x[i] += tmp.x[i], L672-694).
    • The error-correction terms use operands scaled by 2048 for half (scale.hpp L11-20) and are chained through C.
    • mma_rz_sync (L775) is the variant that chains everything through C.
    • sm_70 is supported. cuMpSGEMM 42dd92ea1c calls it (device_tcec_wrapper.hpp L105).
    • The correction terms only matter when inputs are inexact in fp16. 0012’s Q and K are exact fp16, so only the RN-accumulation half applies. INFERRED.
  2. cutlass MmaTensorOpFastF32 (3xTF32 big/small split; mma_tensor_op_fast_f32.h L85-139, L364-373). VERIFIED. It requires sm80 or newer, so it is not usable on V100.
  3. 1Cat attn_split: two d-half chains, RN-summed (§2.10). VERIFIED.
  4. cutlass FMHA large-D path: per-key-block P·V from a cleared accumulator, merged in FP32 outside the tensor core (§2.1e). VERIFIED.
  5. Avoid tensor cores for decode QK: TurboMind SIMT FP32 (§2.8) and llama.cpp VEC (§2.9). VERIFIED. This is the expensive option (§4 item 9).

Background (UNVERIFIED citation details, not read in this pass):


4. Ranked techniques for KATT1, with expected cost

Baseline: version 1 MEASURED 28.6 µs/layer at T6, 35.3 at T12 and 53.1 at T18 (ks64, v640). Costs are INFERRED from instruction counts unless marked; none were measured.

Count used below: QK per warp per 32-key block = 2 n-tiles × 8 k16 steps, so one 8-float fragment add per step is 128 FADD per thread per block. At T6, ks64 and v640 there are 60 CTAs, each with 2 blocks, so about 256 FADD/thread on the critical path, about 0.2-0.4 µs at 1.4 GHz before overlap with HMMA and loads.

# Technique Prior art (pointer, tag) What it fixes Expected cost    
1 QK short chains (C=0 per k16) plus RN FP32 sum. Already in KATT1 v2 as QK_CHAIN=1. TCEC mma_rn_sync L672-694 (V); 1Cat attn_split 2 chains (V) Removes the 32-step RZ chain bias, the dominant term (§1.1). This should also shrink the emulation mismatch, because the model’s sensitivity to HMMA internals scales with chain length (I). +0.2-1.0 µs at T6, +0.5-2 at T18 (I). If register pressure or SASS shows a regression, QK_CHAIN=2 (2 × 4-step chains per partial) is the halfway point.    
2 P·V fresh per 16-key step, o = fmaf(o, alpha, pv). Already in v2 as PV_FRESH=1. cutlass FMHA large-D epilogue merge (V); TCEC (V) Matters most at normal scale (long P·V chains over 640 keys) for “accuracy beyond the bar”. Little effect at q std 2000, where P is one-hot (I). +0.2-1.0 µs (64 FFMA per thread per 16-key step) (I)    
3 exp2 FFMA fold with raw-unit max (details below). TurboMind impl_simt.h L478/L495, reduce.cu L104-106, sink L533 (V); FA-V100 c2eda5e611 softmax.h L70-79 (V); engine sm70_v37/tail.cu L329 (V) Removes the per-element rounding of a*scale at s ≈ 8.4e3 (≤ 4.9e-4 logit, up to 4.9e-4 relative in p). After v2 that is about a quarter to a third of the remaining logit error at q std 2000 (I). About 0 µs. exp2f is no dearer than expf without fast math; one FFMA replaces FMUL + FADD (I).
4 Offset-consistent rescale and merge (details below). No surveyed kernel does this. none (idea from TurboMind’s raw-M (m,l) form) Removes the per-block ≤ 1 ulp(m·c) offset mismatch that TurboMind and FA2 accept (≤ 6.8e-4 relative in p_old at m·c ≈ 1.2e4) (I). About 0 µs: one FMUL per row per block (I)    
5 Compensated (Neumaier) or two-level sum of the 8 chain partials and the 4 warp partials. none in surveyed code (I) Cuts the remaining round-to-nearest summation error from about sqrt(11) roundings to about 1 (I). Worth it only if R6-HMMA2 still shows more than today’s ulps at q std 2000. +0.3-1.0 µs (3 extra FADD per add) (I)    
6 Keep FP32 scores; do not pre-scale Q in fp16. Negative examples: llama.cpp MMA L1278/L1306, xormal L215, ZRayZzz fp16 S (V) Relative FP accumulation error does not depend on scale, so pre-scaling buys no accuracy. 512^-0.5 = 2^-4.5 is not a power of two, so fp16(q*scale) adds a rounding per element. A 2^-4 pre-scale is exact but only helps fp16 range, which is not a problem here (I). 0 µs    
7 Keep the build free of fast math. Negative examples: TurboMind CMake L15, xformers setup.py L456, FA-V100 forks (V) __expf = ex2(x*log2e); its error grows with x (I). 0 µs
8 Adaptive FP32 recompute of near-max keys (details below). TurboMind and llama.cpp use FP32 SIMT for Volta decode outright (V) Exact-ish scores where it matters, at no cost in normal rows (I). About 0.1 µs for the always-on check (a warp-uniform compare on the block max). +2-5 µs only on flagged rows. Kernel complexity: medium (I).    
9 Full FP32 SIMT QK, as TurboMind and llama.cpp VEC do. TurboMind 576-dim MLA decode = MMA_SIMT (V) Removes tensor-core rounding entirely. About +15-25 µs at T6. QK is 2·16·640·512 = 10.5 MFLOP per token, 63 MFLOP at T6; 4.5 µs at the 14 TFLOPS peak, 3-5x that in practice (I). This loses most of the 151 µs/call gain. Not recommended.    
10 m8n8k4 PTX with C=0 per k=4 step (finest TCEC granularity). TCEC works at the k-block level (V) Shortens each RZ chain from 4 steps to 1, but grows the round-to-nearest adds per warp from 8 to 32, so it is a net loss unless paired with #5. Needs the quad-pair fragment layout rewritten (I). +1-3 µs (I). Not recommended.    

Details for row 3.

Details for row 4.

Details for row 8.

Priority.

4.1 Test-side notes (INFERRED, for KATT1’s harness; no change made by me)


5. Summary table

Implementation Scale / log2e QK on sm70 Running max P before P·V Split combine Sink fast-math
cutlass ex41 ×(scale·log2e) after MMA, then exp2 m8n8k4 f32, chained through C -inf fp16 RN none (LSE out) no n/a
xformers (pre-removal) same as cutlass m8n8k4 f32, chained -inf fp16 RN backward only no yes
FA upstream no sm70 - - - - - -
FA-V100 zhinianqin (engine pin) FFMA exp2, raw-unit max m8n8k4 f32, chained 0 init / Check_inf fp16 LSE, expf/logf no yes
FA-V100 ai-bond ×scale after, __expf wmma m16n16k16 f32, chained -1e30 fp16 RN none no yes
FA-V100 ZRayZzz ×scale in exp m8n8k4 f32 → fp16 S -inf fp16 none no n/a
TurboMind decode FFMA exp2, raw-unit M FP32 SIMT -inf fp16 RN, FP32 FMA (m,l) exp2 raw M yes, denominator yes
llama.cpp VEC/TILE/MMA FP32 Q·scale / fp16 / fp16 FP32 / fp16 products / m8n8k4 -FLT_MAX/2 + offset FP32 / half / half (m,l) expf yes, also max n/a
1Cat #1277 served ×scale after (Triton) cuBLAS (I: tensor core, chained) max incl. sink fp16 none yes, max + denominator n/a
1Cat attn_split ×scale after wmma f32, 2 chains, RN-summed -inf fp16 RN, fp16 O (m,l) expf yes, max + denominator n/a
Engine sm70_v37 tail FFMA exp2 (FA-V100) m8n8k4 f32 (FA-V100) -inf, safe fp16 per FA-V100 n/a n/a
KATT1 0012 v2 ×scale after, expf wmma f32, C=0 per k16, RN sum -1e30 floor fp16 hi + lo (m,l) expf FP32 yes, denominator no

All cells VERIFIED except “1Cat #1277 served QK” (INFERRED cuBLAS behaviour) and cells marked n/a (not checked).