Origin. PLI Labs research archive, file
v100-research/deepseek-v4.1-flash/RANGE_AUDIT.md. Published 2026-11-18 in the Volta V100 research repository. Edited for publication. Machine names, private file paths, network addresses, session names and the operator’s name were replaced with generic terms (the V100 server,[private path],[address],the owner). Links to files that are not published were turned into plain text. No figure and no finding was changed. Dates inside the note are its own; later results are in the build pages of this repository.
Lane L-REF, generated 2026-10-07 03:22Z by tools/ds41_ref/range_audit.py from golden.py range statistics. Reference: official deepseek-ai/DeepSeek-V4.1-Flash @ 2cba9e42, FP32 layer-streamed transcription (tools/ds41_ref), batch 1, world_size 1.
Summary and recommendations (L-REF)
Data: 10 runs (prompts x modes x index-K variants), every prefill token and 31 greedy decode steps; max over all of them. Both numerics modes agree to within a few percent.
- Residual stream: BF16, never FP16. Max 3.625e+05 (> 65,504). The extremes sit on BOS tokens — position 0 and also BOS tokens generated mid-sequence (p2_code continuation) — as on V4.
- hc_pre collapse (sublayer input before RMSNorm) reaches 4.617e+05: it must be computed and held in FP32 (or BF16) until the norm; only the normed output may be FP16.
- MoE combine (routed + shared) and its all-reduce stay FP32 (max 2.283e+04 > 8,192). TP4 partials peak lower but the summed value does not fit the rule.
- Attention sublayer output may go FP16 at P5 (max 441 « 8,192).
- Per-expert outputs (port order, weight after w2): max 1.502e+04 vs the 16,384 rule — allowed, but with only 8 % margin on four prompts. L-REF recommendation: keep the FP32 epilogue (a clamp would be a model change, §4.3).
- Engram wkv output, q.k logits and indexer scores are far from any FP16 limit (tables below).
Runs
| prompt | mode | prompt tokens | decode steps | provisional |
|---|---|---|---|---|
| p0_smoke | ref-exact | 6 | 31 | False |
| p0_smoke | v100-semantic | 6 | 31 | False |
| p1_text | ref-exact | 85 | 31 | False |
| p1_text | v100-semantic | 85 | 31 | False |
| p2_code | ref-exact | 158 | 31 | False |
| p2_code | v100-semantic | 158 | 31 | False |
| p3_doc | ref-exact+ownk | 2046 | 31 | False |
| p3_doc | ref-exact | 2046 | 31 | False |
| p3_doc | v100-semantic+ownk | 2046 | 31 | False |
| p3_doc | v100-semantic | 2046 | 31 | False |
ref-exact: max |x| per layer (all prompts, all steps)
Columns: stream = stream, str_attn = stream_attn, hcx_attn = hc.attn_x, hcx_ffn = hc.ffn_x, attn_o = attn.o, attn_out = attn.out, attn_tp4 = attn.out_tp4_partial, qk = attn.qk_logit, exp_w = moe.expert_out_max, exp_unw = moe.expert_out_unweighted_max, shared = moe.shared_out, moe_comb = moe.combine_f32, moe_tp4 = moe.out_tp4_partial, eng_wkv = engram.wkv_out, eng_add = engram.gate_add, idx = idx.score
| layer | stream | str_attn | hcx_attn | hcx_ffn | attn_o | attn_out | attn_tp4 | qk | exp_w | exp_unw | shared | moe_comb | moe_tp4 | eng_wkv | eng_add | idx |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 9.31 | 1.92 | 0.66 | 1.84 | 3.5 | 10.1 | 4.16 | 17.3 | 9.88 | 13.2 | 7.22 | 10.3 | 8.27 | — | — | — |
| 1 | 77.5 | 9.25 | 5.03 | 3.38 | 3.66 | 9.38 | 3.55 | 18.4 | 43.8 | 37 | 2.45 | 43.9 | 19.1 | 104 | 5.57 | — |
| 2 | 68.5 | 72.5 | 44 | 53.8 | 2.3 | 5.81 | 2.93 | 11.1 | 17.6 | 30.6 | 2.31 | 19.7 | 14 | — | — | 6.34 |
| 3 | 59.8 | 64 | 53 | 30.9 | 2.95 | 4.69 | 2.22 | 11.1 | 15.2 | 17.5 | 2.14 | 16.6 | 15.1 | — | — | — |
| 4 | 64.5 | 59 | 61 | 36.5 | 3 | 4 | 1.89 | 10.6 | 7.25 | 11.5 | 11.6 | 11.9 | 7.84 | — | — | — |
| 5 | 86.5 | 64.5 | 40.5 | 49 | 2.62 | 6.12 | 2.23 | 10.1 | 1.41 | 5.97 | 12.8 | 12.8 | 9.86 | — | — | — |
| 6 | 191 | 86.5 | 74.5 | 37 | 2.45 | 2.67 | 1.56 | 9.3 | 88 | 84.1 | 1.88 | 89.8 | 26.8 | — | — | — |
| 7 | 189 | 191 | 55.2 | 119 | 2.52 | 4.94 | 4.29 | 12.3 | 24 | 19.3 | 8.62 | 24.5 | 53.6 | — | — | — |
| 8 | 2.5e+03 | 189 | 84 | 59.8 | 3.17 | 30.8 | 25.6 | 15 | 1.16e+03 | 920 | 3.14 | 1.16e+03 | 386 | — | — | 7.09 |
| 9 | 2.48e+03 | 2.5e+03 | 776 | 255 | 3.12 | 8.81 | 4 | 10.7 | 20.6 | 26.7 | 0.977 | 20.8 | 12.9 | — | — | — |
| 10 | 2.86e+03 | 2.48e+03 | 396 | 564 | 2.8 | 7.34 | 3.19 | 13.9 | 490 | 397 | 0.434 | 490 | 168 | — | — | — |
| 11 | 2.86e+03 | 2.86e+03 | 1.52e+03 | 1.78e+03 | 2.97 | 3.25 | 1.94 | 9.05 | 3.03 | 11.1 | 2.69 | 3.21 | 1.74 | — | — | — |
| 12 | 2.88e+03 | 2.86e+03 | 1.18e+03 | 470 | 4.25 | 8.81 | 3.53 | 16 | 6.94 | 8.63 | 0.832 | 6.99 | 4.87 | — | — | — |
| 13 | 2.94e+03 | 2.88e+03 | 1.82e+03 | 684 | 2.89 | 5 | 2.17 | 14.5 | 55.5 | 47.7 | 17 | 55.7 | 21.8 | — | — | — |
| 14 | 5.31e+03 | 2.94e+03 | 124 | 808 | 3.44 | 23.6 | 19.1 | 13 | 1.26e+03 | 974 | 21.8 | 1.26e+03 | 402 | 224 | 13.9 | 6.34 |
| 15 | 1.382e+04 | 5.31e+03 | 1.98e+03 | 5.41e+03 | 3.48 | 52.8 | 51.7 | 13.6 | 5.92e+03 | 5.32e+03 | 68.5 | 5.93e+03 | 1.7e+03 | — | — | — |
| 16 | 2.112e+04 | 1.382e+04 | 3.17e+03 | 1.96e+03 | 3.53 | 6.62 | 5 | 14.2 | 3.66e+03 | 2.67e+03 | 3.33 | 3.66e+03 | 1.07e+03 | — | — | — |
| 17 | 2.458e+04 | 2.112e+04 | 540 | 4.77e+03 | 4 | 3.33 | 2.34 | 17.8 | 1.71e+03 | 1.34e+03 | 1.24 | 1.71e+03 | 555 | — | — | — |
| 18 | 2.573e+04 | 2.458e+04 | 7.49e+03 | 1.318e+04 | 4.09 | 3.67 | 2.28 | 12.6 | 600 | 444 | 11.1 | 603 | 292 | — | — | — |
| 19 | 2.957e+04 | 2.573e+04 | 5.34e+03 | 6.59e+03 | 5.81 | 6.69 | 3.52 | 28.7 | 1.9e+03 | 1.56e+03 | 86.5 | 1.9e+03 | 521 | — | — | — |
| 20 | 5.376e+04 | 2.957e+04 | 1.306e+04 | 6.08e+03 | 4.91 | 5.09 | 3.62 | 12.3 | 1.907e+04 | 1.317e+04 | 12.8 | 1.908e+04 | 5.3e+03 | — | — | 5.41 |
| 21 | 6.912e+04 | 5.376e+04 | 2.099e+04 | 4.25e+04 | 4.09 | 33.8 | 17.3 | 13.8 | 1.325e+04 | 1.002e+04 | 70.5 | 1.332e+04 | 3.45e+03 | — | — | — |
| 22 | 1.019e+05 | 6.912e+04 | 1.158e+04 | 4.531e+04 | 6.06 | 4.59 | 3.04 | 14 | 1.638e+04 | 1.165e+04 | 29.5 | 1.641e+04 | 4.51e+03 | — | — | — |
| 23 | 1.413e+05 | 1.019e+05 | 3.085e+04 | 1.92e+04 | 4.53 | 4.78 | 5.44 | 11.5 | 1.984e+04 | 1.338e+04 | 162 | 2e+04 | 5.1e+03 | — | — | — |
| 24 | 1.772e+05 | 1.413e+05 | 5.043e+04 | 5.658e+04 | 4.25 | 6.25 | 4.09 | 11.5 | 1.792e+04 | 1.258e+04 | 134 | 1.806e+04 | 4.79e+03 | — | — | 5.28 |
| 25 | 2.109e+05 | 1.772e+05 | 4.966e+04 | 5.171e+04 | 4.19 | 6.53 | 3.5 | 12.2 | 1.69e+04 | 1.231e+04 | 199 | 1.709e+04 | 4.5e+03 | — | — | — |
| 26 | 2.335e+05 | 2.109e+05 | 4.096e+04 | 4.838e+04 | 3.86 | 8.81 | 5.05 | 16.1 | 1.114e+04 | 7.7e+03 | 446 | 1.154e+04 | 2.97e+03 | — | — | — |
| 27 | 2.499e+05 | 2.335e+05 | 1.495e+05 | 1.741e+04 | 4.81 | 6.06 | 4.56 | 12.9 | 1.107e+04 | 9.62e+03 | 248 | 1.132e+04 | 3.13e+03 | — | — | — |
| 28 | 2.949e+05 | 2.499e+05 | 9.984e+04 | 8.09e+04 | 3.83 | 13.7 | 7.58 | 14.2 | 2.202e+04 | 1.474e+04 | 756 | 2.24e+04 | 5.87e+03 | — | — | 5.03 |
| 29 | 3.256e+05 | 2.949e+05 | 3.871e+05 | 7.168e+04 | 6.19 | 8.38 | 7.24 | 10.8 | 1.472e+04 | 1.021e+04 | 644 | 1.51e+04 | 3.88e+03 | — | — | — |
| 30 | 3.297e+05 | 3.256e+05 | 4.526e+05 | 7.834e+04 | 5.81 | 11.1 | 7.9 | 13 | 3.25e+03 | 4.89e+03 | 466 | 3.26e+03 | 862 | — | — | — |
| 31 | 3.359e+05 | 3.297e+05 | 2.785e+05 | 9.677e+04 | 5 | 7.75 | 4.53 | 13.4 | 4.96e+03 | 3.85e+03 | 178 | 4.97e+03 | 1.31e+03 | — | — | — |
| 32 | 3.359e+05 | 3.359e+05 | 3.727e+05 | 1.311e+05 | 3.69 | 8.25 | 3.77 | 12.2 | 196 | 1.46e+03 | 255 | 356 | 122 | — | — | 5.78 |
| 33 | 3.4e+05 | 3.359e+05 | 1.679e+05 | 9.933e+04 | 5.06 | 27 | 11.2 | 13.5 | 6.24e+03 | 4.16e+03 | 776 | 7.02e+03 | 1.92e+03 | — | — | — |
| 34 | 3.42e+05 | 3.4e+05 | 3.174e+05 | 5.632e+04 | 4.16 | 22.9 | 10.4 | 14.8 | 2.64e+03 | 4.54e+03 | 215 | 2.66e+03 | 720 | — | — | — |
| 35 | 3.42e+05 | 3.42e+05 | 3.502e+05 | 6.016e+04 | 3.88 | 11.3 | 5.65 | 12.6 | 2.43e+03 | 2.54e+03 | 16.2 | 2.86e+03 | 763 | — | — | — |
| 36 | 3.584e+05 | 3.42e+05 | 2.112e+04 | 6.81e+04 | 5.16 | 32.8 | 20.6 | 41.9 | 1.114e+04 | 8.33e+03 | 900 | 1.149e+04 | 3.51e+03 | — | — | 5.56 |
| 37 | 3.584e+05 | 3.564e+05 | 1.12e+04 | 5.888e+04 | 5.16 | 13.7 | 13.6 | 18.1 | 1.184e+04 | 7.89e+03 | 1.33e+03 | 1.275e+04 | 3.97e+03 | — | — | — |
| 38 | 3.584e+05 | 3.584e+05 | 3.83e+05 | 3.021e+04 | 4.31 | 35.2 | 17.1 | 86.7 | 4.02e+03 | 3.98e+03 | 434 | 4e+03 | 1.41e+03 | — | — | — |
| 39 | 3.584e+05 | 3.584e+05 | 1.193e+05 | 4.096e+04 | 36 | 484 | 355 | 102 | 1.453e+04 | 9.69e+03 | 2.82e+03 | 1.583e+04 | 4.47e+03 | — | — | — |
Overall maxima (layer, prompt:token position):
stream: 3.584e+05 at layer 39 (p0_smoke:0)stream_attn: 3.584e+05 at layer 39 (p2_code:170)hc.attn_x: 4.526e+05 at layer 30 (p0_smoke:0)hc.ffn_x: 1.311e+05 at layer 32 (p2_code:1)attn.o: 36 at layer 39 (p2_code:77)attn.out: 484 at layer 39 (p3_doc:430)attn.out_tp4_partial: 355 at layer 39 (p3_doc:430)attn.qk_logit: 102 at layer 39 (p1_text:0)moe.expert_out_max: 2.202e+04 at layer 28 (p0_smoke:0)moe.expert_out_unweighted_max: 1.474e+04 at layer 28 (p3_doc:0)moe.shared_out: 2.82e+03 at layer 39 (p2_code:75)moe.combine_f32: 2.24e+04 at layer 28 (p0_smoke:0)moe.out_tp4_partial: 5.87e+03 at layer 28 (p2_code:0)engram.wkv_out: 224 at layer 14 (p3_doc:726)engram.gate_add: 13.9 at layer 14 (p3_doc:0)idx.score: 7.09 at layer 8 (p3_doc:1524)
ref-exact: §4.4 rules
- Attention sublayer output (wo_b, all-reduced) max = 484 → FP16 allowed (rule < 8,192).
- MoE combine (routed + shared) max = 2.24e+04 → FP32 required (rule < 8,192).
- TP4 partial sums before the all-reduce: max = 5.87e+03 (attention and MoE) → below 8,192; the MoE sum exceeds it, so the MoE all-reduce output stays FP32 regardless.
- Per-expert w2 output: reference order (routing weight before w2) max = 2.202e+04; port order (weight after w2) max = 1.474e+04 → FP16 epilogue allowed by the rule (< 16,384), margin 10 %.
- Residual stream max = 3.584e+05: exceeds FP16 range (stream stays BF16, A4; FP32 variance mandatory).
ref-exact+ownk: max |x| per layer (all prompts, all steps)
Columns: stream = stream, str_attn = stream_attn, hcx_attn = hc.attn_x, hcx_ffn = hc.ffn_x, attn_o = attn.o, attn_out = attn.out, attn_tp4 = attn.out_tp4_partial, qk = attn.qk_logit, exp_w = moe.expert_out_max, exp_unw = moe.expert_out_unweighted_max, shared = moe.shared_out, moe_comb = moe.combine_f32, moe_tp4 = moe.out_tp4_partial, eng_wkv = engram.wkv_out, eng_add = engram.gate_add, idx = idx.score
| layer | stream | str_attn | hcx_attn | hcx_ffn | attn_o | attn_out | attn_tp4 | qk | exp_w | exp_unw | shared | moe_comb | moe_tp4 | eng_wkv | eng_add | idx |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 9.31 | 1.79 | 0.617 | 1.78 | 3.47 | 8.44 | 4.16 | 17.3 | 9.88 | 13.2 | 6.59 | 10.3 | 8.27 | — | — | — |
| 1 | 77.5 | 9.25 | 5.03 | 2.88 | 3.66 | 9.12 | 3.55 | 18.4 | 43.8 | 37 | 1.64 | 43.8 | 19.1 | 103 | 1.47 | — |
| 2 | 68.5 | 72.5 | 36.5 | 53.8 | 2.25 | 5.62 | 2.44 | 11 | 17.6 | 30.6 | 2.31 | 19.7 | 14 | — | — | 5.78 |
| 3 | 59.8 | 64 | 53 | 30.9 | 2.95 | 4.69 | 2.1 | 11.1 | 15.2 | 17.5 | 2.14 | 16.6 | 15.1 | — | — | — |
| 4 | 64.5 | 56.2 | 61 | 36.5 | 3 | 4 | 1.78 | 10.3 | 1.55 | 2.14 | 11.6 | 11.9 | 7.84 | — | — | — |
| 5 | 86.5 | 64.5 | 34.2 | 49 | 2.62 | 5.94 | 2.18 | 10.1 | 0.949 | 2.18 | 12.8 | 12.8 | 9.86 | — | — | — |
| 6 | 86.5 | 86.5 | 74.5 | 12.1 | 2.45 | 2.67 | 1.56 | 9.3 | 1.38 | 2.5 | 1.87 | 1.66 | 1.83 | — | — | — |
| 7 | 86.5 | 86.5 | 55.2 | 119 | 2.52 | 4.94 | 4.29 | 12.3 | 24 | 19 | 8.62 | 24.5 | 53.6 | — | — | — |
| 8 | 290 | 86.5 | 84 | 14.4 | 3.17 | 30.8 | 25.6 | 15 | 123 | 112 | 3.12 | 123 | 66.6 | — | — | 7.09 |
| 9 | 290 | 290 | 560 | 255 | 3.12 | 8.81 | 4 | 10.7 | 0.781 | 2.2 | 0.977 | 1.41 | 0.699 | — | — | — |
| 10 | 1.26e+03 | 290 | 396 | 215 | 2.8 | 7.34 | 3.19 | 13.9 | 484 | 393 | 0.434 | 484 | 168 | — | — | — |
| 11 | 1.26e+03 | 1.26e+03 | 1.5e+03 | 1.18e+03 | 2.97 | 3.25 | 1.94 | 9.05 | 0.973 | 1.91 | 2.69 | 3.21 | 1.02 | — | — | — |
| 12 | 1.26e+03 | 1.26e+03 | 1.17e+03 | 378 | 4.25 | 8.31 | 3.29 | 16 | 2.11 | 3.76 | 0.832 | 2.23 | 1.14 | — | — | — |
| 13 | 1.26e+03 | 1.26e+03 | 1.81e+03 | 318 | 2.89 | 5 | 2.17 | 12.8 | 11 | 23.4 | 17 | 16.6 | 15.6 | — | — | — |
| 14 | 1.57e+03 | 1.27e+03 | 84 | 808 | 3.44 | 22.1 | 16.3 | 13 | 165 | 132 | 21.1 | 165 | 61.9 | 224 | 13.9 | 6.34 |
| 15 | 1.318e+04 | 1.57e+03 | 152 | 1.9e+03 | 3.38 | 35.2 | 33 | 13.6 | 5.82e+03 | 4.03e+03 | 42.5 | 5.84e+03 | 1.68e+03 | — | — | — |
| 16 | 1.408e+04 | 1.318e+04 | 420 | 960 | 3.53 | 6.19 | 4.73 | 14.4 | 456 | 330 | 1.56 | 456 | 247 | — | — | — |
| 17 | 1.44e+04 | 1.408e+04 | 434 | 1.11e+03 | 4 | 3.33 | 2.34 | 17.8 | 165 | 132 | 1.24 | 165 | 50.5 | — | — | — |
| 18 | 1.491e+04 | 1.44e+04 | 692 | 1.5e+03 | 4.09 | 3.67 | 2.24 | 12.3 | 229 | 168 | 11.1 | 242 | 76.5 | — | — | — |
| 19 | 1.536e+04 | 1.491e+04 | 5.18e+03 | 3.63e+03 | 5.81 | 5.16 | 3.04 | 28.7 | 125 | 111 | 86.5 | 211 | 75.1 | — | — | — |
| 20 | 5.35e+04 | 1.536e+04 | 8.32e+03 | 5.38e+03 | 4.91 | 5.09 | 3.62 | 12.3 | 1.907e+04 | 1.314e+04 | 12.4 | 1.908e+04 | 5.3e+03 | — | — | 5.41 |
| 21 | 6.861e+04 | 5.35e+04 | 1.338e+04 | 4.25e+04 | 4.06 | 33.8 | 17.3 | 13.8 | 1.318e+04 | 9.97e+03 | 70 | 1.325e+04 | 3.44e+03 | — | — | — |
| 22 | 1.014e+05 | 6.861e+04 | 1.133e+04 | 4.506e+04 | 4.75 | 4.59 | 3 | 14 | 1.638e+04 | 1.16e+04 | 28.4 | 1.641e+04 | 4.51e+03 | — | — | — |
| 23 | 1.413e+05 | 1.014e+05 | 2.048e+04 | 1.894e+04 | 4.53 | 4.53 | 5.14 | 11.5 | 1.984e+04 | 1.338e+04 | 162 | 2e+04 | 5.09e+03 | — | — | — |
| 24 | 1.772e+05 | 1.413e+05 | 2.918e+04 | 5.632e+04 | 4.25 | 4.41 | 3.97 | 11.5 | 1.792e+04 | 1.258e+04 | 128 | 1.806e+04 | 4.78e+03 | — | — | 5.28 |
| 25 | 2.109e+05 | 1.772e+05 | 2.534e+04 | 5.146e+04 | 4.19 | 5.25 | 3.5 | 12.2 | 1.69e+04 | 1.231e+04 | 198 | 1.709e+04 | 4.49e+03 | — | — | — |
| 26 | 2.335e+05 | 2.109e+05 | 4.07e+04 | 4.838e+04 | 3.86 | 8.81 | 5.05 | 16.1 | 1.107e+04 | 7.66e+03 | 438 | 1.148e+04 | 2.97e+03 | — | — | — |
| 27 | 2.499e+05 | 2.335e+05 | 9.779e+04 | 1.741e+04 | 4.81 | 6.06 | 4.56 | 12.9 | 8e+03 | 9.62e+03 | 172 | 8.17e+03 | 2.07e+03 | — | — | — |
| 28 | 2.949e+05 | 2.499e+05 | 3.302e+04 | 5.606e+04 | 3.81 | 9.5 | 7.58 | 14.2 | 2.202e+04 | 1.474e+04 | 388 | 2.24e+04 | 5.86e+03 | — | — | 5.03 |
| 29 | 3.256e+05 | 2.949e+05 | 3.85e+05 | 7.014e+04 | 6.16 | 8.38 | 7.06 | 10.8 | 1.472e+04 | 1.02e+04 | 388 | 1.51e+04 | 3.88e+03 | — | — | — |
| 30 | 3.297e+05 | 3.256e+05 | 4.526e+05 | 7.782e+04 | 5.81 | 9.69 | 7.57 | 11 | 3.2e+03 | 3.86e+03 | 172 | 3.21e+03 | 844 | — | — | — |
| 31 | 3.359e+05 | 3.297e+05 | 2.785e+05 | 9.626e+04 | 4.66 | 6.94 | 3.93 | 13.4 | 4.93e+03 | 3.48e+03 | 64.5 | 4.94e+03 | 1.31e+03 | — | — | — |
| 32 | 3.359e+05 | 3.359e+05 | 3.727e+05 | 5.658e+04 | 3.69 | 7.94 | 3.72 | 12.1 | 5.28 | 22.7 | 90 | 93.8 | 30.9 | — | — | 5.78 |
| 33 | 3.4e+05 | 3.359e+05 | 9.779e+04 | 2.982e+04 | 4.62 | 14.7 | 5.83 | 13.5 | 4.02e+03 | 4.06e+03 | 266 | 4.02e+03 | 1.06e+03 | — | — | — |
| 34 | 3.42e+05 | 3.4e+05 | 3.174e+05 | 1.555e+04 | 4 | 16.4 | 7.71 | 14.8 | 2.64e+03 | 2.08e+03 | 83 | 2.66e+03 | 717 | — | — | — |
| 35 | 3.42e+05 | 3.42e+05 | 3.502e+05 | 5.99e+04 | 3.88 | 8.75 | 4 | 12.6 | 2.43e+03 | 2.54e+03 | 5.94 | 2.86e+03 | 761 | — | — | — |
| 36 | 3.584e+05 | 3.42e+05 | 2.112e+04 | 6.81e+04 | 5.16 | 32.2 | 19.4 | 41.9 | 1.114e+04 | 8.33e+03 | 572 | 1.149e+04 | 3.51e+03 | — | — | 5.56 |
| 37 | 3.564e+05 | 3.543e+05 | 3.78e+03 | 5.862e+04 | 5.16 | 13.7 | 13.6 | 18.1 | 1.184e+04 | 7.89e+03 | 908 | 1.275e+04 | 3.97e+03 | — | — | — |
| 38 | 3.564e+05 | 3.564e+05 | 3.83e+05 | 3.008e+04 | 3.56 | 32.8 | 16.4 | 60.1 | 3.98e+03 | 2.66e+03 | 434 | 4e+03 | 1.41e+03 | — | — | — |
| 39 | 3.584e+05 | 3.564e+05 | 7.936e+04 | 4.096e+04 | 30 | 484 | 355 | 93 | 1.453e+04 | 9.69e+03 | 2.75e+03 | 1.583e+04 | 4.47e+03 | — | — | — |
Overall maxima (layer, prompt:token position):
stream: 3.584e+05 at layer 39 (p3_doc:0)stream_attn: 3.564e+05 at layer 39 (p3_doc:0)hc.attn_x: 4.526e+05 at layer 30 (p3_doc:0)hc.ffn_x: 9.626e+04 at layer 31 (p3_doc:0)attn.o: 30 at layer 39 (p3_doc:549)attn.out: 484 at layer 39 (p3_doc:430)attn.out_tp4_partial: 355 at layer 39 (p3_doc:430)attn.qk_logit: 93 at layer 39 (p3_doc:0)moe.expert_out_max: 2.202e+04 at layer 28 (p3_doc:0)moe.expert_out_unweighted_max: 1.474e+04 at layer 28 (p3_doc:0)moe.shared_out: 2.75e+03 at layer 39 (p3_doc:859)moe.combine_f32: 2.24e+04 at layer 28 (p3_doc:0)moe.out_tp4_partial: 5.86e+03 at layer 28 (p3_doc:0)engram.wkv_out: 224 at layer 14 (p3_doc:726)engram.gate_add: 13.9 at layer 14 (p3_doc:0)idx.score: 7.09 at layer 8 (p3_doc:1524)
ref-exact+ownk: §4.4 rules
- Attention sublayer output (wo_b, all-reduced) max = 484 → FP16 allowed (rule < 8,192).
- MoE combine (routed + shared) max = 2.24e+04 → FP32 required (rule < 8,192).
- TP4 partial sums before the all-reduce: max = 5.86e+03 (attention and MoE) → below 8,192; the MoE sum exceeds it, so the MoE all-reduce output stays FP32 regardless.
- Per-expert w2 output: reference order (routing weight before w2) max = 2.202e+04; port order (weight after w2) max = 1.474e+04 → FP16 epilogue allowed by the rule (< 16,384), margin 10 %.
- Residual stream max = 3.584e+05: exceeds FP16 range (stream stays BF16, A4; FP32 variance mandatory).
v100-semantic: max |x| per layer (all prompts, all steps)
Columns: stream = stream, str_attn = stream_attn, hcx_attn = hc.attn_x, hcx_ffn = hc.ffn_x, attn_o = attn.o, attn_out = attn.out, attn_tp4 = attn.out_tp4_partial, qk = attn.qk_logit, exp_w = moe.expert_out_max, exp_unw = moe.expert_out_unweighted_max, shared = moe.shared_out, moe_comb = moe.combine_f32, moe_tp4 = moe.out_tp4_partial, eng_wkv = engram.wkv_out, eng_add = engram.gate_add, idx = idx.score
| layer | stream | str_attn | hcx_attn | hcx_ffn | attn_o | attn_out | attn_tp4 | qk | exp_w | exp_unw | shared | moe_comb | moe_tp4 | eng_wkv | eng_add | idx |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 9.21 | 1.92 | 0.66 | 1.85 | 3.46 | 10.1 | 4.18 | 17.5 | 9.83 | 13.1 | 7.23 | 10.2 | 8.28 | — | — | — |
| 1 | 76.9 | 9.17 | 5.01 | 3.35 | 3.65 | 10.2 | 3.58 | 18.4 | 43.7 | 36.6 | 2.49 | 43.8 | 19.9 | 104 | 5.83 | — |
| 2 | 68.2 | 71.9 | 43.4 | 55.8 | 2.27 | 5.84 | 2.94 | 11 | 17.9 | 31 | 2.3 | 19.9 | 14.2 | — | — | 5.91 |
| 3 | 59.8 | 63.9 | 54 | 31.4 | 2.95 | 4.74 | 2.25 | 11.2 | 15.6 | 17.5 | 2.13 | 16.8 | 15.5 | — | — | — |
| 4 | 65.8 | 58.4 | 62.1 | 36.9 | 3.02 | 4 | 1.87 | 10.6 | 7.25 | 11.4 | 11.4 | 11.7 | 7.64 | — | — | — |
| 5 | 87.6 | 65.7 | 42.6 | 49.5 | 2.64 | 6.05 | 2.17 | 9.99 | 1.4 | 5.92 | 12.9 | 13 | 9.64 | — | — | — |
| 6 | 193 | 87.5 | 76.2 | 36.7 | 2.45 | 2.66 | 1.6 | 9.26 | 88.8 | 84.5 | 1.99 | 90.7 | 26.7 | — | — | — |
| 7 | 191 | 192 | 55.7 | 120 | 2.62 | 4.94 | 4.29 | 12 | 23.7 | 19.1 | 8.37 | 24.1 | 52.9 | — | — | — |
| 8 | 2.52e+03 | 191 | 85.1 | 58.5 | 3.18 | 30.6 | 25.5 | 15 | 1.17e+03 | 925 | 3.16 | 1.17e+03 | 393 | — | — | 7.24 |
| 9 | 2.5e+03 | 2.52e+03 | 787 | 258 | 3.13 | 8.8 | 4.03 | 10.9 | 20.6 | 26.5 | 0.971 | 20.8 | 12.8 | — | — | — |
| 10 | 2.88e+03 | 2.5e+03 | 403 | 573 | 2.79 | 7.4 | 3.12 | 14.1 | 489 | 397 | 0.435 | 490 | 169 | — | — | — |
| 11 | 2.88e+03 | 2.88e+03 | 1.55e+03 | 1.79e+03 | 2.92 | 3.26 | 1.97 | 9.08 | 2.95 | 10.9 | 2.66 | 3.18 | 1.74 | — | — | — |
| 12 | 2.9e+03 | 2.88e+03 | 1.19e+03 | 480 | 4.15 | 8.63 | 3.53 | 16 | 7.01 | 8.72 | 0.843 | 7.05 | 4.84 | — | — | — |
| 13 | 2.96e+03 | 2.9e+03 | 1.91e+03 | 694 | 2.9 | 5.16 | 2.16 | 14.7 | 55.5 | 47.7 | 17.1 | 55.7 | 21.7 | — | — | — |
| 14 | 5.35e+03 | 2.96e+03 | 125 | 814 | 3.43 | 23.4 | 19.6 | 13.2 | 1.27e+03 | 976 | 21.4 | 1.27e+03 | 407 | 224 | 13.9 | 6.46 |
| 15 | 1.379e+04 | 5.35e+03 | 2.02e+03 | 5.44e+03 | 3.47 | 53.6 | 52 | 15 | 5.91e+03 | 5.27e+03 | 67.3 | 5.93e+03 | 1.71e+03 | — | — | — |
| 16 | 2.114e+04 | 1.379e+04 | 3.2e+03 | 1.99e+03 | 3.51 | 6.66 | 5.22 | 13 | 3.68e+03 | 2.69e+03 | 3.33 | 3.68e+03 | 1.07e+03 | — | — | — |
| 17 | 2.447e+04 | 2.113e+04 | 541 | 4.88e+03 | 4.48 | 3.32 | 2.37 | 18.8 | 1.69e+03 | 1.32e+03 | 1.36 | 1.69e+03 | 545 | — | — | — |
| 18 | 2.567e+04 | 2.447e+04 | 7.53e+03 | 1.323e+04 | 4.12 | 3.52 | 2.3 | 12.1 | 612 | 453 | 11.4 | 616 | 299 | — | — | — |
| 19 | 2.953e+04 | 2.567e+04 | 5.26e+03 | 6.67e+03 | 5.88 | 6.37 | 3.24 | 30 | 1.94e+03 | 1.6e+03 | 87.1 | 1.94e+03 | 534 | — | — | — |
| 20 | 5.377e+04 | 2.953e+04 | 1.323e+04 | 6.05e+03 | 4.92 | 5.04 | 3.7 | 14.3 | 1.906e+04 | 1.308e+04 | 12.1 | 1.907e+04 | 5.3e+03 | — | — | 5.32 |
| 21 | 6.943e+04 | 5.376e+04 | 2.11e+04 | 4.249e+04 | 4.12 | 34.9 | 18.2 | 13.6 | 1.348e+04 | 1.019e+04 | 71.8 | 1.355e+04 | 3.52e+03 | — | — | — |
| 22 | 1.034e+05 | 6.939e+04 | 1.145e+04 | 4.555e+04 | 6.06 | 4.58 | 3 | 14.4 | 1.699e+04 | 1.201e+04 | 29.9 | 1.702e+04 | 4.65e+03 | — | — | — |
| 23 | 1.444e+05 | 1.034e+05 | 3.117e+04 | 1.939e+04 | 4.39 | 4.61 | 5.34 | 11.4 | 2.041e+04 | 1.376e+04 | 163 | 2.057e+04 | 5.25e+03 | — | — | — |
| 24 | 1.823e+05 | 1.444e+05 | 5.286e+04 | 5.684e+04 | 4.2 | 6.36 | 4.2 | 11.4 | 1.89e+04 | 1.312e+04 | 134 | 1.903e+04 | 5.04e+03 | — | — | 5.27 |
| 25 | 2.161e+05 | 1.823e+05 | 5.066e+04 | 5.189e+04 | 4.13 | 5.81 | 3.56 | 13.4 | 1.667e+04 | 1.211e+04 | 197 | 1.687e+04 | 4.44e+03 | — | — | — |
| 26 | 2.402e+05 | 2.161e+05 | 4.09e+04 | 4.849e+04 | 3.86 | 8.91 | 5.24 | 16.8 | 1.163e+04 | 8.04e+03 | 457 | 1.205e+04 | 3.11e+03 | — | — | — |
| 27 | 2.564e+05 | 2.402e+05 | 1.487e+05 | 1.764e+04 | 4.84 | 5.87 | 4.29 | 13.5 | 1.141e+04 | 9.89e+03 | 254 | 1.166e+04 | 3.23e+03 | — | — | — |
| 28 | 3.021e+05 | 2.564e+05 | 1.001e+05 | 8.168e+04 | 3.83 | 13.8 | 7.6 | 14.5 | 2.243e+04 | 1.502e+04 | 745 | 2.283e+04 | 5.97e+03 | — | — | 4.99 |
| 29 | 3.313e+05 | 3.021e+05 | 3.942e+05 | 7.372e+04 | 6.17 | 8.57 | 7.3 | 10.9 | 1.429e+04 | 9.91e+03 | 649 | 1.464e+04 | 3.75e+03 | — | — | — |
| 30 | 3.349e+05 | 3.313e+05 | 4.617e+05 | 7.861e+04 | 5.79 | 11.4 | 8.37 | 12.9 | 3.13e+03 | 4.83e+03 | 456 | 3.14e+03 | 829 | — | — | — |
| 31 | 3.413e+05 | 3.349e+05 | 2.823e+05 | 9.561e+04 | 4.96 | 6.93 | 3.84 | 12.9 | 5.08e+03 | 3.82e+03 | 177 | 5.09e+03 | 1.34e+03 | — | — | — |
| 32 | 3.413e+05 | 3.411e+05 | 3.8e+05 | 1.32e+05 | 3.71 | 8.34 | 3.66 | 12.3 | 197 | 1.45e+03 | 250 | 351 | 119 | — | — | 5.68 |
| 33 | 3.451e+05 | 3.413e+05 | 1.676e+05 | 1.013e+05 | 5 | 27.2 | 11.2 | 13.6 | 6.25e+03 | 4.22e+03 | 761 | 7.01e+03 | 1.91e+03 | — | — | — |
| 34 | 3.466e+05 | 3.451e+05 | 3.232e+05 | 5.636e+04 | 4.13 | 22.8 | 10.5 | 14.9 | 2.61e+03 | 4.89e+03 | 228 | 2.63e+03 | 709 | — | — | — |
| 35 | 3.464e+05 | 3.466e+05 | 3.542e+05 | 5.99e+04 | 3.7 | 11.7 | 5.71 | 12.9 | 2.42e+03 | 2.52e+03 | 15.6 | 2.85e+03 | 758 | — | — | — |
| 36 | 3.625e+05 | 3.464e+05 | 2.127e+04 | 7.074e+04 | 5.13 | 32.7 | 21 | 39.9 | 1.066e+04 | 9.12e+03 | 919 | 1.158e+04 | 3.25e+03 | — | — | 5.86 |
| 37 | 3.607e+05 | 3.593e+05 | 1.142e+04 | 6.02e+04 | 5.34 | 10.4 | 10.5 | 18 | 1.191e+04 | 7.94e+03 | 1.33e+03 | 1.267e+04 | 4.42e+03 | — | — | — |
| 38 | 3.608e+05 | 3.599e+05 | 3.888e+05 | 3.061e+04 | 3.95 | 36.1 | 19 | 90.3 | 4.2e+03 | 4.91e+03 | 462 | 4.22e+03 | 1.26e+03 | — | — | — |
| 39 | 3.593e+05 | 3.601e+05 | 1.314e+05 | 4.15e+04 | 36 | 441 | 365 | 102 | 1.222e+04 | 8.15e+03 | 2.89e+03 | 1.219e+04 | 3.49e+03 | — | — | — |
Overall maxima (layer, prompt:token position):
stream: 3.625e+05 at layer 36 (p2_code:186)stream_attn: 3.601e+05 at layer 39 (p2_code:186)hc.attn_x: 4.617e+05 at layer 30 (p2_code:186)hc.ffn_x: 1.32e+05 at layer 32 (p2_code:1)attn.o: 36 at layer 39 (p2_code:77)attn.out: 441 at layer 39 (p3_doc:428)attn.out_tp4_partial: 365 at layer 39 (p3_doc:428)attn.qk_logit: 102 at layer 39 (p1_text:0)moe.expert_out_max: 2.243e+04 at layer 28 (p2_code:0)moe.expert_out_unweighted_max: 1.502e+04 at layer 28 (p2_code:0)moe.shared_out: 2.89e+03 at layer 39 (p2_code:88)moe.combine_f32: 2.283e+04 at layer 28 (p2_code:186)moe.out_tp4_partial: 5.97e+03 at layer 28 (p2_code:186)engram.wkv_out: 224 at layer 14 (p3_doc:726)engram.gate_add: 13.9 at layer 14 (p1_text:0)idx.score: 7.24 at layer 8 (p3_doc:1521)
v100-semantic: §4.4 rules
- Attention sublayer output (wo_b, all-reduced) max = 441 → FP16 allowed (rule < 8,192).
- MoE combine (routed + shared) max = 2.283e+04 → FP32 required (rule < 8,192).
- TP4 partial sums before the all-reduce: max = 5.97e+03 (attention and MoE) → below 8,192; the MoE sum exceeds it, so the MoE all-reduce output stays FP32 regardless.
- Per-expert w2 output: reference order (routing weight before w2) max = 2.243e+04; port order (weight after w2) max = 1.502e+04 → FP16 epilogue allowed by the rule (< 16,384), margin 8 %.
- Residual stream max = 3.625e+05: exceeds FP16 range (stream stays BF16, A4; FP32 variance mandatory).
v100-semantic+ownk: max |x| per layer (all prompts, all steps)
Columns: stream = stream, str_attn = stream_attn, hcx_attn = hc.attn_x, hcx_ffn = hc.ffn_x, attn_o = attn.o, attn_out = attn.out, attn_tp4 = attn.out_tp4_partial, qk = attn.qk_logit, exp_w = moe.expert_out_max, exp_unw = moe.expert_out_unweighted_max, shared = moe.shared_out, moe_comb = moe.combine_f32, moe_tp4 = moe.out_tp4_partial, eng_wkv = engram.wkv_out, eng_add = engram.gate_add, idx = idx.score
| layer | stream | str_attn | hcx_attn | hcx_ffn | attn_o | attn_out | attn_tp4 | qk | exp_w | exp_unw | shared | moe_comb | moe_tp4 | eng_wkv | eng_add | idx |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 9.21 | 1.78 | 0.617 | 1.77 | 3.25 | 8.46 | 4.18 | 17.5 | 9.83 | 13.1 | 6.64 | 10.2 | 8.28 | — | — | — |
| 1 | 76.9 | 9.17 | 5.01 | 2.88 | 3.65 | 9.18 | 3.58 | 18.4 | 43.2 | 36.6 | 1.66 | 43.3 | 19.9 | 103 | 1.46 | — |
| 2 | 68.2 | 71.9 | 36 | 55.8 | 2.27 | 5.54 | 2.51 | 11 | 17.9 | 31 | 2.3 | 19.9 | 14.2 | — | — | 5.91 |
| 3 | 59.8 | 63.9 | 54 | 31.4 | 2.95 | 4.74 | 2.12 | 11.2 | 15.2 | 17.5 | 2.13 | 16.5 | 15 | — | — | — |
| 4 | 65.8 | 56.2 | 62.1 | 36.9 | 3.02 | 4 | 1.8 | 10.2 | 1.55 | 2.16 | 11.4 | 11.7 | 7.64 | — | — | — |
| 5 | 87.6 | 65.7 | 34.3 | 49.5 | 2.64 | 6.04 | 2.17 | 9.99 | 0.929 | 2.12 | 12.9 | 13 | 9.64 | — | — | — |
| 6 | 87.5 | 87.5 | 76.2 | 12.2 | 2.45 | 2.66 | 1.6 | 9.26 | 1.36 | 2.62 | 1.99 | 1.77 | 1.94 | — | — | — |
| 7 | 87.5 | 87.5 | 55.7 | 120 | 2.51 | 4.94 | 4.29 | 12 | 23.7 | 18.7 | 8.37 | 24.1 | 52.9 | — | — | — |
| 8 | 290 | 87.4 | 85.1 | 14.4 | 3.18 | 30.6 | 25.5 | 15 | 123 | 113 | 3.12 | 124 | 66.8 | — | — | 7.24 |
| 9 | 290 | 290 | 561 | 258 | 3.13 | 8.8 | 4.03 | 11.2 | 0.774 | 2.1 | 0.971 | 1.42 | 0.708 | — | — | — |
| 10 | 1.27e+03 | 290 | 403 | 218 | 2.79 | 7.4 | 3.12 | 14.1 | 489 | 396 | 0.435 | 489 | 169 | — | — | — |
| 11 | 1.27e+03 | 1.27e+03 | 1.55e+03 | 1.19e+03 | 2.92 | 3.26 | 1.89 | 9.08 | 0.961 | 1.87 | 2.66 | 3.18 | 1.03 | — | — | — |
| 12 | 1.27e+03 | 1.27e+03 | 1.19e+03 | 385 | 4.15 | 8.45 | 3.27 | 16 | 2.08 | 3.75 | 0.843 | 2.2 | 1.13 | — | — | — |
| 13 | 1.27e+03 | 1.27e+03 | 1.83e+03 | 320 | 2.9 | 5.16 | 2.16 | 13 | 10.8 | 23 | 17.1 | 16.7 | 15.6 | — | — | — |
| 14 | 1.58e+03 | 1.28e+03 | 84.8 | 814 | 3.43 | 22.5 | 17.2 | 13.7 | 166 | 132 | 21.4 | 166 | 61.6 | 224 | 13.9 | 6.46 |
| 15 | 1.336e+04 | 1.58e+03 | 153 | 1.91e+03 | 3.35 | 35.5 | 33.3 | 13.5 | 5.9e+03 | 4.08e+03 | 43.3 | 5.91e+03 | 1.7e+03 | — | — | — |
| 16 | 1.43e+04 | 1.335e+04 | 425 | 968 | 3.51 | 6.66 | 4.75 | 14 | 475 | 345 | 1.55 | 475 | 260 | — | — | — |
| 17 | 1.463e+04 | 1.429e+04 | 443 | 1.11e+03 | 4.48 | 3.32 | 2.37 | 18.8 | 167 | 134 | 1.36 | 167 | 50.9 | — | — | — |
| 18 | 1.511e+04 | 1.463e+04 | 702 | 1.52e+03 | 4.12 | 3.52 | 2.3 | 12.1 | 231 | 170 | 11.4 | 245 | 76.9 | — | — | — |
| 19 | 1.553e+04 | 1.511e+04 | 5.26e+03 | 3.64e+03 | 5.88 | 5.17 | 2.83 | 30 | 124 | 111 | 87.1 | 212 | 74.9 | — | — | — |
| 20 | 5.313e+04 | 1.554e+04 | 8.46e+03 | 5.45e+03 | 4.92 | 5.04 | 3.7 | 14.3 | 1.881e+04 | 1.299e+04 | 11.7 | 1.882e+04 | 5.23e+03 | — | — | 5.32 |
| 21 | 6.896e+04 | 5.311e+04 | 1.321e+04 | 4.205e+04 | 4.08 | 34.9 | 18.2 | 13.6 | 1.346e+04 | 1.018e+04 | 71.4 | 1.353e+04 | 3.51e+03 | — | — | — |
| 22 | 1.028e+05 | 6.892e+04 | 1.141e+04 | 4.49e+04 | 4.74 | 4.58 | 2.79 | 14.4 | 1.696e+04 | 1.199e+04 | 29.7 | 1.699e+04 | 4.64e+03 | — | — | — |
| 23 | 1.438e+05 | 1.028e+05 | 2.045e+04 | 1.898e+04 | 4.39 | 4.61 | 5.34 | 11.4 | 2.04e+04 | 1.375e+04 | 162 | 2.056e+04 | 5.24e+03 | — | — | — |
| 24 | 1.812e+05 | 1.438e+05 | 2.916e+04 | 5.622e+04 | 4.19 | 4.44 | 4.06 | 11.4 | 1.862e+04 | 1.306e+04 | 134 | 1.877e+04 | 4.97e+03 | — | — | 5.19 |
| 25 | 2.149e+05 | 1.812e+05 | 2.526e+04 | 5.126e+04 | 4.13 | 5.33 | 3.56 | 13.4 | 1.662e+04 | 1.207e+04 | 196 | 1.682e+04 | 4.43e+03 | — | — | — |
| 26 | 2.389e+05 | 2.149e+05 | 4.042e+04 | 4.798e+04 | 3.86 | 8.91 | 5.24 | 16.8 | 1.161e+04 | 8.02e+03 | 455 | 1.204e+04 | 3.1e+03 | — | — | — |
| 27 | 2.544e+05 | 2.389e+05 | 9.961e+04 | 1.752e+04 | 4.84 | 5.87 | 4.29 | 13.5 | 7.56e+03 | 9.87e+03 | 180 | 7.74e+03 | 1.97e+03 | — | — | — |
| 28 | 3e+05 | 2.544e+05 | 3.343e+04 | 5.625e+04 | 3.83 | 9.88 | 7.6 | 14.5 | 2.243e+04 | 1.502e+04 | 397 | 2.283e+04 | 5.97e+03 | — | — | 4.99 |
| 29 | 3.293e+05 | 3e+05 | 3.914e+05 | 7.048e+04 | 6.17 | 8.57 | 7.3 | 10.9 | 1.429e+04 | 9.91e+03 | 354 | 1.464e+04 | 3.75e+03 | — | — | — |
| 30 | 3.329e+05 | 3.293e+05 | 4.587e+05 | 7.861e+04 | 5.79 | 9.88 | 7.9 | 11.7 | 3.13e+03 | 3.75e+03 | 165 | 3.14e+03 | 829 | — | — | — |
| 31 | 3.393e+05 | 3.329e+05 | 2.808e+05 | 9.553e+04 | 4.75 | 6.93 | 3.84 | 12.9 | 5.07e+03 | 3.58e+03 | 68.1 | 5.08e+03 | 1.34e+03 | — | — | — |
| 32 | 3.393e+05 | 3.391e+05 | 3.774e+05 | 5.748e+04 | 3.71 | 7.96 | 3.66 | 12.3 | 5.08 | 21.9 | 85.2 | 88.9 | 29 | — | — | 5.68 |
| 33 | 3.431e+05 | 3.393e+05 | 9.866e+04 | 2.983e+04 | 4.68 | 14.4 | 5.76 | 13.6 | 4.13e+03 | 4.21e+03 | 285 | 4.15e+03 | 1.09e+03 | — | — | — |
| 34 | 3.446e+05 | 3.431e+05 | 3.21e+05 | 1.59e+04 | 4.02 | 16.5 | 7.98 | 14.9 | 2.6e+03 | 2.05e+03 | 81.9 | 2.62e+03 | 708 | — | — | — |
| 35 | 3.443e+05 | 3.446e+05 | 3.526e+05 | 5.98e+04 | 3.7 | 8.94 | 3.89 | 12.9 | 2.41e+03 | 2.52e+03 | 5.85 | 2.84e+03 | 756 | — | — | — |
| 36 | 3.604e+05 | 3.443e+05 | 2.121e+04 | 7.074e+04 | 5.13 | 32.7 | 20 | 39.7 | 8.75e+03 | 9.12e+03 | 678 | 9.58e+03 | 2.81e+03 | — | — | 5.62 |
| 37 | 3.586e+05 | 3.573e+05 | 3.74e+03 | 6.02e+04 | 5.34 | 10.4 | 10.5 | 18 | 1.191e+04 | 7.94e+03 | 884 | 1.267e+04 | 4.42e+03 | — | — | — |
| 38 | 3.588e+05 | 3.579e+05 | 3.866e+05 | 3.059e+04 | 3.56 | 33.8 | 17.1 | 64 | 4.2e+03 | 4.91e+03 | 382 | 4.21e+03 | 1.26e+03 | — | — | — |
| 39 | 3.572e+05 | 3.581e+05 | 7.989e+04 | 4.15e+04 | 28 | 441 | 365 | 95.5 | 1.222e+04 | 8.15e+03 | 2.78e+03 | 1.219e+04 | 3.49e+03 | — | — | — |
Overall maxima (layer, prompt:token position):
stream: 3.604e+05 at layer 36 (p3_doc:0)stream_attn: 3.581e+05 at layer 39 (p3_doc:0)hc.attn_x: 4.587e+05 at layer 30 (p3_doc:0)hc.ffn_x: 9.553e+04 at layer 31 (p3_doc:0)attn.o: 28 at layer 39 (p3_doc:457)attn.out: 441 at layer 39 (p3_doc:428)attn.out_tp4_partial: 365 at layer 39 (p3_doc:428)attn.qk_logit: 95.5 at layer 39 (p3_doc:0)moe.expert_out_max: 2.243e+04 at layer 28 (p3_doc:0)moe.expert_out_unweighted_max: 1.502e+04 at layer 28 (p3_doc:0)moe.shared_out: 2.78e+03 at layer 39 (p3_doc:1772)moe.combine_f32: 2.283e+04 at layer 28 (p3_doc:0)moe.out_tp4_partial: 5.97e+03 at layer 28 (p3_doc:0)engram.wkv_out: 224 at layer 14 (p3_doc:726)engram.gate_add: 13.9 at layer 14 (p3_doc:0)idx.score: 7.24 at layer 8 (p3_doc:1521)
v100-semantic+ownk: §4.4 rules
- Attention sublayer output (wo_b, all-reduced) max = 441 → FP16 allowed (rule < 8,192).
- MoE combine (routed + shared) max = 2.283e+04 → FP32 required (rule < 8,192).
- TP4 partial sums before the all-reduce: max = 5.97e+03 (attention and MoE) → below 8,192; the MoE sum exceeds it, so the MoE all-reduce output stays FP32 regardless.
- Per-expert w2 output: reference order (routing weight before w2) max = 2.243e+04; port order (weight after w2) max = 1.502e+04 → FP16 epilogue allowed by the rule (< 16,384), margin 8 %.
- Residual stream max = 3.604e+05: exceeds FP16 range (stream stays BF16, A4; FP32 variance mandatory).
Raw: [...]/range_audit.json; per-run stats: [private path].