Skip to the content.

Origin. PLI Labs research archive, file v100-research/deepseek-v4.1-flash/TENSOR_CENSUS.md. Published 2026-11-18 in the Volta V100 research repository. Edited for publication. Machine names, private file paths, network addresses, session names and the operator’s name were replaced with generic terms (the V100 server, [private path], [address], the owner). Links to files that are not published were turned into plain text. No figure and no finding was changed. Dates inside the note are its own; later results are in the build pages of this repository.

Lane L-MODEL, 2026-10-06. Source: deepseek-ai/DeepSeek-V4.1-Flash @ 2cba9e42aa026125f3ed06c6d98c1db82f7ca027. Machine-readable: tensor_census.json (aggregates, per-layer, stage splits, every non-routed-expert tensor) and tensor_census_full.tsv.gz (all 96,085 tensors: name, dtype, shape, bytes, shard, class, layer, params).

Method

For each of the 48 shards: HTTP range read bytes=0-7 (little-endian u64 header length), then bytes=8-(8+len-1) (header JSON), from https://huggingface.co/<repo>/resolve/<rev>/<shard>. Every tensor’s data_offsets span was checked against numel × dtype size (no mismatch), and 8 + header_len + data_end was checked against the HF LFS file size for every shard: 48/48 match. Scripts: [...]/{fetch_headers.sh,census.py,census_report.py}; raw headers in [private path].

Quantity Value
Tensors 96,085
Tensor bytes (sum of data spans) 510,286,023,000 B = 475.241 GiB (= index total_size, exact)
File bytes (incl. 48 headers) 510,296,708,312 B
Largest shard model-00047 = layer-1 Engram (101,535,150,272 data bytes in one file); model-00048 = layer-14 Engram

Dtype codes in the headers

Header dtype Count Meaning Used for
I8 47,232 packed FP4 E2M1, two values per byte, packed along K (in-features). Low nibble = even K index, high nibble = odd (decode table in inference/convert.py:13-15,31-34); reference views it as torch.float4_e2m1fn_x2 (convert.py:181) routed experts only (backbone + MTP)
F8_E8M0 47,589 UE8M0 power-of-two scale (torch.float8_e8m0fnu) every scale: experts (1 per 32 along K), FP8 dense (1 per 32×32 block), Engram rows (1 per 32 channels)
F8_E4M3 357 FP8 E4M3FN dense FP8 linears, shared experts, Engram tables, Engram wkv, DSpark main_proj
BF16 520 bf16 embed, head, norms, gate.weight, compressor wkv/wgate, indexer wk/weights_proj, engram q/k_weight, markov head, vision
F32 387 fp32 hc_*, attn_sink, gate.bias, gate.bias_vl

There is no F4/U8/F8_E5M2 dtype in the official checkpoint. Quantization config (config.json): quant_method=fp8, weight_block_size=[32,32], scale_fmt=ue8m0, activation_scheme=dynamic, expert_dtype=fp4.

Bytes by class (official checkpoint)

Group : class Bytes GiB Tensors Params dtypes
backbone : routed_experts.weight (FP4) 271,790,899,200 253.1250 46,080 543,581,798,400 I8
backbone : routed_experts.scale 16,986,931,200 15.8203 46,080 — E8M0
backbone routed experts total 288,777,830,400 268.9453      
backbone : engram.embed.weight 196,613,849,600 183.1109 2 196,613,849,600 E4M3
backbone : engram.embed.scale 6,144,182,800 5.7222 2 — E8M0
Engram tables total 202,758,032,400 188.8331      
backbone : engram.wkv (+scale) 314,880,000 0.2933 4 314,572,800 E4M3+E8M0
backbone : engram.q_weight/k_weight 163,840 0.0002 4 81,920 BF16
backbone : attn.wq_a (+scale) 262,400,000 0.2444 80 262,144,000 E4M3+E8M0
backbone : attn.wq_b (+scale) 1,679,360,000 1.5640 80 1,677,721,600 E4M3+E8M0
backbone : attn.wkv (+scale) 104,960,000 0.0978 80 104,857,600 E4M3+E8M0
backbone : attn.wo_a (+scale) 1,343,488,000 1.2512 80 1,342,177,280 E4M3+E8M0
backbone : attn.wo_b (+scale) 1,679,360,000 1.5640 80 1,677,721,600 E4M3+E8M0
backbone : attn q_norm/kv_norm/attn_sink 153,600 0.0001 120 74,240 BF16/F32
backbone : attn.compressor (wkv, wgate, norm) 36,704,256 0.0342 11 18,352,128 BF16
backbone : attn.indexer (wq_b+scale, weights_proj, wk, k_norm) 45,130,752 0.0420 32 43,516,416 E4M3/E8M0/BF16
backbone : shared_expert.weight 1,415,577,600 1.3184 120 1,415,577,600 E4M3
backbone : shared_expert.scale 1,382,400 0.0013 120 — E8M0
backbone : gate.weight 157,286,400 0.1465 40 78,643,200 BF16
backbone : gate.bias + gate.bias_vl 122,880 0.0001 80 30,720 F32
backbone : hc_* 157,295,040 0.1465 240 39,323,760 F32
backbone : attn_norm/ffn_norm 819,200 0.0008 80 409,600 BF16
embed.weight 1,323,827,200 1.2329 1 661,913,600 BF16
head.weight 1,323,827,200 1.2329 1 661,913,600 BF16
norm.weight 10,240 0.0000 1 5,120 BF16
mtp.* (DSpark, 3 blocks) total 7,932,874,632 7.3881   14,225,362,530  
— of which mtp routed experts (128/blk, weight+scale) 7,219,445,760 6.7236 2,304 13,589,544,960 I8+E8M0
— of which mtp dense (attn, shared, gate, hc, main_proj, markov, confidence) 713,428,872 0.6644      
vision.* + aligner.* + image_{start,end,newline} 970,536,960 0.9039 266 485,268,480 BF16

Group totals: backbone 498,734,946,768 B (464.483 GiB) · embed/head/norm 2,647,664,640 B (2.466 GiB) · mtp 7,932,874,632 B (7.388 GiB) · vision 970,536,960 B (0.904 GiB).

Community census check (JigSawPT discussion figures)

Claim Header-exact Verdict
routed experts MXFP4 268.9 GiB 288,777,830,400 B = 268.945 GiB (backbone; +6.724 GiB more in MTP) verified
Engram tables 188.8 GiB 202,758,032,400 B = 188.833 GiB verified
“rest” 9.6 GiB backbone non-expert non-table + embed/head/norm = 9.171 GiB (9,847,742,808 B); +MTP-dense 0.664 → 9.835; +vision 0.904 approximately; partition differs (their 9.6 is not reproducible to the byte from any natural partition; use 9.171 GiB)

Parameter counts (header-derived; FP4 counted as 2 params per I8 byte)

Quantity Params Note
All tensors 763,205,315,794  
“Backbone” (all text weights excl. Engram tables, MTP, vision; incl. embed+head) 551,880,835,184 = README/report “552B backbone”
Engram tables 196,613,849,600 = report “196B Engram parameters”
Routed experts (backbone) 543,581,798,400 35,389,440 per expert × 384 × 40
DSpark/MTP 14,225,362,530  
Vision + aligner 485,268,480  
Active per decoded token (6/384 routed × 40 + all dense of 40 layers + head + 48 Engram rows) ≈ 16.13 B report: 16B
Active per prefill token (encoder only, layers 0–19) ≈ 7.89 B report: 8B (CED: decoder runs only on the last n_win=128 tokens)

Per-tensor shapes (one layer; all 40 layers identical except where noted)

Tensor dtype shape bytes
attn.wq_a / .scale E4M3 / E8M0 [1280, 5120] / [40, 160] 6,553,600 / 6,400
attn.wq_b / .scale E4M3 / E8M0 [32768, 1280] / [1024, 40] 41,943,040 / 40,960
attn.wkv / .scale E4M3 / E8M0 [512, 5120] / [16, 160] 2,621,440 / 2,560
attn.wo_a / .scale E4M3 / E8M0 [8192, 4096] / [256, 128] 33,554,432 / 32,768
attn.wo_b / .scale E4M3 / E8M0 [5120, 8192] / [160, 256] 41,943,040 / 40,960
attn.q_norm / kv_norm / attn_sink BF16 / BF16 / F32 [1280] / [512] / [64] 2,560 / 1,024 / 256
attn.compressor.wkv (layers 2,8,14,20) BF16 [512, 5120] 5,242,880
attn.compressor.wgate (layers 2,8,14 only; ratio-1 layer 20 has none) BF16 [512, 5120] 5,242,880
attn.compressor.norm (2,8,14,20) BF16 [512] 1,024
attn.indexer.wq_b / .scale (2,8,14,20,24,28,32,36) E4M3 / E8M0 [4096, 1280] / [128, 40] 5,242,880 / 5,120
attn.indexer.weights_proj (same 8 layers) BF16 [32, 5120] 327,680
attn.indexer.wk / k_norm (2,8,14,20 only) BF16 [128, 512] / [128] 131,072 / 256
ffn.gate.weight / bias / bias_vl BF16 / F32 / F32 [384, 5120] / [384] / [384] 3,932,160 / 1,536 / 1,536
ffn.shared_experts.w1,w3 / .scale E4M3 / E8M0 [2304, 5120] / [72, 160] 11,796,480 / 11,520 each
ffn.shared_experts.w2 / .scale E4M3 / E8M0 [5120, 2304] / [160, 72] 11,796,480 / 11,520
ffn.experts.{0..383}.w1,w3 / .scale I8(FP4×2) / E8M0 [2304, 2560] / [2304, 160] 5,898,240 / 368,640 each
ffn.experts.{0..383}.w2 / .scale I8(FP4×2) / E8M0 [5120, 1152] / [5120, 72] 5,898,240 / 368,640
hc_{attn,ffn}_fn / _base / _scale F32 [24, 20480] / [24] / [3] 1,966,080 / 96 / 12 each
attn_norm, ffn_norm BF16 [5120] 10,240 each
engram.embed.weight (L1 / L14) E4M3 [384006168, 256] / [384016682, 256] 98,305,579,008 / 98,308,270,592
engram.embed.scale (L1 / L14) E8M0 [384006168, 8] / [384016682, 8] 3,072,049,344 / 3,072,133,456
engram.wkv / .scale E4M3 / E8M0 [25600, 6144] / [800, 192] 157,286,400 / 153,600
engram.q_weight, k_weight BF16 [4, 5120] 40,960 each
embed.weight, head.weight BF16 [129280, 5120] 1,323,827,200 each
mtp.{0,1,2} attn (wq_a, wq_b, wkv, wo_a, wo_b, norms, sink) as backbone as backbone 126,758,656 each (no compressor/indexer)
mtp.{0,1,2}.ffn.experts.{0..127}.* I8 / E8M0 as backbone 2,406,481,920 per block
mtp.{i}.ffn.gate.weight / bias / bias_vl BF16 / F32 [128, 5120] / [128] 1,310,720 / 512
mtp.0.main_proj / .scale, mtp.0.main_norm E4M3 / E8M0, BF16 [5120, 15360] / [160, 480], [5120] 78,643,200 / 76,800, 10,240
mtp.2.markov_head.embed / .head BF16 [129280, 256] each 66,191,360 each
mtp.2.confidence_head.proj BF16 [1, 5376] 10,752
mtp.2.norm BF16 [5120] 10,240

MTP has no embed.weight/head.weight (tied to the backbone’s, convert.py:108-110, model.py:1212-1213).

Bits per weight: routed experts = (17,694,720 + 1,105,920) B × 8 / 35,389,440 = 4.25 bpw (MXFP4: E2M1 + one E8M0 per 32). Dense FP8 = 8 + 8/1024 = 8.008 bpw. Engram rows = (256 + 8) B / 256 = 8.25 bpw.

Per-layer bytes (checkpoint-native; “GPU” = everything except the Engram tables)

dense_fp16 = the same layer if every F8_E4M3 tensor were expanded to FP16 (scales dropped); BF16 counted as FP16, F32 kept. Experts stay MXFP4 in both columns.

Layer Attention mode Routed experts B Dense B GPU native B GiB GPU w/ dense FP16 B Engram table B (host)
0 SWA only 7,219,445,760 170,055,128 7,389,500,888 6.882 7,551,347,672 0
1 SWA only +Engram 7,219,445,760 327,577,048 7,547,022,808 7.029 7,866,002,392 101,377,628,352
2 CSA2(r=2,Full) 7,219,445,760 186,248,920 7,405,694,680 6.897 7,572,779,224 0
3–7 CSA2(r=2,Reuse) 7,219,445,760 170,055,128 7,389,500,888 6.882 7,551,347,672 0
8 CSA2(r=2,Full) 7,219,445,760 186,248,920 7,405,694,680 6.897 7,572,779,224 0
9–13 CSA2(r=2,Reuse) 7,219,445,760 170,055,128 7,389,500,888 6.882 7,551,347,672 0
14 CSA2(r=2,Full) +Engram 7,219,445,760 343,770,840 7,563,216,600 7.044 7,887,433,944 101,380,404,048
15–19 CSA2(r=2,Reuse) 7,219,445,760 170,055,128 7,389,500,888 6.882 7,551,347,672 0
20 CSA2(r=1,Full) + candidate source 7,219,445,760 181,006,040 7,400,451,800 6.892 7,567,536,344 0
21–23, 25–27, 29–31, 33–35, 37–39 CSA2(r=1,Reuse) 7,219,445,760 170,055,128 7,389,500,888 6.882 7,551,347,672 0
24, 28, 32, 36 CSA2(r=1,Reindex) 7,219,445,760 175,630,808 7,395,076,568 6.887 7,562,161,112 0
40 (mtp.0) DSpark SWA + main_proj 2,406,481,920 246,161,880 2,652,643,800 2.470 2,893,056,984 0
41 (mtp.1) DSpark SWA 2,406,481,920 167,431,640 2,573,913,560 2.397 2,735,760,344 0
42 (mtp.2) DSpark SWA + Markov/confidence heads 2,406,481,920 299,835,352 2,706,317,272 2.520 2,868,164,056 0

(Per-layer rows for every layer individually are in tensor_census.json → derived.per_layer.)

Aggregate Bytes GiB
Backbone GPU-resident, native (layers 0–39 excl. Engram tables + embed + head + final norm) 298,624,579,008 278.116
same with all FP8 dense expanded to FP16 305,454,618,048 284.477
MTP / DSpark (3 blocks, native; embed+head shared with backbone) 7,932,874,632 7.388
Engram tables (host-resident candidate) 202,758,032,400 188.833
Vision + aligner (text-only bring-up: load-skip) 970,536,960 0.904

The reference convert.py:157-173 dequantizes wo_a to BF16 (+33.5 MB per layer, +1.25 GiB total); the numbers above are checkpoint-native.

Pipeline-stage splits (native bytes, weights only — no KV, workspace, CUDA context)

Embed (1.233 GiB) on stage 0, head + final norm (1.233 GiB) on the last stage. TP4 per-GPU = stage / 4 (experts split evenly by EP or TP; Engram tables excluded, MTP excluded).

Layout Stage layers Stage GiB per GPU (TP4) GiB with FP16 dense, per GPU GiB
8 GPU PP2×TP4 0–19 139.21 34.80 35.63
  20–39 138.90 34.73 35.49
12 GPU PP3×TP4 (13/14/13) 0–12 90.88 22.72 23.25
  13–26 96.53 24.13 24.70
  27–39 90.71 22.68 23.17
12 GPU PP3×TP4 (14/13/13) 0–13 97.76 24.44 25.01
  14–26 89.64 22.41 22.94
  27–39 90.71 22.68 23.17
12 GPU, (13/13/14) 0–12 / 13–25 / 26–39 90.88 / 89.64 / 97.60 22.72 / 22.41 / 24.40  

Consequences (arithmetic, not measurement):

TP/kernel alignment facts for the port