Skip to the content.

Origin. PLI Labs research archive, file v100-research/findings/V-C-speedups-untried.md. Published 2026-11-18 in the Volta V100 research repository. Edited for publication. Machine names, private file paths, network addresses, session names and the operator’s name were replaced with generic terms (the V100 server, [private path], [address], the owner). Links to files that are not published were turned into plain text. No figure and no finding was changed. Dates inside the note are its own; later results are in the build pages of this repository.

Scope: every concrete way to make inference faster on a multi-board Tesla V100-SXM2-32GB server (sm_70 / Volta), with emphasis on options NOT YET TRIED on this server. Decode-tok/s focused (the target workload is single-stream, max_num_seqs=1, long context). Research-first and adversarial: marketing tok/s are distrusted, and every technique that silently needs sm_80+ is flagged as a non-starter on Volta.

Method / provenance. Findings combine (a) on-box inspection of the installed 1Cat-vLLM-1.0.0 conda env and a private benchmark setup (ground truth), (b) two parallel deep-web research lanes (spec-decode/quant; attention/MoE/serving), and (c) corroboration from internal notes. Per the eval-provenance rule, no number here is decision-grade until re-measured on our own box with /v1/models verified — most public numbers are A100/H100 (sm_80+) and are labelled (NOT V100). Confidence is per claim (high / med / low).


0. The two facts that reorganize the whole question

  1. The model architecture matters before the GPU. Qwen3.5 / Qwen3.6 (the A3B-class targets) are hybrid Gated-DeltaNet (linear-attention) + gated-attention MoE models (Qwen3-Next lineage; 3.6 mixes the two ~4:1). That recurrent SSM state changes which speculative-decoding methods are even correct, independent of Volta. The 122B-A10B target is conventional full-attention and behaves differently. So “what speeds up decode” is driven first by model family, second by sm_70. (Confidence: high — corroborated on-box by qwen3_5_mtp/qwen3_next_mtp model types in the fork; see §1.)

  2. Our workload is the best case for latency tricks and the worst case for throughput marketing. At max_num_seqs=1, decode is pure memory-bandwidth-bound weight streaming with the GPU under-utilized. Spec-decode multipliers are larger at batch=1 than the batched QPS numbers vendors quote; conversely CUDA-graph and kernel-launch overhead is a larger fraction of step time at batch=1, so removing it helps more here than the headline suggests. (Confidence: high — mechanistic.)


1. Speculative / assisted decoding on V100

On-box reality (verified): the installed fork ships the full vLLM v1 spec-decode suite — vllm/v1/spec_decode/{ngram_proposer,suffix_decoding,eagle,medusa,draft_model,dflash}.py — and config/speculative.py registers ngram, suffix, eagle/eagle3, and the MTP family including qwen3_5_mtp and qwen3_next_mtp. There is no sm_70 guard anywhere in the spec-decode path; the only capability checks are for multimodal/M-RoPE, not Volta. So the machinery is present and arch-agnostic on the proposer side — the question is which method is correct and fast for our models.

MTP (built-in multi-token-prediction heads) — the winner for this server

n-gram / prompt-lookup (method: ngram)

Suffix decoding (method: suffix, Arctic / SuffixDecoding)

Draft-model speculative decoding (small Qwen draft + big Qwen target)

EAGLE / EAGLE-3 / EAGLE-3.1

Medusa

DFlash (block-diffusion draft, Qwen3.5) — watch, don’t deploy

Ranking — single-stream (max_num_seqs=1) document drafting

Rank Method sm_70? Safe on 3.5/3.6-GDN? Expected decode × (batch=1, V100) Conf
1 MTP (built-in head) Yes — fork-validated Yes (decoupled) ~1.7–2.2× high
2 ngram (prompt-lookup) Yes (kernel-free) No — ✅ 122B full-attn only ~1.5–2.5× on repetitive text med
3 suffix (Arctic) Yes (CPU proposer) Likely unsafe — ✅ 122B only ~1.5–3× on repetitive med
4 EAGLE3 Partial (chain not tree) No head for our models ~1.4–1.9× (Qwen3 proxy) med
5 draft-model Yes (force non-FlashInfer) n/a ~1.2–2× (weak for MoE) low-med
— Medusa / DFlash Yes / on-box superseded / experimental — —

Bottom line: MTP is the default for the A3B-class targets (only method that is both Volta-validated and GDN-correct, weights are free in the checkpoint). For the 122B Qwen3 full-attention target, ngram/suffix become viable and may beat MTP on heavily-quoted boilerplate — A/B them. Start MTP-1; try 2–3 if acceptance holds.


2. Quantization for SPEED on Volta (decode tok/s, not just fit)

On-box reality (verified): the fork’s AWQ path is TurboMind-derived, not Marlin. The warmup module model_executor/warmup/awq_sm70_warmup.py literally “primes the TurboMind small-shape autotuner,” and there are dedicated sm_70 ops: awq_sm70_moe.py, a single-token compact MoE op (awq_moe_single_token_sm70_out, gated by VLLM_SM70_AWQ_ENABLE_SINGLE_TOKEN_COMPACT), and a persistent GEMM-shape LUT cache (VLLM_SM70_GEMM_LUT_PATH, sm70_gemm_import_cache/export_cache). This is real Volta-specific speed engineering — and it means if a log/flag says “marlin,” that is a naming/repack artifact; the kernel executing on Volta is the TurboMind sm_70 path.

Marlin / GPTQ-Marlin — DO NOT run on Volta

INT8 W8A8 — no Volta tensor-core path, unlikely to win

FP8 — fit/KV only, not a faster decode-compute path

Winner = AWQ-4bit (TurboMind sm_70)

KV-cache quant for speed/quality (llama.cpp lane)

Quant bottom line: AWQ-4bit (TurboMind sm_70) is the best single-stream decode quant on Volta. Marlin/GPTQ-Marlin are sm_80-locked, INT8 W8A8 has no Volta TC path, FP8 helps fit not speed. The biggest additional decode win on top of AWQ is stacking MTP (§1) — quant shrinks bytes/token, MTP cuts sequential steps; they multiply.


3. Attention kernels on Volta

On-box reality (verified): flash_attn_v100_cuda.cpython-312-x86_64-linux-gnu.so is a real compiled CUDA extension, loaded by vllm/v1/attention/backends/flash_attn_v100.py, which provides paged decode + dense/paged prefill Volta kernels with a strict Triton/eager fallback policy (TritonAttentionImpl base). Selecting it also flips a tuned max_num_batched_tokens default on sm_70 (arg_utils.py: cap (7,0) + ≥30 GiB + VLLM_ATTENTION_BACKEND=FLASH_ATTN_V100). The launch scripts already use --attention-backend FLASH_ATTN_V100 + VLLM_SM70_ENABLE_LM_HEAD_FASTPATH=1. So on the vLLM lane we already have a purpose-built Volta FA — the untried work is on the llama.cpp lane and in not regressing into the broken Triton path.

llama.cpp -fa — the most important “it actually works now” finding

What to AVOID on the vLLM lane (silent slow / hangs)

Ranking — what actually accelerates attention on sm_70: (1) 1Cat-vLLM FLASH_ATTN_V100 (already in use); (2) llama.cpp -fa HEAD build (prefill win); (3) XFORMERS/TORCH_SDPA as safe vLLM baseline; (✗) TRITON_ATTN / stock-Triton FA / FA2 / FlashInfer — avoid.


4. MoE-specific speedups

On-box reality (verified): the fork has sm_70 MoE quant kernels (awq_sm70_moe.py, single-token compact MoE op). The untried levers are a V100 fused-MoE tuning JSON, the TP-vs-EP / island placement decision, and CPU expert-offload to serve bigger MoEs than HBM allows.

The single biggest free MoE win: a V100 fused-MoE tuning JSON

Expert parallelism (EP) — do NOT enable on this server

CPU/RAM offload of cold experts — the lever to serve a bigger MoE

Synthesized offload decode bands (V100 + fast EPYC DDR; derived from math + nearest proxies, NO direct V100 data — re-measure): ~120B-A5B (gpt-oss) ~12–22 t/s (DDR5) / ~6–10 (DDR4); GLM-4.5-Air ~A12B ~8–12 / ~6–10; Qwen3-235B-A22B ~9–14 / ~5–8; DeepSeek 671B-A37B ~9–15 / ~5–8. Confidence: med.


5. Serving-level tuning (verify Volta)

Framing: stock vLLM dropped sm_70 build targets at ~0.20 (we use the 1Cat fork). The V1 engine (default ≥0.8.0) changes defaults for chunked prefill, prefix caching, max-num-seqs, and CUDA-graph mode — confirm whether the fork runs V1 or V0 (VLLM_USE_V1); it determines half the answers below.

CUDA graphs on Volta — we are probably leaving 10–28% on the table

--max-num-seqs — set LOW on single-stream lanes

Chunked prefill

Prefix caching — high ROI for repeated long prompts

NCCL / topology

Multi-LoRA — the old wall is gone, but unverified on V100


6. The “haven’t-been-tried” experiment list for THIS server — ranked by upside ÷ effort

Anchors today (verified on-box, but note provenance): the 1cat-bench numbers (35B-A3B-AWQ 121 tok/s, 122B-A10B-AWQ 77, 27B 81) were captured on another machine, TP2 (launch_27b/35b.sh → CUDA_VISIBLE_DEVICES=1,2, --tensor-parallel-size 2), not a 4-card board of the V100 server. The task brief calls these “the V100 server TP4” anchors. Action E0 below: re-measure on an NVLink quad of the V100 server with /v1/models verified to settle the discrepancy before attributing any speedup.

# Experiment Lane Expected upside (batch=1 decode) Effort Conf Risk / gate
E1 MTP-1 spec-decode on 35B-A3B-AWQ (then 122B): --speculative-config '{"method":"mtp","num_speculative_tokens":1}', sweep 1→3 vLLM (1Cat) ~1.7–2.2× Low high watch acceptance; GDN-safe
E2 Drop --enforce-eager, PIECEWISE CUDA graphs, cudagraph_capture_sizes=[1,2] vLLM ~1.1–1.3× + VRAM back Low high(works)/med(mag) bisect if a big AWQ MoE crashes
E3 Generate V100 fused_moe tuning JSON (benchmark_moe.py) per model E,N,dtype vLLM kills “sub-optimal MoE” penalty (single-digit %–?) Low-med high one-time per model/quant
E4 ngram + suffix on the 122B full-attention target (A/B vs MTP) vLLM ~1.5–2.5× on boilerplate Low med 122B ONLY — corrupts GDN
E5 llama.cpp HEAD -fa + q8_0/q8_0 KV rebuild (CUDA_ARCH=70) llama.cpp big prefill/TTFT win, neutral decode Low-med high decode unchanged at batch=1
E6 --max-num-seqs 1–4 + raise --gpu-memory-utilization, chunked-prefill OFF (V0) vLLM longer context / less warmup (not raw tok/s) Low high keep > PP stages on PP lanes
E7 Prefix caching ON + verify nonzero hit-rate on FLASH_ATTN_V100 vLLM skips repeated-prompt prefill (TTFT) Low high/med confirm hit-rate not silent-0
E8 TP-in-island + PP-across-PCIe to serve a bigger MoE; NCCL_PROTO=LL vs LL128 A/B vLLM enables bigger model; small decode-latency % Med high(placement)/med(NCCL) PP bubbles at low concurrency
E9 CPU expert-offload of a 120B–235B MoE via mainline llama.cpp -ot/--n-cpu-moe, NUMA-pinned llama.cpp serve bigger MoE at ~9–15 t/s Med high(works)/med(num) gated by EPYC DDR bandwidth
E10 ik_llama.cpp ≥ PR #1785 for CPU+GPU MoE offload A/B vs mainline ik_llama often faster than mainline on V100 Med high(V100 data) unofficial on Volta; --split-mode graph
E11 EAGLE3 on 122B using the lmsys/nvidia 235B-A22B head as a proxy vLLM ~1.4–1.9× if it loads Med-high low-med no exact head; chain-not-tree on Volta
E12 Multi-LoRA smoke test on V100 (rank-matched adapter) vLLM enables per-task adapters Low med Triton-FMA; may silent-ignore
✗ EP / DeepEP / Marlin / INT8-W8A8 / native-FP8 / FlashInfer / TRITON_ATTN / NVLS — none on Volta — high see kill-list

Top 3 to run first (highest upside ÷ effort, all on the existing 1Cat-vLLM lane): E1 (MTP), E2 (CUDA graphs), E3 (MoE tuning JSON) — E1 and E2 multiply (MTP cuts steps; graphs cut per-step overhead) on top of the AWQ-4bit byte savings already in place. Settle E0 (provenance) first so the multipliers are measured against the correct v100server-TP4 baseline.


7. Explicit “needs sm_80+ / sm_90 / NVSwitch — WON’T HELP VOLTA” kill-list

  1. FlashAttention-2 (flash-attn, vLLM FLASH_ATTN) — sm_80+ (hard error “Ampere or newer”).
  2. FlashInfer backend — sm_75+ fast paths; “remove on SM_70/SM_75.”
  3. vLLM TRITON_ATTN — not “needs sm_80” but HANGS on sm_70; use FLASH_ATTN_V100/XFORMERS.
  4. Stock Triton tutorial flash-attention — hangs on Volta (mma layout); FMA fallback otherwise.
  5. FULL CUDA-graph mode (attention in-graph) — needs FA3 = sm_90; V100 capped at PIECEWISE.
  6. Marlin / GPTQ-Marlin / FP8-Marlin — sm_80+. Machete — sm_90. CUTLASS-MoE / DeepGEMM / FlashInfer-TRTLLM-MoE — sm_90/sm_100. V100 → Triton fallback. (The fork’s “AWQ-Marlin” is really TurboMind sm_70 — see §2.)
  7. DeepEP (+ DeepGEMM, NVSHMEM/IBGDA) — Hopper sm_90; won’t build for sm_70.
  8. --enable-expert-parallel over PCIe / TP-only — “naive” all-to-all “not suitable for production.”
  9. NCCL_NVLS_ENABLE / NVLS (NVLink SHARP) — 3rd-gen NVSwitch + sm_90; self-disables on SXM2.
  10. NCCL symmetric-memory kernels (2.27+) / vLLM SymmMemCommunicator — unsupported CC ≤ 7.5.
  11. Native bfloat16 compute — sm_80+; V100 casts bf16→fp16 (verify model tolerance).
  12. Native FP8 compute decode — no sm_70 FP8; FP8 is fit/KV only, not faster than AWQ-4bit.
  13. INT8 W8A8 tensor-core path — V100 has no INT8 IMMA (dp4a SIMT only); W8A8 is Hopper/Ada.
  14. Punica CUDA LoRA kernels — required CC ≥ 8.0; obsolete (Triton replaced it) — gate gone, but a pre-Aug-2024 vLLM will still refuse LoRA on V100.
  15. ik_llama.cpp’s officially supported GPU tier — “Turing or newer” on paper (works in practice; DeepSeek-MLA FA is Turing+).
  16. ktransformers peak (AMX) kernels — Intel Sapphire-Rapids AMX; EPYC has none (AVX fallback).
  17. Stock vLLM wheels ≥ 0.20 — sm_70 dropped from build targets; use the 1Cat fork / ≤0.18.x / source.

8. Honest evidence gaps (drive the experiments, don’t trust the bands)


Sources (primary)

Compiled 2026-06-13. No system changes made; on-box facts inspected read-only. Re-measure every multiplier on a TP4 NVLink quad of the V100 server before relying on it.