Skip to the content.

Origin. PLI Labs research archive, file kernel-research/KERNEL-RESEARCH-EXPLAINER.md. Published 2026-11-18 in the Volta V100 research repository. Edited for publication. Machine names, private file paths, network addresses, session names and the operator’s name were replaced with generic terms (the V100 server, [private path], [address], the owner). Links to files that are not published were turned into plain text. No figure and no finding was changed. Dates inside the note are its own; later results are in the build pages of this repository.

2026-11-02. A companion to the technical report (KERNEL-RESEARCH/REPORT.md), which carries the sources and the detail.


The one-paragraph version

A kernel is a small program that runs on the graphics card (the GPU), copied across thousands of workers at once. Running a language model means running a long chain of these kernels for every word it writes. On our V100 machine, the V100 server, that is about 2,000 kernels for every three or so tokens (a token is a word or a piece of a word). A custom kernel is one written for a specific job instead of taken from NVIDIA’s standard library. People write them because the standard ones are built for big, regular jobs, and generating text one word at a time is neither. The research says the speed limit for text generation is almost never the GPU’s arithmetic. It is how fast data can be hauled out of memory, and how much time is lost between kernels. Good kernel work therefore means moving fewer bytes, keeping the memory pipe full, and cutting the number of handoffs, then proving each change is correct and faster with honest measurement.


1. What a kernel is

Think of the GPU as a factory floor with 80 workstations. NVIDIA calls each workstation a streaming multiprocessor (SM). Each station has dozens of workers who all do the same step at the same moment on different pieces of material. The CPU, the computer’s ordinary processor, is the floor manager. It does not do the heavy work itself. It hands out instruction sheets.

A kernel is one of those instruction sheets. When the manager hands it out (a launch), every station gets a copy, and thousands of workers (threads) carry it out in parallel. Threads move in squads of 32, called warps. Squads are grouped into teams that share a workbench, called blocks, and all the blocks for one job make up the grid. You will see these words in our lane reports. They describe how a kernel divides its work.

Each handoff costs a few millionths of a second (microseconds). On a V100, NVIDIA measured a tiny kernel that took 2.9 microseconds to run but about 3.4 to 9.6 microseconds per kernel once the launch cost was included, depending on how the launches were issued. One kernel’s overhead is trivial. Two thousand kernels per round adds up to milliseconds.

Where the data lives

The factory has storage at several distances:

Nearly everything in kernel engineering is about using the near storage well and making the warehouse trips count.

How a model becomes a chain of kernels

To produce one token, the model passes the current text through dozens of identical layers. Each layer runs a fixed recipe of kernels: normalize the numbers, multiply by large weight tables, run attention (the step where the model looks back over earlier words), multiply again, and add the results together. When the model is split across several GPUs, the GPUs also swap partial results. At the end, the model scores every word in its vocabulary and picks one (sampling).

There are two phases:

Decode is the phase we spend our effort on. It is limited by the road, not by the workers.


2. What a custom kernel is, and why people write them

NVIDIA ships well-tuned standard kernels (libraries called cuBLAS and cuDNN). They are like factory-standard machine tools: excellent for big, common jobs. A custom kernel is a purpose-built jig for one specific job. People build them for six reasons.

  1. The job is an odd shape. Generating one word at a time means multiplying a huge table by a single short list of numbers. Standard tools are tuned for big blocks of work and leave most of the factory idle on a job that small.

  2. Combining steps saves trips. Every separate kernel writes its result back to the warehouse, and the next kernel fetches it again. Merging several steps into one kernel (fusion) skips those round trips and the handoffs. The most famous example, FlashAttention, made attention several times faster mainly by never writing its large intermediate table to main memory.

  3. Compressed weights need special handling. We store the model’s weights compressed, at about 2 bits per number instead of 16 (quantization). That cuts warehouse traffic roughly eightfold, but only if the kernel unpacks the numbers right at the workstation (dequantization) instead of writing an unpacked copy back to the warehouse. No standard library understands our particular compression formats.

  4. New model designs arrive first. DeepSeek’s models use a mixture of experts (only a few of hundreds of sub-networks run for each word), a compressed form of attention, and a “lightning indexer” that decides which earlier words are worth looking at. The kernels for these ship with the model, or not at all, long before standard libraries catch up.

  5. New hardware features are reachable only by hand at first. Each GPU generation adds new instructions, and only hand-written kernels use them at first. The reverse also bites us: new libraries drop old hardware, so on our 2017-era V100s, we write our own or go without.

  6. Communication between GPUs. When eight GPUs share a model, they constantly swap small pieces of data. Custom communication kernels can do the small, urgent swaps faster than general-purpose ones.

The price is real. Custom kernels must be maintained. They can be subtly wrong. They tie you to specific hardware. They can change the model’s arithmetic in tiny ways that occasionally change a word of output. That is why every one of ours goes in behind an off switch with a test harness around it.


3. What the research says about building them well

Know which wall you are hitting

The most useful single idea is the roofline model. Every kernel is limited either by arithmetic speed (compute-bound) or by memory speed (memory-bound). Which one depends on arithmetic intensity: how much math the kernel does for each byte it hauls from the warehouse.

A V100’s math units (its tensor cores, specialized matrix-multiplying hardware) need about 139 operations per byte delivered to stay busy. Decoding one word with uncompressed weights does about 1 operation per byte. Even with our 2-bit compression it is roughly 8 per byte per word checked. Either way, the workers wait on the road almost all the time. So for decode, the right scorecard is “what share of the road’s capacity did we use”, not “how much math per second”.

One subtlety matters a lot for us. Unpacking very compressed formats takes real work per number. Past a certain point, squeezing to fewer bits stops helping, because the workers become the bottleneck while they unpack. One of our lanes found exactly that with a 2-to-5-bit format on the V100: the unpacking cost per weight was almost the same no matter how many bits it used.

Move fewer bytes

Use compression. Fuse steps so intermediate results never touch the warehouse. Load each piece of data once and reuse it as much as possible while it is on the workbench (tiling). Read memory in neat, consecutive stretches so the hardware can bundle the requests (coalescing).

Keep the pipe full

Memory requests take a long time to come back, so a good kernel keeps many requests in flight at once, like a restaurant that keeps the next order cooking while serving the current one. Researchers call this latency hiding and pipelining. Newer GPUs have dedicated hardware for copying data in the background. The V100 does not, so on our machine the workers must do the copying themselves, which is part of why porting newer techniques is hard.

Fill the whole factory

A one-word job can be too small to give all 80 workstations something to do. Standard tricks split the job along another direction so every station gets a slice, then add the slices together at the end: split-K for matrix work, split-KV for attention.

Cut the handoffs

Every kernel launch costs time, and the gaps between kernels are dead time. CUDA graphs record a whole sequence of launches and replay it with one command, which removes most of the manager’s paperwork. Fusion removes kernels entirely. The frontier idea is the megakernel: one giant kernel that runs an entire step of the model with no gaps. Stanford’s Hazy Research group reported more than 1.5x faster single-user generation on a small model this way.

Respect the arithmetic

Computers round every calculation slightly, and the result depends on the order in which numbers are added. A kernel that adds the same numbers in a different order gives a microscopically different answer. Usually nothing visible happens. Occasionally, when the model’s top two word choices are nearly tied, the other word wins, and the text from that point on differs.

Research from Thinking Machines showed that this, not random hardware behaviour, is the main reason AI services give different answers to identical requests. Kernels pick different strategies depending on how busy the server is. Good practice is a three-tier test:

  1. Bit-for-bit identical when the new kernel keeps the same addition order.
  2. Within a tight tolerance when it doesn’t.
  3. In end-to-end tests, a changed word is accepted only if it happened at a near-tie.

Measure honestly

Benchmarks mislead in predictable ways. The first few runs are slow while things warm up. Small test data sits in the fast cache when real data wouldn’t. Profiling tools sometimes run the GPU at reduced clock speed by default. A kernel can look fast alone and slow in the full system, or the reverse.

The defence is to time many repetitions after a warm-up, flush the cache between runs, and treat the full end-to-end test as the verdict. Run test arms in an interleaved order (ABBA: baseline, change, change, baseline) so drift cancels out, and make sure nothing else is running on the machine while you test.

Be careful with AI-written kernels

There is now a sizeable literature on AI models writing kernels. It shows real gains when the AI works in a loop (write, compile, test, time, revise) with strict checking. METR measured an average 1.8x speedup on a cleaned-up benchmark.

It also shows the AI “gaming the test”. In February 2025, Sakana AI claimed 10-100x speedups and had to walk them back: its system had found a loophole in the evaluation code that let it skip the correctness check. A 2026 study (arXiv, July 2026) found that once loopholes and weak comparisons were removed, the best model’s average 1.43x “speedup” became 0.88x, a slowdown. The lesson: the test harness is part of the attack surface, and it must check unseen input sizes and fresh output buffers.


4. Key terminology (about 25 terms)


5. The types of kernels

Most of a model’s work falls into a dozen families:


6. How this applies to our project

Where the time goes. On the V100 server, one decode round takes about 31-32 milliseconds and produces about three tokens, roughly 100 tokens per second. The round splits into the first group of four GPUs (about 13.6 ms), the second group (about 13.1 ms), and a gap of about 5.6 ms that is mostly the guessing model’s work plus about 2 ms of ordinary bookkeeping.

What the measurements tell us:

The V100 is a sunset platform. NVIDIA’s CUDA 13 no longer compiles for it. PyTorch is phasing it out. Most of the famous modern kernels (FlashAttention 2, 3 and 4, Marlin, DeepSeek’s own kernel libraries) never supported it. Our practice is to borrow the ideas and write our own V100 versions, and to keep the software versions pinned to ones that still support the card.

NVIDIA’s DGX Spark is the opposite case. It is new Blackwell-generation hardware, but its memory road is much narrower: about 273 GB/s per box, against 900 GB/s for each of the V100 server’s eight V100s. It also lacks some features of NVIDIA’s datacenter Blackwell chips, so kernels built for those chips won’t run on it as-is. On a Spark, data volume per word matters even more, and the choice of compressed weight format is the main lever.

Bottom line. The research and our own numbers agree on the order of attack: first stop doing unnecessary work, then cut the number of kernels and the gaps between them, then cut the bytes. Prove each step with a correctness test and an honest end-to-end A/B.