The Model Shapes the Building

Kian opens the workloads chapter of his course with Jensen Huang on stage in Taipei in June 2026, making a blunt argument: if you have a gigawatt of power and can't get more, throughput per watt is your revenue. The more useful part of the chapter is what comes after it, where Kian asks a question that sits underneath the watts: what is the model actually doing for each user, and how much of the machine does each user occupy while it does it?

The answer decides more of the building than I expected. A chatbot reply happens in two phases that stress different parts of a GPU. Every user holds a chunk of fast memory for as long as their request runs, and that chunk grows with every word. And promising each user a faster stream of words costs far more capacity than the speedup itself would suggest. Put together, those facts set how many GPUs a given crowd of users needs, what kind of memory those GPUs carry, and increasingly which chips go into which racks. My reading of the chapter is that the workload has to be designed in before the concrete is poured, because memory and bandwidth per user end up fixed in the hardware you buy and the hall you build around it.

This part stays with inference, the job of answering users. Training shows up once, as a contrast in how much memory it needs.

A reply happens in two phases

Language models read and write in tokens, fragments of words a few characters long. When you send a prompt, the model handles it in two distinct phases.

The first is prefill. The model takes your whole prompt, say a thousand tokens, and runs all of it through the network at once. As it goes, it computes two vectors for every prompt token, a key and a value. Kian describes these as the model's memorized meaning of each earlier token: a compact record of what that token contributed, which later tokens will need to look back at. Together they form the KV cache.

The second phase is decode. Now the model generates its answer one token at a time, and each new token has to consult everything that came before it. Without the cache it would have to recompute keys and values for the entire conversation at every step. With the cache it just reads them back and appends one new key and value for the token it produced. Think of someone summarizing a long document who keeps running notes so they never have to reread from page one.

The price of those notes is that they have to stay somewhere fast. The KV cache lives in GPU memory for the whole life of the request, and it grows linearly: every token of prompt and every token of answer adds the same fixed amount.

The two phases also lean on different parts of the chip. Prefill is a giant parallel matrix multiplication, because all the prompt tokens are known up front and can be processed side by side. That work is limited by compute, how many arithmetic operations per second the GPU can do. Decode is the opposite. Each step has to stream the model's weights (or, for a mixture-of-experts model, the weights of the experts the token is routed to) plus that request's KV cache out of memory and into the compute units, all to produce a single token. Very little arithmetic happens per byte moved. So at typical serving batch sizes, decode is limited by memory bandwidth, how fast data can be pulled out of HBM, the stacks of high-bandwidth DRAM packaged right next to the GPU die. If you want the background on why bandwidth became the binding constraint and how HBM gets its speed, the memory-stack series covers it in The Memory Wall and How HBM Works.

Prefill runs the whole prompt through the model at once as one big parallel matrix multiply, so it is limited by compute; decode produces one token at a time and has to stream the weights and the KV cache out of memory for each one, so it is limited by memory bandwidth

The two phases map onto the two delays a user notices. How long you wait before the first word appears is mostly prefill (plus any time spent queued). How smoothly the rest of the answer streams is decode.

Every user rents a slice of memory

Kian's worked example uses Llama 3.1 70B, a widely deployed open model with about 70 billion parameters.

Start with the weights. Stored in BF16, a 16-bit number format, each parameter takes 2 bytes, so the weights alone need 70 billion × 2 = 140 GB (strictly about 141 GB, since the real count is 70.6 billion). That cost is fixed whether one person uses the model or a thousand.

The KV cache is the per-user cost. The formula is short: 2 (one key and one value) × the number of layers × the number of KV heads × the size of each head × bytes per number. Llama 3.1 70B has 80 layers. It uses grouped-query attention, where its 64 attention heads share just 8 sets of keys and values, which already keeps the cache much smaller than it could be. Each head has 128 dimensions, stored in BF16. Multiply it out:

2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes, about 320 KiB per token.

That sounds small. Now give a user an 8K context (8,192 tokens) and their cache is 8,192 × 327,680 bytes = 2.5 GiB. Suppose that after the weights and working space, a GPU has 64 GB left over for KV cache. That fits about 25 requests at once (24 or 25, depending on whether you count gigabytes in decimal or binary). Quadruple the context to 32K and each request needs 10 GiB, so the same GPU holds about 6.

Leave 64 GB of a GPU for KV cache and it holds about 25 requests at an 8K context; give each request 32K and the same memory holds only about 6

Context length divides concurrency. Double how much each user is allowed to keep in the conversation and you roughly halve how many users a GPU can serve at the same moment. And the trend has been toward longer contexts: since reasoning models arrived in late 2024, models routinely write long internal thinking traces before answering, and every one of those tokens sits in the cache like any other.

Training is the useful contrast here, because it needs far more memory per model before a single user shows up. A training step runs the data forward through the model, runs backward to compute gradients (how each weight should change), then has an optimizer update the weights. The textbook recipe for mixed-precision training with the Adam optimizer keeps about 16 bytes per parameter: 2 for the BF16 weights, 2 for the gradients, 4 for a full-precision master copy of the weights, and 4 each for Adam's two running averages. For a 70B model that's about 1.12 TB, roughly 8× the 140 GB needed to serve it, and that's before counting activations, the intermediate results saved during the forward pass for use in the backward pass. With activations included the gap approaches an order of magnitude. Real training runs shard and trim this state in various ways (ZeRO partitioning, lower-precision formats), as Kian notes, but the gap doesn't go away. Holding one copy of a model for training takes far more HBM, and so far more GPUs, than holding one copy to serve it.

Model designers are shrinking the slice

If the story stopped there, you could draw a straight line: longer contexts, more users, so memory demand climbs forever. Kian warns against doing that naively, because model architects have been attacking the KV cache directly.

The best-known example comes from DeepSeek. Its multi-head latent attention (MLA), introduced with DeepSeek-V2 in May 2024, stores one compressed latent vector per token instead of full keys and values for every head, and the paper reported a 93.3% smaller KV cache than DeepSeek's earlier 67B model. DeepSeek V4, previewed on 24 April 2026, went further with a mix of what it calls compressed sparse attention and heavily compressed attention layers, stores the cache in FP8 (one byte per number), and supports a million-token context natively.

The ratios DeepSeek published are the clearest way to see the effect. At a 1M-token context, V4-Pro needs about 10% of the KV cache that V3.2 would, and V4-Flash about 7%. One caveat on absolute numbers: V3.2 only supports 128K tokens natively, so its roughly 50 GB at a million tokens is a projection of what it would need if stretched that far. On that projection, V4-Pro lands around 5 GB and V4-Flash around 3.5 GB. DeepSeek then shipped V4.1-Flash on 10 September 2026, and write-ups of its technical report put its cache at roughly 890 bytes per token, using 4-bit KV storage and a second generation of the compressed attention. Set that next to Llama 3.1 70B's 327,680 bytes and it's about 370× less memory per token.

Other labs take a different route. The Qwen3.8 model that appears on NVIDIA's benchmark chart in the next section mixes ordinary full-attention layers with linear-attention layers, which replace the ever-growing cache with a fixed-size running state. NVIDIA says this keeps memory bounded as context stretches toward a million tokens.

So should you expect memory demand to fall? Kian's answer is no, and he reaches for the Jevons paradox: when something gets more efficient to use, people use more of it. Cheaper tokens mean more requests, longer contexts and more agents running in parallel, so total memory demand keeps rising even as each token gets lighter. That's his judgment rather than something anyone has measured, but I find it plausible. What strikes me more, from the building's point of view, is how fast the target moves. A cluster gets specified well before it runs, around assumptions about how much memory each user will hold, and the models it ends up serving may hold a small fraction of that, or attract many more users.

Batching is how decode earns its keep

Back to decode's problem. Every step drags the model's weights out of HBM, and that transfer costs time and energy. If the GPU then multiplies those weights against a single user's token, most of the transfer is wasted: you paid to move a mountain of data and did a sliver of math with it.

The fix is batching: process many users' tokens in the same step. Read the weights once, multiply them against 8 requests instead of 1, and you get 8× the useful arithmetic for the same read. This is the main reason serving efficiency (and, in Kian's framing, tokens per watt) climbs with batch size. Push the batch large enough and decode starts to look compute-bound instead of bandwidth-bound.

Here the two earlier sections meet. Every request in the batch brings its own KV cache, and all those caches have to fit in memory at once. So the memory budget caps the batch, and the batch sets the efficiency. Longer contexts mean fewer requests fit, which means smaller batches, which means each weight read gets amortized over fewer users. The context length a provider allows quietly sets how efficiently its hardware can run.

How the batch is managed matters too. Kian uses a bus analogy that works well. Fixed batching is a bus that only lets new passengers on once everyone currently aboard has reached their stop. Requests start together and the batch only refills when the slowest one finishes, so a user who wanted a one-line answer leaves an empty seat that rides along until someone else's essay is done. Continuous batching is how a real bus works: people get on and off at every stop, and you only turn someone away when every seat is taken. The moment one request finishes, a waiting request takes its slot on the next step.

The idea first appeared as "iteration-level scheduling" in a 2022 system called Orca, and the open-source vLLM engine popularized it. Today it's standard: vLLM and SGLang use it, and NVIDIA's TensorRT-LLM calls it in-flight batching. The piece of software deciding who boards, the scheduler, is constantly trading one user's speed against the hardware's efficiency, and that trade has a measurable shape.

Speed per user is the expensive promise

First, the serving vocabulary.

• Throughput is total tokens per second across all users. A hundred users each getting 10 tokens per second is 1,000 tokens per second of throughput.
• Time to first token (TTFT) is the wait before the answer starts, dominated by prefill.
• Inter-token latency (ITL) is the gap between consecutive tokens for one user once the answer is streaming. In the example above, each user gets 10 tokens per second, so their ITL is 1/10 of a second, 100 ms. (You can't get it from the 1,000 aggregate; latency is a per-user experience.)
• Interactivity is tokens per second per user, the reciprocal of ITL. At 40 tokens per second, tokens arrive 25 ms apart. At 20 it's 50 ms, at 80 it's 12.5 ms.

Throughput is what the operator cares about, because it's how much product the hardware turns out. Interactivity is what the user feels. The question is how much of one you give up to buy the other, and Kian answers it with a chart NVIDIA published on 12 August 2026.

The setup: one GB300 NVL72 rack (72 GPUs) serving Alibaba's Qwen3.8-2.4T-A95B, an open-weights model released that August. It's a mixture-of-experts model, meaning it has 512 specialist sub-networks and routes each token to only a few of them, so although it has 2.4 trillion parameters in total, only about 95 billion are active for any given token. The run used FP8 numbers, multi-token prediction (a form of speculative decoding, where the model drafts several tokens ahead and then checks them), NVIDIA's TensorRT-LLM engine, and a task of 8K tokens in and 1K tokens out. NVIDIA labels the configuration "out of the box."

The chart puts interactivity (tokens per second per user) on the horizontal axis and throughput, measured in tokens per second per GPU, on the vertical. That unit is easy to misread, so I'll convert to users per rack as we go: users per rack = per-GPU throughput × 72 ÷ per-user speed. The readings below are eyeballed from the chart, so treat them as approximate.

At the far left, with each user getting a sluggish 10 to 15 tokens per second, each GPU produces just over 4,000 tokens per second. At the far right, each user gets just over 350 tokens per second, but each GPU manages only about 230. In between, the curve bends sharply:

• At 100 tokens per second per user, each GPU produces about 950, so about 10 users per GPU, or roughly 700 per rack.
• At 200 tokens per second per user, about 620 per GPU, so about 3 users per GPU, roughly 220 per rack.
• At 300 tokens per second per user, about 340 per GPU, so about 1 user per GPU, roughly 80 per rack.

Going from 100 to 300 tokens per second per user makes each user 3× faster and cuts the number of users the rack can serve by about 8×. If the trade were linear, tripling speed would cost you two-thirds of your users; instead it costs you seven-eighths.

The bend follows from the batching section, as far as I can tell. A big batch is what amortizes the weight reads. To make each user faster, the scheduler has to finish each step sooner, which means fewer users per step and less sharing of each expensive read. Past a point you're paying almost the full cost of moving the weights for a handful of people.

This curve is how a speed promise gets priced. A customer's service-level agreement (SLA) sets a minimum interactivity, say at least 100 tokens per second for every user. The operator reads the curve at that speed and gets the number of users each rack can carry, and from that, the cost per user. Kian calls the curve one of the most important metrics in the business, since it tells you whether a given hardware setup can serve a given product profitably at all. It also moves. Kian expects software optimizations to push this out-of-the-box curve up and to the right, and NVIDIA itself says further work, including its 4-bit NVFP4 format, is planned, so these FP8 numbers are a first result.

Splitting the work across different chips

Once you see that prefill wants compute and decode wants bandwidth, an obvious question follows: why run both on the same hardware? Running each phase on hardware suited to it is called disaggregation, and it's shipping today. NVIDIA's Dynamo serving software, for instance, routes prefill and decode to different groups of GPUs, so a burst of long prompts doesn't stall everyone's streaming answers.

The more radical version puts decode work on chips built around SRAM, the memory built directly into the chip itself. SRAM is much faster than HBM but holds far less, so these designs spread a model across many chips. Kian points to Cerebras as a natural fit for decode: its WSE-3, a single processor the size of a whole wafer, carries 44 GB of on-chip SRAM with roughly 21 PB/s of on-chip memory bandwidth.

NVIDIA's own answer is the Groq 3 LPX, built on technology from Groq, the inference chip startup whose designs NVIDIA licensed (and whose founder and president it hired) in December 2025. An LPX rack holds 256 Groq 3 LP30 chips, each with 500 MB of SRAM, for 128 GB total and 40 PB/s of on-chip bandwidth, with 640 TB/s of scale-up bandwidth between chips. It's designed to sit next to a Vera Rubin NVL72, the GPU generation after Grace Blackwell, and NVIDIA said in late August 2026 that LPX was in full production, with Nebius as the first cloud to adopt it.

The interesting part is where the line gets drawn. The Rubin GPUs keep the KV cache and do the prefill, plus the attention part of every decode step, the part that reads back through the cache. The LPX chips run the expert and feed-forward layers of each decode step, the bulk of the weights that would otherwise be dragged out of HBM token after token. Intermediate results shuttle between GPU and LPX on every token, and Dynamo orchestrates the handoff. So the split runs inside decode as well as between the two phases. The principle is the same either way: put each kind of work next to the memory and compute it actually needs.

For the building, this is where the model starts to show up in the floor plan. A disaggregated deployment means two different kinds of rack side by side, probably with different power and cooling profiles, wired together closely enough that activations can cross between them on every token. I suspect we'll see more of this, where the serving strategy decides the rack mix rather than one rack type being repeated across the hall. How those racks are wired together is part six.

Lower power can still mean more energy

One last point from the chapter, because it trips people up when comparing hardware. Power is a rate, how many watts you draw right now. Energy is the total, power multiplied by time, and energy is what you actually pay for.

Kian's example: Run A draws 100 kW for 10 minutes and uses 16.7 kWh. Run B draws a gentler 80 kW but takes 15 minutes and uses 20 kWh, 20% more energy for the same job. Lower power is not automatically more efficient. The number that matters for an inference fleet is energy per completed token, and a faster, hungrier configuration can win on it.

Power still matters in its own right, because the building has to survive the peaks. Training makes that hard: thousands of GPUs computing and then pausing in lockstep produce large, fast power swings that the electrical system has to absorb. That story belongs with backup power and storage in part five. Kian's summary for the chapter carries over to both: size the supply from the workload's measured power trace (its peak, its total energy and how fast it changes) and never from the average alone.

If you use AI products, this chapter offers a likely explanation for a couple of pricing patterns. A long conversation or a giant pasted document holds a big slice of fast memory on someone's GPU for as long as it runs, and it pushes other users off that GPU, so where a provider charges more for long contexts, I suspect that's a large part of the reason. The same curve would explain a "fast" tier costing more than its speedup alone suggests: on NVIDIA's chart, tripling the token rate per user cut the people a rack could serve by about 8×.

Next, the building has to get its electricity, and right now getting it early matters more than getting it cheap: Speed to Power.

---

This series is compiled from Kian's (@neuralkian) freeCodeCamp course "Inside an AI Factory – Power, Cooling, and GPUs Explained" (originally titled From Watts to Tokens; course notes at kiankyars.github.io/gigawatt). The framework and most examples are Kian's; I reorganized it by theme and re-checked the numbers, so any errors are mine. Companies are named only as examples, and nothing here is investment advice.