How Ten Thousand GPUs Talk
Kian has a small worked example in the networking chapter that carries most of this part. Four servers finish a training step and have to combine their results before anyone can start the next one. Three of them are done in 20 ms. The fourth takes 50 ms. The step takes 50 ms, and for 30 of those milliseconds three expensive machines do nothing. Stretch that to tens of thousands of GPUs and the same rule applies to the wires between them: the most crowded switch or uplink in the building sets the pace of the whole job.
A cluster runs at the speed of its slowest card and its most crowded link, which means the network can't be wired up after the hall is finished. Where the racks stand decides how long the cables are. Cable length decides whether a link can be copper or has to be light. The medium and the switch layout decide how much bandwidth each GPU actually gets when everyone talks at once. That's why the network topology gets drawn together with the floor plan. Part one called data the third flow through the building; this part follows it from the GPU out to the edge of the campus and on to the next one.
A GPU sits inside three networks, sorted by distance
The industry splits GPU networking into three tiers by how far the data has to go. Scale-up is inside one rack. Scale-out is between racks in the same building. Scale-across is between sites that can be hundreds of kilometers apart. NVIDIA adopted "scale-across" as the official name of that third tier in August 2025, when it launched a product line for it.
A GPU sits inside three networks sorted by distance: scale-up over NVLink inside one rack, scale-out over InfiniBand or Ethernet between racks in a building, and scale-across over fiber between campuses
Scale-up is where GPUs come closest to acting like one machine. In an NVL72 rack, the 72 GPUs connect through nine NVLink switch trays (NVLink is NVIDIA's own GPU-to-GPU link). Each Blackwell GPU gets 1.8 TB/s of NVLink bandwidth, about 130 TB/s across the rack. There's no cable between every pair of GPUs. Every GPU plugs into the switch trays, so any GPU reaches any other through one switch hop, at full speed, and no pair has to wait for another pair to finish (the industry term is non-blocking).
How much fits in that one domain? Kian's example is Kimi K3, the 2.8-trillion-parameter open-weight model Moonshot AI released in July 2026. The arithmetic is short. At 4 bits per parameter the weights come to about 1.4 TB, or about 2.8 TB at 8 bits. A GB200 NVL72 carries roughly 13.4 TB of HBM (72 GPUs at about 186 GB each). A full copy of a model this size fits inside a single rack, with room left over for the KV cache that part two spent so long on.
Past 72 GPUs, data has to leave the NVLink domain. In standard builds, traffic between racks runs over InfiniBand or Ethernet, and NVLink stops at the rack. This is an easy point to blur, so the exceptions are worth stating precisely. The GB200 NVL36x2 configuration splits one 72-GPU NVLink domain across two racks, and NVIDIA's GTC 2026 roadmap carries NVLink across racks over optics with the Feynman generation around 2028, with domains of 576 or 1,152 GPUs. Today, the path out of a rack goes GPU, then NIC (network interface card, the adapter that puts data onto the network), then a chain of switches over optical links, then another NIC, then the destination GPU. On a GB300 NVL72, each GPU has its own ConnectX-8 SuperNIC at 800 Gb/s.
The standard scale-out example is Llama 3.1 405B, which Meta pre-trained in 2024 on more than 16,000 H100s inside one building. Meta had announced two 24,576-GPU clusters in March 2024, one on Ethernet and one on NVIDIA's Quantum-2 InfiniBand, both giving each GPU 400 Gb/s; the Llama 3 run used the Ethernet one, built as eight pods of about 3,000 GPUs each. Kian calls 24K GPUs "actually small" next to what's being built now. The largest single sites today hold hundreds of thousands of GPUs (Microsoft says "hundreds of thousands" of Blackwell GPUs per Fairwater site); you only reach millions when you count a whole company's fleet.
The step down from scale-up to scale-out is steep. NVLink's 1.8 TB/s adds both directions together, so it's 900 GB/s each way. The GB300's 800 Gb/s NIC is 100 GB/s each way. That's a ninefold drop the moment data crosses the rack wall. Much of the rest of this part follows from that gap: traffic that talks constantly has to stay inside the rack, and the traffic that leaves has to be the kind that can tolerate the narrower pipe.
Every step waits for the last one in
Training is chatty because the model is split across GPUs, and the pieces have to stay in sync. Tensor parallelism cuts a single layer's matrix math across several GPUs, which then swap partial results many times per step. Data parallelism runs copies of the model on different slices of data; after each backward pass, the copies average their gradients (the corrections each one computed) so they all stay identical. That averaging is a collective operation called all-reduce, and a collective finishes only when the last participant does. Hence the 20 ms and 50 ms servers.
Kian draws the facility lesson from this. Suppose you build a training data center and economize on scale-out bandwidth, skipping InfiniBand, say. NVLink inside each rack is still fast, and it doesn't help. Cross-rack traffic becomes the bottleneck, training crawls, and the building has lost the reason it was built. His conclusion is that hardware and software have to be designed together, with the expected communication pattern shaping the build from the start.
He also lists four things that decide how much of the installed network a job actually gets to use: routing, congestion control, the software that runs collectives, and job placement (which GPUs a job lands on). I found this the most practical sentence in the chapter, because it means two clusters with identical switches can deliver very different usable bandwidth depending on software and scheduling.
Leaf, spine, and the price of a thin uplink
The hardware Kian walks through is NVIDIA's InfiniBand kit from the H100 generation. The NIC is a ConnectX-7: 400 Gb/s, attached to the server over PCIe Gen5 x16.
A note on units, because the chapter trips on them. Lowercase b is a bit, uppercase B is a byte, and a byte is 8 bits. 400 Gb/s is 50 GB/s. Kian jokes about "Jensen math": quoting bits because the number looks eight times bigger, and adding both directions together. The bits half is just a joke, since networking has always quoted line rates in bits. The both-directions half is a real marketing habit, and both headline figures in this part use it: NVLink's 1.8 TB/s, and the 51.2 Tb/s of the switch below.
That switch is the Quantum-2 QM9700. It's one rack unit tall, with 32 OSFP cages on the front; each cage takes a twin-port transceiver or cable, giving 64 ports at 400 Gb/s. The NICs live in the servers, not in the switch.
Switches like this are arranged as a leaf–spine network. Servers plug into leaf switches. Leaves plug upward into spine switches. Two servers on the same leaf talk through that leaf alone; two servers on different leaves go leaf, spine, leaf. Each hop adds latency, and Kian compares the layers to a CPU's cache hierarchy, where each step up to a slower tier costs you. NVIDIA's DGX H100 SuperPOD uses this design. It's common but not universal.
Kian's teaching version of the wiring goes like this. A DGX H100 is an 8-GPU server. Put four of them under one leaf and you have 32 GPUs, so 32 downlinks. Give the leaf 32 uplinks to the spine and every GPU can send off the leaf at full speed. That's 1:1, or non-blocking. (The real SuperPOD is "rail-optimized": each leaf takes one GPU's link from each of 32 servers, rather than all eight GPUs of four servers. The arithmetic is the same, so the simplified version is fine for intuition.)
Now cut the uplinks. The ratio of downlink bandwidth to uplink bandwidth is called oversubscription. Kian's example: four server ports at 400 Gb/s give 1,600 Gb/s coming into the leaf, and two uplinks at 400 Gb/s give 800 Gb/s going out. That's 2:1. Moving 256 Gb (32 GB) off the leaf takes at least 0.32 s; at 1:1 it would take 0.16 s. Those are lower bounds, one direction, no protocol overhead. Traffic that stays on the leaf doesn't notice. Traffic that leaves takes twice as long whenever every server pushes out at once, which is what tends to happen in a synchronized training step. The extreme version has a name, incast: many senders converge on one receiver or one narrow uplink and overflow it.
Oversubscription: four servers push 1,600 Gb/s into a leaf switch that has only 800 Gb/s of uplinks to the spine, so at 2:1 traffic leaving the leaf queues whenever everyone sends at once
Why would anyone oversubscribe on purpose? Presumably because every uplink costs switch ports, optics and power. Meta's Llama 3 cluster is a clean example of doing it deliberately. Inside each 3,072-GPU pod, the network has full bisection bandwidth, meaning no oversubscription at all. The layer that joins the eight pods runs at 1:7. Meta then placed the traffic to fit: tensor and pipeline parallelism, which need lots of bandwidth, stay inside a pod, and data parallelism, which tolerates delay best, crosses the thin layer.
Meta's Ethernet cluster also suggests the fabric's brand matters less than you might think. InfiniBand is NVIDIA's; Ethernet is an open standard. What lets Ethernet compete is RDMA (remote direct memory access), where the NIC writes straight into another machine's memory without the CPU copying data in and out. RDMA over Ethernet is called RoCE, and Meta trained Llama 3.1 405B on it.
Leaf–spine isn't the only shape. Google's TPUs use a torus: each chip is wired directly to its neighbors, six of them in a 3D torus, from TPU v4 through Ironwood. AWS Trainium2 has its own scale-up link, NeuronLink, arranged as a 2D torus inside a 16-chip server and a 3D torus across a 64-chip UltraServer, with AWS's EFA network for scale-out. Dragonfly, where groups of nodes connect to other groups through fast long links, comes from supercomputing. Kian's advice is to know these exist without memorizing them, which seems right.
Distance picks the cable
Copper weakens and smears high-speed electrical signals as it gets longer. At 400G, NVIDIA's passive copper cables come in lengths from half a meter to 2 meters; active copper, with a small chip boosting the signal, stretches to about 3 m. That's enough inside a rack, and copper there is cheap and draws almost no power. Beyond a few meters you switch to fiber, which needs a transceiver at each end to turn electricity into light and back.
Fiber itself comes in two kinds. Multimode fiber has a wide core, so light travels down it along many paths (modes) at slightly different lengths. The paths smear each pulse, which limits how far a fast signal survives: NVIDIA's 400G SR4 multimode optics reach 50 m. Single-mode fiber has a narrow core and one path, so the pulse stays sharp over far longer distances; 400G DR4 single-mode optics reach 500 m. Both carry light at about the same speed, so latency per kilometer is essentially identical. The rule that falls out is to use the cheapest medium the distance allows: copper inside the rack, multimode across a row, single-mode across the hall.
That rule turns rack placement into a cabling and power question, and Microsoft's Fairwater buildings show how far it can push the architecture. They are two stories, which is unusual for data centers. Kian reads the second floor as roughly doubling compute per footprint. Microsoft's own stated reason is about the network: stacking racks in three dimensions shortens the cable runs, which keeps latency down for a design where every GPU sits on one flat network. The price is an upper floor that has to hold racks weighing well over a tonne each. In this case the network helped decide how many floors the building has.
The other way to shorten the expensive electrical path is to move the optics. In a normal switch, the signal travels from the switch chip across the circuit board to a pluggable module in the front panel, and that board trace is lossy. Co-packaged optics (CPO) puts the optical engine right next to the switch chip, so the electrical leg shrinks to millimeters. Less power goes into pushing signals through copper, and more bandwidth fits; the cost is harder packaging. Kian ties this to the voltage-regulator lesson from part four: keep the lossy electrical segment as short as possible. NVIDIA says its CPO switches are 3.5 times more power-efficient than links built on pluggables, which is a vendor claim. In 2026 CPO ships in switches: NVIDIA's Quantum-X Photonics InfiniBand switches are in early deployments, its Spectrum-X Photonics Ethernet switches were declared in full production in August, and Broadcom sells CPO Ethernet switches. CPO on the GPU's own links is roadmap, tied to Feynman in 2028. The light-versus-copper series covers the crossover in depth, and The Packaging War covers the fight over where the optics should sit.
Google took a different route for its TPU pods: switches that don't read the data at all. An ordinary switch converts each packet to electricity, reads its address, and sends it on. An optical circuit switch (OCS) uses tiny MEMS mirrors that tilt to aim light from any input fiber to any output fiber. Change the mirror angles and you've physically rewired the network. Google's Palomar OCS has 136 by 136 ports. Data rides on light around 1310 nm, and a separate 850 nm beam, read by a camera, is used to steer the mirrors into alignment.
A TPU v4 pod has 4,096 chips: 64 cubes of 4×4×4 chips, joined through 48 of these OCS units. Google calls the whole chip-to-chip fabric ICI, for inter-chip interconnect. Because the mirrors can be re-aimed, a failed cube doesn't take the pod down. When Google trained Gemini 1.0 Ultra, it re-wired cubes into new 3D tori in about 10 seconds and kept spare cubes in each SuperPod as hot standbys. This was the part of the chapter I enjoyed most: fault tolerance by re-aiming mirrors.
Leaving the building
Eventually some data has to leave the campus: model weights shipped elsewhere, users' requests coming in, or a training job that spans sites. The path runs from the cluster to a router, then to a patch panel that organizes the physical fiber, then to the meet-me room, where the building's fiber cross-connects to carriers such as AT&T or to the owner's private fiber. All of that needs floor space, and the fiber entry paths and ducts have to be planned so they don't fight the power and cooling routes.
There's a redundancy choice here too. Say three fiber routes enter the site from different directions so that one backhoe can't cut them all. Do you bring them into three handoff rooms or merge them into one? One room is simpler and cheaper, and it puts all three routes back through a single point of failure. It's the same shared-bus trap that part five found in the backup power chain.
Distance also sets a floor on latency. Light in fiber moves at about 200,000 km/s, roughly two-thirds of its speed in vacuum, or about 5 µs per kilometer. Two facilities 100 km apart are 0.5 ms apart one way and about 1 ms round trip, before any routing or equipment adds more. Kian recommends Jeff Dean's well-known table of latency numbers for this kind of intuition: a round trip inside one data center is about 0.5 ms, and California to the Netherlands and back is about 150 ms.
Microsoft's Fairwater is the biggest scale-across example in the course. The Wisconsin site at Mount Pleasant was unveiled on 18 September 2025 and the Atlanta site on 12 November 2025. Microsoft links them with a dedicated "AI WAN" into what it calls an AI superfactory, and says it added more than 120,000 miles of new fiber across the US in the past year. The two sites are about 700 miles (about 1,130 km) apart, so even a perfectly straight fiber would take at least 5.6 ms one way and 11 ms round trip, and real routes are longer. Ordinary internet links between the sites would also be measured in milliseconds. What Microsoft is selling is the absence of queuing: dedicated fiber that carries only its own traffic, "congestion-free, at nearly the speed of light." When every step waits for the slowest participant, removing the occasional traffic jam is probably worth more than shaving distance.
Microsoft also says Fairwater's network is designed for training models with "hundreds of trillions of parameters." Kian is skeptical; his guess is that the facility will be obsolete before models reach that size, and he finds the congestion-free fiber the more relevant claim. Google, for what it's worth, already trained Gemini 1.0 Ultra across multiple data centers in 2023. The split it used matches Meta's: model parallelism inside each SuperPod, data parallelism across them. Both companies put the most delay-tolerant traffic on the longest, thinnest link.
Training and inference want different networks
Most of this chapter is about training, and that's deliberate. A single copy of even a 2.8-trillion-parameter model can live in one rack, and serving more users mostly means adding more copies. Large serving deployments do still spread across racks (prefill and decode on separate machines, or experts in a mixture-of-experts model spread wide), but pre-training is the workload that needs the whole fabric working at once.
Google has started building that difference into the chips. It announced its eighth-generation TPUs at Cloud Next on 22 April 2026, as two parts: TPU 8t for training and TPU 8i for inference, with general availability planned for later in 2026. TPU 8t keeps the 3D torus, at 9,600 chips per superpod. TPU 8i moves to a design Google calls Boardfly, which it says was inspired by dragonfly, cutting the worst-case path from 16 hops to 7 in a pod of about 1,000 chips. The target is the all-to-all traffic of mixture-of-experts inference, where every chip may need to reach every other chip quickly. Google chose a different topology for each workload within a single generation.
Pre-training also has to be fed. Training data runs to petabytes, so it gets staged ahead of time in tiers. Meta's Research SuperCluster, announced in January 2022 with 6,080 A100s in its first phase, used 175 PB of bulk flash storage, a 46 PB cache, and another 10 PB of fast shared file storage, with datasets prepared once and reused across runs. Kian's point is that storage and its network take their own rack space, power, cooling and cabling, and that this weighs much more on a pre-training cluster than on an inference one.
If you rent GPUs, or just read announcements about new clusters, the GPU count probably tells you less than any other number in the release. The questions that decide how fast a job runs are how many GPUs share one NVLink domain, what the uplink ratio is above them, and whether the sites are joined by dedicated fiber or by shared links. A cluster that looks twice as big on paper could plausibly train slower if its slowest link is thinner.
Every switch, optic and GPU in this part turns its watts into heat. Part seven follows it out.
This series is compiled from Kian's (@neuralkian) freeCodeCamp course "Inside an AI Factory – Power, Cooling, and GPUs Explained" (originally titled From Watts to Tokens; course notes at kiankyars.github.io/gigawatt). The framework and most examples are Kian's; I reorganized it by theme and re-checked the numbers, so any errors are mine. Companies are named only as examples, and nothing here is investment advice.