What the model says
Step explorer
One forward pass, broken into operators and placed on the hardware roofline. Change anything; the model recomputes in your browser.
Roofline
Each dot is one operator, at its arithmetic intensity and the FLOP/s it achieves (including launch overhead). Dot area is its share of step time. Ops below the slanted roof are memory-bound.
Where time goes
Slowest operators
Design space
Every layout the sweep could fit, at decode batch 1 to 512, for 2,048-token prompts and 256-token answers. Up and right is better: interactivity per user against throughput per device.
Pareto frontier
Frontier points
Serving simulation
A discrete-event simulator with continuous batching and chunked prefill, driven by Poisson arrivals. The curve below is Llama-3.1-8B on one H100 as load rises.
Load sweep: Llama-3.1-8B, 1x H100, 1,024 in / 256 out
Time to first token, per request
Validation on a real GPU
Measured on a Colab T4 with PyTorch and Hugging Face transformers, then compared with the model. Every fourth measurement is held out of the fit.
Loading...
Method and assumptions
Operators. Each layer is mapped to GEMMs with (M, N, K) per tensor-parallel rank, plus attention, elementwise work and collectives. FLOPs and bytes give arithmetic intensity; time is the larger of compute (tile-padded FLOPs at achievable peak) and memory traffic (bytes at achievable bandwidth), plus a fixed launch cost. Collectives use an alpha-beta model, taking the faster of ring and one-shot all-reduce.
Architectures. Loaded from Hugging Face config.json: dense GQA (Llama, Qwen), mixture of experts (Mixtral, DeepSeek-V3, with expected active experts per batch), and DeepSeek's multi-head latent attention (weight-absorbed decode over the 576-dim latent). Parameter counts are tested against published sizes.
LPU. All weights live in on-chip SRAM across many chips, executed as a chip-to-chip pipeline. An op larger than one chip's SRAM is split across chips and all-reduced. KV cache must also fit on chip.
Heterogeneous. Disaggregated serving (prefill on GPUs, decode on LPUs, KV cache shipped over a 400 Gb/s link), and an attention-FFN split (attention and KV on GPUs, FFN weights on LPUs, activations crossing twice per layer). Both are labeled speculative.
Validation status. Unit and property tests check invariants (no op beats its roofline, latency is monotonic in batch, Pareto points are non-dominated, calibration recovers known parameters). The Validation section compares predictions with real GPU measurements; where no measurement exists, treat absolute latencies as estimates and comparisons as the point.