Prefill and decode: diagnose latency before tuning it.

First-token latency contains several waits

A slow first token is a symptom, not a diagnosis. The request may be waiting in a queue, retrieving documents, loading a model, processing its prompt, or buffering the first chunk before the browser receives it.

Record timestamps at admission, retrieval completion, model execution start, first model token, and first client-visible token. Measure decode separately using inter-token intervals and the time to complete the response.

A useful approximation is: client time to first token = network and application work + queue delay + uncached prefill + first-step and delivery overhead. A faster prefill kernel cannot fix a saturated queue.

The same weights see different matrix shapes

A token ID selects an embedding vector. During prefill, the vectors for multiple prompt positions pass through each transformer layer. Causal masking prevents a position from attending to later positions.

For a linear layer with weight matrix W, a prompt segment of T vectors forms a matrix X. Computing XW offers opportunities to reuse loaded weight tiles across those T positions. Layers still depend on the outputs of earlier layers.

During ordinary single-sequence decode, one new vector passes through the model per step. Batching adds vectors from other sequences. Speculative decoding and other execution strategies can change this simple picture.

Read the roofline as an estimate

Arithmetic intensity is operations performed per byte moved. In a simplified dense linear layer, T positions require about 2PT operations for P weights. If each weight occupies s bytes, intensity from weight traffic alone is about 2T/s.

At two bytes per weight, a one-position calculation gives about one operation per byte. With 128 positions it gives about 128. More reuse can shift the limit from memory bandwidth toward arithmetic throughput.

That estimate excludes attention, cache reads, activation traffic, kernel launches, and imperfect reuse. The interactive figure deliberately isolates a simplified model. Its crossover is not a hardware guarantee.

Quantization changes storage and the available kernels. It can affect both prefill and decode. Benchmark the actual runtime rather than assuming that prompt processing receives no benefit.

Long prompts and shared service change the result

Full causal attention has a quadratic number of position pairs during prefill. Efficient kernels reduce memory traffic and intermediate storage, but do not make that underlying pair count disappear. Other attention architectures differ.

During decode, attention also accesses the retained context. At long contexts, KV traffic can become substantial. A model with fast short-prompt generation may behave differently after a large document.

SARATHI studies chunked prefill: prompt work is divided into pieces and scheduled alongside decode. This can reduce interference, but chunk size and scheduling still determine the latency trade-off.

Benchmark cold, warm, and cached requests

Run a prompt-length sweep with a fixed output limit. Separate cold startup from a loaded model. For each length, compare a new prefix with a genuinely cached prefix, using the server's hit metrics to confirm reuse.

Repeat at the concurrency your users will create. Report median and tail latency, cancellations, and failures alongside aggregate throughput. A good optimization improves the required user experience under that load.