Batching: improve throughput without losing latency control.

Weight reuse is the opportunity

A dense model generating one token for one sequence performs relatively little arithmetic per byte of weights read. Processing several sequences together gives the runtime more work to do with loaded weight tiles.

That can raise aggregate tokens per second substantially. It does not imply that eight requests deliver exactly eight times the throughput or leave every user's latency unchanged. Kernel shape, cache traffic, and scheduling still matter.

Define both metrics: aggregate throughput counts output across all requests; per-request latency describes the experience of one user. Optimizing one without tracking the other can produce a fast-looking but frustrating service.

Estimate a decode step, then measure it

A simplified dense-model estimate is step time ≈ max(2PB/C, (W + BK)/BW). P is parameter count, B batch size, C effective arithmetic throughput, W weight bytes, K cache bytes read per sequence, and BW effective bandwidth.

The model assumes weight reuse across the batch. It omits several overheads and should be treated as an optimistic teaching estimate. Aggregate token throughput is approximately B divided by step time; per-sequence speed is its reciprocal.

The figure shows how the limiting resource can change. Use a memory budget that leaves room for the OS, temporary buffers, and other services; its weight-plus-cache calculation is not a complete admission policy.

Continuous batching changes the scheduling unit

Orca introduced iteration-level scheduling for transformer serving: work can be scheduled at generation-step boundaries rather than waiting for an entire fixed batch to finish. This helps with requests of different lengths.

PagedAttention addresses another constraint: fragmented and duplicated KV allocation. More efficient allocation can make larger useful batches possible without increasing physical memory.

Chunked prefill divides incoming prompt work so it can be scheduled alongside ongoing decode. SARATHI studies this approach. It creates a tunable balance rather than a guarantee that a long prompt never delays another request.

Set limits that reflect the service target

Choose targets for time to first token, inter-token latency, and completion time. State the percentile and the workload. A median that looks good can hide a long tail of stalled sessions.

Sweep concurrency with a representative mix of prompt and output lengths. Include arrivals over time, cancellations, and long-context requests. Record queue delay, rejection rate, cache pressure, and achieved throughput.

Set maximum active sequences, a per-step token budget, request length limits, and queue limits from those results. Reject or defer work that cannot meet the service target. An unbounded queue hides overload until users time out.

Separate interactive and bulk workloads when their objectives conflict. If they share a pool, define priorities and test whether either class can starve the other.

MoE needs a different traffic model

In an MoE model, tokens can select different experts. A batch may therefore read a larger union of expert weights than one token. The fixed-W approximation above becomes less representative.

Expert overlap, placement, routing balance, and communication all influence the curve. Measure the chosen architecture directly. Do not extrapolate dense-model speedups from the active-parameter count.

Keep the load-test inputs and runtime configuration with the result. The output of tuning should be an operating envelope the service can enforce, not a single peak-throughput number.