Unified memory: budget weights, context, and concurrency.

One pool, several competing allocations

MLX documents direct CPU and GPU access to the same memory pool on Apple silicon. This avoids an explicit transfer between separate CPU and GPU memory pools for shared arrays. It does not eliminate allocations or every internal copy.

Budget total memory as weights + KV cache + runtime working space + application services + operating system + headroom. Retrieval, OCR, and other desktop work compete with inference if they run on the same machine.

Apple's current Mac Studio specifications list an M5 Ultra configuration with up to 512GB of unified memory. That is installed capacity, not a promise that one model can use every byte or that every size will run at an acceptable speed.

Estimate weight memory from the actual format

For P parameters stored at b bits each, the raw code payload is P × b / 8 bytes. A 70-billion-parameter model at four bits therefore starts at 35GB in decimal units, before metadata and any tensors stored at higher precision.

In an illustrative affine format, each group of 64 weights also stores a 16-bit scale and a 16-bit offset. That adds 0.5 bits per weight: 4.5 bits effective storage, or about 39.4GB for the same parameter count.

Those figures are estimates for the stated format. Read the model files and runtime allocation metrics. Mixed tensor formats, padding, conversion buffers, and allocator caching can make peak memory larger than the file size.

Add context memory per active sequence

For conventional attention with grouped key/value heads, KV bytes per token = 2 × layers × KV heads × head dimension × bytes per cache element. The factor of two covers keys and values.

An example with 32 layers, eight KV heads, a 128-element head, and two-byte cache values needs 131,072 bytes per token. At 8,192 cached positions, that is 1GiB per sequence. Eight independent sequences need 8GiB before overhead.

Use the model's actual attention architecture. Sliding windows, compressed latent caches, quantized caches, and shared prefixes change the budget. Reserve space for both prompt and generated tokens.

Capacity does not predict token speed

A model that fits can still be too slow. Decode may be limited by memory traffic, attention over a long cache, arithmetic, or the runtime's kernels. Concurrency changes the balance.

Advertised memory bandwidth is not measured application throughput.

Measure cold load, warm time to first token, inter-token latency, completed requests per minute, and peak memory. Run short and long prompts at the intended simultaneous load. Include the retrieval and document-processing work.

Record the model hash, quantization format, runtime version, prompt lengths, output lengths, and cache state. Without those details, a tokens-per-second result cannot be compared meaningfully with another run.

Leave room for operations

Test load spikes, cancellation, long outputs, and a restart. Set admission limits before memory pressure triggers uncontrolled swapping or failures. A desktop that also runs other applications needs a larger operating margin.

Keeping an old model warm during an update requires additional memory. If both versions do not fit, plan a drain-and-reload window or another host. A rollback plan should describe how long it takes to become ready again.