Quantization: memory savings, rounding error, and evaluation.

Specify what is being quantized

A label such as four-bit does not describe the whole execution. It may refer to weight storage while activations, accumulators, or selected tensors use higher precision. Other methods also quantize activations or the KV cache.

MLX supports multiple quantization modes, including affine and low-precision floating-point formats. Group size, scale format, and kernel support are part of the model configuration. There is no universal four-bit arithmetic path.

Weight-only quantization can reduce memory capacity and traffic. Actual speed also depends on unpacking, dequantization, matrix shape, and available kernels. A smaller file is not sufficient evidence of a faster workflow.

Work through a simple affine grid

For an illustrative group with minimum m and maximum M, choose b bits and scale s = (M − m) / (2^b − 1). Store q = round((w − m) / s), and reconstruct the value as q × s + m.

At four bits there are 16 codes. If m = −1 and M = 1, s = 2/15. A weight of 0.2 maps to code 9 and reconstructs exactly as 0.2; a weight of 0.25 maps to the same code and reconstructs as 0.2.

For this unclipped nearest-grid scheme, ideal arithmetic gives an error bound of s/2. Quantized scales, clipping, and other formats change that bound. If every weight in a group is equal, the implementation must handle the zero-range case.

The figure uses 16 values so the rounding is visible. Production group sizes vary; 64 is one common configuration, not a requirement of quantization.

Include metadata and outliers

Suppose 64 four-bit weights share a 16-bit scale and a 16-bit offset. The codes require 256 bits and metadata adds 32, giving 4.5 bits per weight. Other formats have different overheads.

A large outlier widens a min-to-max grid and reduces resolution for ordinary values. Smaller groups can localize the effect while increasing metadata. Calibration-based methods use additional information about the model's behavior.

GPTQ optimizes reconstruction using approximate second-order information. AWQ uses activation information when choosing weight scaling. Neither turns one bit width into a universal quality guarantee.

Do not assume rounding errors are independent or always cancel. Their effect depends on the weights, inputs, nonlinearities, and the rest of the network.

Distinguish changed text from failed work

The final scores, or logits, determine the next-token distribution. Perturbing close scores can change the selected token. That alters the later context, so continuations may diverge; they need not differ at every later position.

The output illustration is a synthetic experiment, not a benchmark of a deployed model. It demonstrates sensitivity. It cannot establish a production error rate or prove that numbers and names are always the first capabilities to fail.

Compare formats on the same acceptance set

Hold the base checkpoint, tokenizer, prompt template, task inputs, and scoring rules fixed. Record the quantization format, calibration procedure, runtime version, memory use, latency, and task scores for each candidate.

Evaluate evidence extraction, exact identifiers, structured outputs, and refusal to invent missing fields where those matter.

A distribution metric or a general benchmark is useful context; neither replaces the workflow's acceptance criteria.

Use deterministic tools for calculations at every model precision. Validate units and source values when a model incorporates the result into prose. A formatting error or copied digit can occur without quantization being its cause.

Choose the smallest format that meets the measured quality and operating requirements. Preserve the prior model artifact so a regression has a practical rollback.