What Is Model Quantization in AI? Smaller, Faster Models

Quantization is a way of shrinking an AI model by storing its numbers with less precision. Instead of keeping each value as a 32-bit or 16-bit floating-point number, a quantized model might keep it as an 8-bit integer. The model then needs less memory and can often run faster, at the price of some accuracy. This article explains how the conversion works, the main methods and what trade-offs a user should expect.

The idea in one paragraph

A neural network is mostly a very large collection of numbers called weights, plus intermediate numbers called activations that appear while it runs. Hugging Face's documentation describes quantization as representing those weights and activations with low-precision data types such as 8-bit integers (int8) instead of the usual 32-bit floating point (float32). Fewer bits per number means a smaller file, and integer arithmetic can be faster on suitable hardware.

How much memory it saves

NVIDIA's technical blog gives a worked example. A Llama 2 model with 7 billion parameters needs about 14 GB for its weights in 16-bit form, because each parameter takes 2 bytes. Stored in 8-bit form, the weights need about 7 GB, half as much. The blog adds that models which generate text also keep a working memory called the KV cache, which can add several gigabytes at long context lengths such as 4,096 tokens.

How a float becomes an integer

Going from float32 to float16 is straightforward because both use the same style of number. Going to int8 is harder, because an 8-bit integer can hold only 256 different values while a float can cover a huge range. The task is to map a range of float values, from a lowest value a to a highest value b, onto the integers.

Hugging Face describes the usual scheme with two settings. The scale says how big a step between neighbouring integers is. The zero-point is the integer that stands for the real value zero; keeping zero exactly representable matters because zero appears everywhere in these models. A real number is turned into an integer by dividing by the scale, adding the zero-point and rounding. Values outside the chosen range are clipped to the nearest end.

NVIDIA shows a small example using a method called AbsMax, which sets the scale from the largest absolute value in the data. With values spanning roughly minus 6.2 to 12.8, the scale comes out near 0.028, and the number 6.4 maps to about 228.57 before rounding. Rounding is where precision is lost: two nearby real values can end up as the same integer.

Symmetric and asymmetric

The scheme above, with a zero-point, is called affine or asymmetric. A common special case is symmetric quantization, where the real range runs from minus a to plus a and the zero-point is fixed at 0. Hugging Face notes this gives up one of the 256 integer values but can be faster because some additions can be skipped. NVIDIA states that its TensorRT tools use the symmetric approach and that asymmetric quantization gave no significant accuracy gain over it in its discussion.

Per-tensor versus finer groups

A tensor is a block of numbers, such as one layer's weights. The simplest choice is one scale for the whole tensor. A finer choice is one scale per channel or per small block. Both sources agree on the trade: finer groups isolate unusual large values, which reduces error, but they require storing more scale values.

Deciding the range: calibration

Weights are known in advance, so their range is easy to measure. Activations depend on the input, so the range is harder. Hugging Face lists three approaches:

• Dynamic quantization: the range is worked out on the fly while the model runs. It needs little preparation but adds some overhead.
• Static quantization: the ranges are measured beforehand by passing example data through the model. Hugging Face says around 200 examples is typically enough.
• Quantization-aware training: the model is trained while simulating the rounding errors, so it learns to cope with them.

NVIDIA groups the first two under post-training quantization, applied to a model that is already trained, and contrasts them with quantization-aware training, which usually recovers accuracy better but requires more work.

The costs

NVIDIA states plainly that quantization can lead to some accuracy degradation compared with the original model. How much depends on the model, the number of bits and the method. Techniques with names such as AWQ, GPTQ and SmoothQuant aim to reduce that loss at low bit widths.

Energy savings are not guaranteed either. Hugging Face describes lower energy use as true only in theory and cites one benchmark study on two NVIDIA graphics cards. That study found that for small models of under 3 billion parameters, a 4-bit format could raise energy use by 25 to 56 percent despite cutting memory by 75 percent, because the work of converting numbers back costs more than the saved memory traffic. For models of 5 billion parameters and above, energy stayed near the 16-bit level while memory fell. These results come from a single third-party benchmark and one hardware family.

Why it matters to ordinary users

Quantization is a main reason large models can be run on laptops, phones and embedded devices that would otherwise lack the memory. When a model is offered in versions labelled with bit counts such as 8-bit or 4-bit, those labels describe how heavily it has been quantized. A smaller version fits on smaller hardware; the original is usually the reference for quality.

Sources

• Quantization, Hugging Face Optimum documentation
• Model Quantization: Concepts, Methods, and Why It Matters, NVIDIA Technical Blog