Quantization: Scalar, Product and Binary

Vectors are the memory bill, and quantization is how you cut it — by 4×, 16× or 32×, in exchange for accuracy you get most of back through rescoring. What each method throws away, and the arithmetic for deciding before you pay for the RAM.

Part 3 showed that the HNSW graph is a minority of your memory at default settings. The majority is the vectors themselves, and there is a lot of them.

This part is about making them smaller. The arithmetic is exact — a float32 is four bytes whatever machine you run on — so unlike most of the trade-offs in this series, you can work out what quantization saves you before you buy anything.

What it costs you is accuracy, and that part is measured, using Part 4.

---

Try this first

A million vectors, 1,024 dimensions each, float32.

Work out the memory for the raw vectors. Then work out what it becomes if each number is stored as a single byte instead of four. Then as a single bit.

Three numbers. Do them before reading on, because the third one is the surprise, and the gap between the second and third is the whole reason three methods exist rather than one.

---

The bill you are trying to cut

The vectors’ size is multiplication, not a benchmark:

bytes = points × dimensions × bytes_per_number

At a million points and 1,024 dimensions, in float32:

1,000,000 × 1,024 × 4 = 4,096,000,000 bytes ≈ 3.8 GiB

That’s before the HNSW graph, before payloads, before ids. It is the floor of what the collection costs to keep in RAM, and it scales linearly with both your corpus and your model’s output size — which is why the choice of a 1,024-dimension model over a 384-dimension one is a memory decision as much as a quality one.

Quantization attacks bytes_per_number.

Every figure in that table is arithmetic you can check with a calculator. Nothing was measured and nothing depends on my machine.

---

Scalar quantization: four bytes to one

The simplest idea that works. Each dimension’s float becomes an 8-bit integer.

To do that you need a range to map onto. Qdrant looks at the distribution of values in that dimension across your data, takes a quantile to trim outliers, and maps that range linearly onto 0–255.

from qdrant_client import models

client.create_collection(
    "notes",
    vectors_config=models.VectorParams(size=1024, distance=models.Distance.COSINE),
    quantization_config=models.ScalarQuantization(
        scalar=models.ScalarQuantizationConfig(
            type=models.ScalarType.INT8,
            quantile=0.99,      # trim the extreme 1% so outliers don't stretch the range
            always_ram=True,    # keep the quantized vectors in RAM even if the originals are on disk
        )
    ),
)

What it throws away: precision within each dimension. A float32 can represent about seven significant decimal digits; an int8 gives you 256 levels across the range. The direction of the vector is approximately preserved, which is what the distance metric cares about.

Why quantile matters. If one value in a dimension is wildly larger than the rest, a naive min–max mapping spends most of its 256 levels on empty space, and every ordinary value collapses into a handful of buckets. Trimming at 0.99 clips that outlier and spends the range where the data actually is.

What it costs in accuracy: the least of the three. This is the default recommendation for a reason — a 4× memory saving for a small accuracy loss that rescoring largely recovers.

---

Product quantization: 16× and up

Scalar quantization treats each dimension independently. Product quantization looks at groups of dimensions together.

Split the vector into chunks. For each chunk position, cluster the chunks seen across your data into a fixed number of representative centroids — a codebook. Now a chunk is stored not as its numbers but as the id of the nearest centroid, which fits in a byte or two.

client.create_collection(
    "notes",
    vectors_config=models.VectorParams(size=1024, distance=models.Distance.COSINE),
    quantization_config=models.ProductQuantization(
        product=models.ProductQuantizationConfig(
            compression=models.CompressionRatio.X16,   # x4, x8, x16, x32, x64
            always_ram=True,
        )
    ),
)

What it throws away: everything about a chunk except which centroid it was nearest. Two different chunks that happened to be near the same centroid become identical.

What it costs: more accuracy than scalar, and — importantly — more CPU at query time. Distances have to be computed through the codebook rather than directly, which is more work than comparing bytes. Product quantization is the choice when memory is genuinely the binding constraint and you have CPU to spare.

It also costs build time: the codebook has to be trained on your data before anything can be encoded.

---

Binary quantization: one bit per dimension

The extreme. Each number becomes a single bit — essentially, is this dimension above or below a threshold.

client.create_collection(
    "notes",
    vectors_config=models.VectorParams(size=1024, distance=models.Distance.COSINE),
    quantization_config=models.BinaryQuantization(
        binary=models.BinaryQuantizationConfig(always_ram=True)
    ),
)

What it throws away: almost everything, per dimension. What it keeps is the sign pattern across all of them, and in high dimensions that pattern carries a surprising amount of the signal.

What it buys: 32× compression, and speed — comparing two binary vectors is an XOR and a population count, which modern CPUs do extremely fast. Binary quantization is the only one of the three that reliably makes search faster as well as smaller.

Where it fails: it needs high dimensionality and it needs the embeddings to tolerate it. It works well on many 1,024-dimension and larger models and poorly on small ones. At 384 dimensions there are simply fewer bits to carry the signal, and you should expect a bigger accuracy hit than the headline suggests.

This is the one you must measure, not assume. Whether binary quantization is usable is a property of your specific embedding model, and the only way to find out is Part 4’s procedure on your own vectors.

---

Rescoring: how you get the accuracy back

Here is the part that makes all of this practical, and it is the part people miss.

Quantized vectors don’t have to be the final answer. They can be a filter.

The search uses the small, fast, lossy vectors to produce a shortlist — more candidates than you actually need. Then it re-reads the original full-precision vectors for just those candidates, computes exact distances, and re-ranks.

hits = client.query_points(
    "notes",
    query=vector,
    limit=10,
    search_params=models.SearchParams(
        quantization=models.QuantizationSearchParams(
            rescore=True,        # re-rank the shortlist using the original vectors
            oversampling=2.0,    # fetch 2x the candidates before rescoring
        )
    ),
).points

oversampling=2.0 with limit=10 means the quantized pass retrieves 20 candidates, those 20 are rescored against the true vectors, and the best 10 of those are returned.

The trade is clean:

• The quantized pass is cheap and slightly wrong, and it only has to get the right answers somewhere in its top 20 rather than in its top 10.
• The rescoring pass is exact, and it only touches 20 vectors.

That is why aggressive quantization is viable at all. Binary quantization with rescoring is a completely different proposition from binary quantization alone, and comparing them as if they were the same thing is the most common mistake in this area.

The catch: rescoring needs the original vectors to still exist somewhere readable. If they have been pushed to disk (Part 10), each rescore is a disk read, and oversampling multiplies how many. Memory saved in one place turns into I/O in another — which is the honest shape of the whole trade.

---

Explain it like I’m ten

Imagine a library where every book has a detailed summary on its cover.

The summaries are long and take ages to read, but they’re accurate. Finding the right book means reading a lot of long summaries.

Scalar quantization is rewriting every summary in shorter words. Still one summary per book, just less detail in each.

Product quantization is noticing that summaries fall into a few hundred common types, and replacing each one with “this is a type 47 summary”. Enormously shorter. But now two books with slightly different summaries look identical, because they’re both type 47.

Binary quantization is replacing each summary with a row of yes/no ticks — does it mention a war, does it mention the sea, is it funny. Tiny. Astonishingly, if you have a thousand boxes to tick, that’s often enough to find roughly the right shelf.

Rescoring is what saves all of this. You use the short version to grab twenty likely books off the shelf, and then you read the twenty real summaries properly and pick the best one. You did the slow careful reading twenty times instead of a million times.

Where the analogy breaks: book summaries are written by people to be informative, so shortening them loses meaning in ways you could predict. A vector’s dimensions mean nothing individually — no single number is “mentions the sea” — so which information survives compression is not something you can reason about. It has to be measured.

The precise version

Let v ∈ ℝ^d be stored at b bits per component instead of 32. Memory scales by b/32, exactly, and that is the only guaranteed part.

Scalar (int8): per dimension i, an affine map from a clipped range [l_i, u_i] onto {0..255}, where the bounds come from a quantile of the observed distribution. Reconstruction error per component is bounded by half a step, (u_i − l_i)/510, so error grows with the clipped range — which is why the quantile matters more than the bit depth.

Product: partition the d dimensions into m subspaces and learn a codebook of k centroids per subspace, typically by k-means. A vector becomes m codes. Distance is computed through precomputed query-to-centroid distance tables, so the query-time cost is table lookups and sums rather than direct arithmetic. Error is the sum of per-subspace quantization errors and is governed by k and m rather than by anything about the individual vector.

Binary: each component maps to one bit by sign or threshold. For normalised vectors, Hamming distance between the bit patterns is a monotone proxy for angular distance in expectation, with variance that falls as d rises. That is the formal reason binary works better in high dimensions: the estimator is the same, but at d = 1536 it has far less variance than at d = 384.

Rescoring turns any of these into a two-stage retrieval: a cheap approximate stage with recall R over an oversampled candidate set of size o·k, then an exact re-rank. End-to-end accuracy is bounded by the recall of the first stage — rescoring cannot recover a document the quantized pass never retrieved — which is exactly what raising oversampling is for.

---

Trade-offs

Scalar against nothing. 4× less memory for a small accuracy loss, most of it recoverable by rescoring, and no codebook to train. This is the default worth reaching for first.

Product against scalar. 16× or more instead of 4×, at the cost of more accuracy loss, a training step, and more CPU per query. Take it when RAM is the binding constraint and CPU isn’t.

Binary against everything. The largest saving and the only one that speeds up distance computation. Requires high dimensionality and a model that survives it. Unusable without measuring; potentially excellent with rescoring.

Rescoring against I/O. It recovers accuracy by reading original vectors. If those live on disk, you have converted a memory problem into a disk-read problem, multiplied by your oversampling factor.

always_ram against total footprint. Keeping quantized vectors pinned in RAM while the originals go to disk is the usual production shape: the fast lossy pass never touches disk, and only the rescore does.

---

Common mistakes

Comparing quantization methods without rescoring. Binary alone and binary with rescoring are different systems. Nearly every disappointing binary-quantization result is the first being judged as though it were the second.

Assuming binary works because a blog post said so. It depends on your model’s dimensionality and training. At 384 dimensions expect a real accuracy cost. Measure it.

Forgetting the graph and payloads. Quantization shrinks vectors. The HNSW graph (Part 3) and your payloads are untouched, so a 32× vector saving is not a 32× collection saving.

Leaving quantile at a value that doesn’t suit your data. A single extreme outlier can consume most of the int8 range and quietly degrade every comparison.

Oversampling without measuring. More candidates means more rescoring work. oversampling is a tunable with a cost, not a free accuracy dial.

Rescoring against vectors you moved to disk, at high oversampling. Each query becomes many random reads. This is the configuration that looks fine in a small test and falls over under concurrency.

Expecting quantization to fix a bad model. It only ever loses information. If retrieval quality is already marginal, compression is not where to look.

---

Interview questions

1. A million vectors at 1,536 dimensions in float32. How much RAM for the vectors?

Answer

1,000,000 × 1,536 × 4 = 6,144,000,000 bytes, about 5.7 GiB — vectors only, before the HNSW graph, payloads and ids. The point of the question is whether you compute it rather than guess, and whether you remember that it’s the *floor*. A strong answer continues into what to do about it: scalar quantization takes it to about 1.4 GiB, binary to about 0.18 GiB, and the graph is unaffected by any of them. **Follow-up:** what else scales with dimensions? Distance computation time, and the quality of binary quantization — which gets better as dimensionality rises.

2. What does scalar quantization actually store?

Answer

One byte per dimension: the original float mapped linearly onto 0–255 across a range derived from the data’s distribution in that dimension, with a quantile applied to clip outliers. Reconstruction error is bounded by half a quantization step, so it grows with the width of the clipped range. That’s why the `quantile` parameter matters — a single extreme value widens the range and degrades every other value’s precision. **Follow-up:** why is it usually the safe default? It’s the least lossy of the three, needs no training step, and its 4× saving is often enough.

3. Explain rescoring, and what it can’t do.

Answer

The quantized vectors retrieve an oversampled shortlist cheaply; the original full-precision vectors are then read for just those candidates and used to re-rank. You get near-exact ordering while doing the expensive comparison only a few dozen times. What it cannot do is recover a document the quantized pass never returned. End-to-end recall is capped by the first stage’s recall, which is precisely why `oversampling` exists — a wider shortlist gives the exact stage more chances to find the right answer. **Follow-up:** what does rescoring cost? Reading the original vectors. If they’re on disk, that’s disk I/O multiplied by the oversampling factor.

4. When would you choose product over scalar quantization?

Answer

When memory is the binding constraint and 4× isn’t enough. Product gets 16× or more by replacing groups of dimensions with codebook ids. You pay in three currencies: more accuracy loss, a codebook training step at build time, and more CPU per query, since distances go through lookup tables rather than direct arithmetic. So it’s the right answer on a memory-bound box with CPU headroom, and the wrong one on a CPU-bound service. **Follow-up:** how would you decide between them in practice? Run Part 4’s recall/latency sweep for both against your own vectors, with rescoring on, and look at where each lands.

5. Why does binary quantization work better at higher dimensions?

Answer

Because the Hamming distance between sign patterns is an estimator of angular distance, and its variance falls as the number of dimensions rises. At 1,536 dimensions you’re averaging over far more bits than at 384, so the estimate is much more stable. Practically: expect binary to be viable on large modern embedding models and to hurt on small ones, and never adopt it without measuring on your own model. **Follow-up:** what makes it fast as well as small? Comparing bit patterns is XOR plus a population count — single CPU instructions — rather than floating-point arithmetic per dimension.

6. Does quantization change recall, latency, or both?

Answer

Both, and not always in the same direction. Recall falls, because information was discarded — how much depends on the method and your data, and rescoring recovers much of it. Latency can go either way. Smaller vectors mean less memory bandwidth per comparison, and binary comparison is genuinely cheaper, so search can get faster. But product quantization adds table lookups, and rescoring adds a second pass. Net effect is a measurement, not a prediction. **Follow-up:** what’s the one thing that reliably improves? Memory. That part is arithmetic and is guaranteed.

7. Your binary-quantized collection has terrible recall. What do you try?

Answer

First check rescoring is on and raise `oversampling` — binary alone is not the configuration anyone should ship, and a too-narrow shortlist caps everything downstream. Then check dimensionality: if the model outputs a few hundred dimensions, binary may simply not be viable, and scalar is the answer. Then check the measurement itself, with Part 4’s list — is the collection actually indexed, are the queries from the right distribution. **Follow-up:** if raising oversampling fixes recall but costs too much latency, what then? Move to scalar quantization: less compression, much less loss, no oversampling needed to be usable.

8. What does always_ram do, and when does it matter?

Answer

It pins the quantized vectors in memory. The common production shape is quantized vectors in RAM with the originals on disk: the hot search path never touches disk, and only rescoring does. It matters as soon as the full-precision vectors don’t fit in RAM, which is the situation quantization exists to address. Without it you can end up paging the thing you compressed in order to make it fast. **Follow-up:** what’s the risk of that shape? Rescoring becomes disk-bound, so the oversampling factor turns directly into random reads per query.

---

Sources

• Qdrant documentation, https://qdrant.tech/documentation/guides/quantization/ — scalar, product and binary quantization, quantile, always_ram, rescore and oversampling.
• The configuration shapes shown here were accepted and read back from Qdrant 1.19.1: ScalarQuantizationConfig(type="int8", quantile, always_ram), ProductQuantizationConfig(compression) with ratios from x4 to x64, and BinaryQuantizationConfig.
• Jégou, H., Douze, M. and Schmid, C., Product Quantization for Nearest Neighbor Search, IEEE TPAMI 2011 — the codebook decomposition that product quantization implements.
• The memory figures throughout are arithmetic (points × dimensions × bytes_per_number) and were not measured.

What to remember

The vectors are the memory bill and quantization is the discount. Scalar gives 4× by storing each number in a byte; product gives 16× or more by replacing groups of numbers with codebook ids; binary gives 32× by keeping one bit per dimension, and is the only one that also speeds up comparison.

The savings are exact arithmetic. The accuracy cost is not — it depends on your model, your dimensionality and your data, and the only honest way to know it is to measure it on your own vectors.

And nearly all of it hinges on rescoring: use the cheap lossy vectors to build a shortlist, then re-rank that shortlist against the originals. Judge any quantization method with rescoring on, because that is how you would actually run it.

Compression is free to calculate and expensive to assume. The bytes you save are arithmetic; the accuracy you lose is a measurement.