Blog

Recall Is a Measurement, Not a Setting

Written by

How to find out whether your index is actually returning the right answers — building ground truth, computing recall@k, and the four ways a benchmark quietly lies to you. The methodology part of the series, and the one the others depend on.

Part 3 ended on a promise: every HNSW parameter buys recall with a different currency, and what it costs on your data is something you measure.

This part is that measurement. It is the most useful thing in the series and the least glamorous, because there is nothing to configure — you are building a ruler, and then checking that the ruler is straight.

There are no timings in this part. That’s deliberate: a number measured on my laptop tells you nothing about your cluster, and the point here is the procedure, which transfers exactly.

Try this first

Someone hands you a vector database and says “recall is 0.95”.

Write down three questions you would ask before believing it.

If your three questions are about ef and m and the corpus size, read on — those are the least important. The ones that decide whether the number means anything are elsewhere.

What recall actually is

Pick a query. Ask the index for the best k results. Separately, work out the true best k by comparing the query against every vector. Recall@k is the fraction of the true set that the index actually returned.

def recall_at_k(truth, got, k):
    """truth and got are lists of id-lists, one per query."""
    hits = [len(set(t[:k]) & set(g[:k])) / k for t, g in zip(truth, got)]
    return sum(hits) / len(hits)

That’s the whole definition, and three things follow from it immediately.

It is a set overlap, not an ordering. If the index returns the right ten documents in the wrong order, recall@10 is 1.0. If you care about order — and for anything a human reads, you do — recall is not the metric you want; it is the metric that tells you the candidates were right. Ranking quality is a separate question, and Part 9 is where it matters.

It needs ground truth, and ground truth is expensive. The only way to know the true nearest neighbours is to compute all the distances. That is exactly the operation you built an index to avoid, which is why you do it once, offline, on a sample, and store the answer.

It is an average over a query set. So the query set is part of the measurement. Change it and the number changes, with no code change anywhere.

Building ground truth

Qdrant will do this for you. exact=True bypasses the index and scans:

from qdrant_client import models

truth = []
for q in queries:
    hits = client.query_points(
        "notes", query=q.tolist(), limit=K,
        search_params=models.SearchParams(exact=True),   # no index, no approximation
        with_payload=False,
    ).points
    truth.append([h.id for h in hits])

Two habits make this trustworthy.

Verify the ground truth against a second implementation. Compute the same thing in numpy and check the two agree completely. If your own exact search and Qdrant’s exact search disagree, the bug is in your comparison — an id mapping, a normalisation, a slice — and every recall number you derive from it is fiction. This check costs a few lines and it is the one that catches the errors that silently invalidate everything:

import numpy as np

# corpus is (n, dim), already normalised; row index == point id
scores = corpus @ q
mine = np.argsort(-scores)[:K].tolist()

assert set(mine) == set(truth[i]), "ground truth disagrees with itself — stop here"

Store it. Ground truth is a function of the corpus and the query set, not of any index setting. Compute it once, write it to a file, and reuse it across every parameter you try. If you recompute it inside the loop you will be measuring your exact search as much as your index.

When exact search is too slow to be ground truth

At ten million vectors a full scan per query is painful, and you need hundreds of queries. Two standard ways out:

  • Sample the corpus. Build ground truth on a random million-vector subset and measure recall there. It answers a slightly different question — recall on a smaller index — but the shape of the recall/latency curve transfers well enough to choose parameters.
  • Use a dataset that ships ground truth. The public ANN benchmark sets come with the true neighbours precomputed. Convenient, and the trap is in the next section: those vectors are not yours.

The four ways a benchmark lies

Every one of these produces a number that looks fine and means nothing. Three of them have appeared in this series’ own labs.

1. The query set comes from the wrong distribution

Queries must look like the queries your system will actually get. The usual way to get that is to hold out part of your data: embed real user queries if you have them, or take real documents out of the corpus before indexing and use them as queries.

Draw queries from somewhere else and you measure something unrelated. Part 1’s lab drew uniform-random query vectors against a clustered corpus and reported recall@10 of 0.35 for a perfectly healthy index — a query sitting nowhere near any cluster gives the greedy walk no gradient to follow. The index was fine; the question was nonsense.

This is why every serious benchmark dataset ships a query set rather than telling you to generate one.

2. The collection isn’t actually indexed

Below indexing_threshold (default 10,000 per segment) Qdrant builds no HNSW graph, and below full_scan_threshold (also 10,000) it won’t use one it has. Either way you get brute-force results at recall 1.0, and ef does nothing.

info = client.get_collection("notes")
assert info.indexed_vectors_count == info.points_count, (
    f"only {info.indexed_vectors_count:,} of {info.points_count:,} vectors are indexed"
)

A green collection is not a fully indexed one. Check the count, not the status.

3. The data has the wrong shape

Recall is a property of your embeddings as much as of your parameters. The same index at the same ef gave Part 1 recall of about 0.52 on structureless random vectors and 1.0 on tidy clusters. Both were “correct” measurements of entirely different things.

If you benchmark on synthetic vectors, you are measuring your generator. If you benchmark on a public dataset of image descriptors and then deploy text embeddings from a different model at a different dimensionality, you have measured someone else’s geometry. Use your own vectors, from your own model. It is the only corpus whose answer applies to you.

4. The comparison changes two things at once

If setting A differs from setting B in ef and in corpus size, or in m and in which machine ran it, the difference tells you nothing about either. Change one thing. This is ordinary experimental discipline and it is violated constantly, usually by re-running a benchmark after an unrelated deploy.

The curve, not the number

“Recall is 0.95” is not a useful statement, because recall without latency is free — set ef to your corpus size and recall is 1.0.

What you actually want is the trade-off curve: recall and latency together, as ef sweeps. Then you pick the recall your product needs and read off what it costs.

Recall and cost as ef rises all of them none recall ef — how much of the graph each query explores → smaller larger recall cost per query the knee

Schematic. This part runs no lab, so nothing here is a measured value and no number is printed on the curve — the shape is a property of the algorithm, while where the knee falls is a property of your corpus and has to be measured. Switch scenario for what the same sweep looks like when the collection is too small to be indexed, which is the most common way a recall benchmark lies.

rows = []
for ef in (16, 32, 64, 128, 256, 512):
    got, timings = [], []
    for q in queries:
        start = time.perf_counter()
        hits = client.query_points(
            "notes", query=q.tolist(), limit=K,
            search_params=models.SearchParams(hnsw_ef=ef), with_payload=False,
        ).points
        timings.append(time.perf_counter() - start)
        got.append([h.id for h in hits])
    timings.sort()
    rows.append({
        "ef": ef,
        "recall": recall_at_k(truth, got, K),
        "p50_s": timings[len(timings) // 2],
        "p95_s": timings[int(len(timings) * 0.95)],
    })

The shape this produces is always the same, and knowing the shape is most of the value:

  • Recall rises steeply at first. Going from a very small ef to a moderate one buys a lot.
  • Then it flattens. Past some point you are paying linearly for almost nothing.
  • Latency rises roughly with ef throughout.

The interesting region is the knee — the last ef before the curve goes flat. Below it you are giving up accuracy cheaply available; above it you are buying accuracy at a terrible exchange rate.

Where the knee sits depends on your data, which is the entire reason you measure rather than copy a number out of a blog post.

Measuring latency without fooling yourself

This is where Part 1 went wrong in public, so it is worth stating the rules plainly.

Warm up first. The first queries against a fresh collection pay for caches that later queries don’t. Discard them.

Quote a percentile, not a mean and never a maximum. The mean hides the tail that users actually complain about. The maximum is the worst of N tries, not a bound — run it again and you get a different one. p50 and p95 or p99, with the number of queries stated.

Repeat the whole thing. Not just more queries — more runs, ideally interleaved with the thing you’re comparing against, so that a machine getting busier halfway through hits both arms equally.

Don’t quote a difference between two noisy numbers. If you measure the round trip and subtract it to get “search time”, check that the difference is much larger than the spread across runs. If it isn’t, you have no number. Part 1 claimed a sub-millisecond search time this way; repeating the measurement gave answers that differed threefold, and one run made the “floor” slower than the thing it was supposedly bounding.

Say what it ran on. Cores, RAM, whether the client and server shared a machine, and whether anything else was running. Absolute latencies do not travel between machines. Shapes do.

Explain it like I’m ten

Imagine a spelling test with a hundred words, and the answer sheet is locked in a drawer.

Your friend takes the test quickly and gets 95 right. To know that, someone had to open the drawer and check — that’s ground truth, and getting it means doing all the work properly at least once.

Now: was 95 good? It depends. If your friend spent an hour, maybe not. If they spent a minute, that’s excellent. The score alone doesn’t tell you anything without the time, and the time doesn’t tell you anything without the score. You need both, together.

And one more thing: if the test words were all easy ones, 95 means less than it looks. Whoever chose the words decided most of the result before anyone sat down.

Where the analogy breaks: a spelling test has one right answer per word, fixed forever. A nearest-neighbour answer depends entirely on what else is in the corpus — add a million documents and the true top ten for the same query changes. Ground truth is not a permanent fact about a query; it is a fact about a query and a corpus, and it expires when the corpus does.

The precise version

Let C be the corpus, Q a held-out query set, and Nk(q) the true k-nearest neighbours of q under the collection’s distance metric. For an index returning Rk(q):

recall@k = (1/|Q|) · Σ_q |N_k(q) ∩ R_k(q)| / k

This is set recall; it is insensitive to the order within the returned set, and since |Rk| = k it coincides with precision@k, which is why nobody quotes precision here.

The estimate has sampling error from the finite query set. With |Q| queries and k results, the mean is over |Q|·k binary outcomes, but they are correlated within a query, so treating them as independent understates the error. Practically: report |Q|, and be suspicious of differences of a couple of points between settings when |Q| is in the tens.

A separate, non-obvious source of variance: HNSW construction is multithreaded and not deterministic. Build the same data twice with the same parameters and you get slightly different graphs and slightly different recall. Comparing two settings on single builds confounds the setting with the build. Build each more than once.

Trade-offs

Ground truth quality against cost. Exact search over the full corpus is the real answer and is expensive. A sampled subset is cheap and answers a slightly different question. Pick deliberately, and say which you did.

Query set size against confidence. More queries is a tighter estimate and a slower loop. A few hundred held-out queries is the usual working point; a dozen tells you the shape and nothing finer.

Your own corpus against a public benchmark. Yours is the only one whose answer applies to you. A public one is comparable with published numbers and costs no embedding time. If you use a public set, don’t transfer its parameter choices to your data.

Recall against ranking quality. Recall says the right candidates came back. It says nothing about the order, and nothing about whether the “right” candidates are actually good answers — that’s a retrieval-quality question needing labelled relevance, not just nearest neighbours.

Common mistakes

Benchmarking below the thresholds. Under 10,000 vectors per segment there is no graph and no approximation. Recall 1.0, ef irrelevant, conclusions worthless.

Recomputing ground truth inside the parameter loop. Slow, and it makes every row depend on the exact search rather than on the setting you’re varying.

Comparing single builds. HNSW’s build is non-deterministic. A two-point recall difference between one build at m=16 and one at m=32 may be entirely the build.

Quoting recall without latency. Free to make it 1.0. The pair is the result.

Using the maximum latency as a bound. It’s the worst of N tries. Use p95 or p99 and say how many queries.

Reusing ground truth after the corpus changed. It expires the moment you add or delete points. Regenerate it, or you are scoring against yesterday’s answers.

Trusting a recall number you can’t reproduce twice. Run the whole measurement again before you act on it. This is the cheapest possible check and it catches most of the above.

Interview questions

1. What is recall@10, precisely?

Answer For each query, the fraction of the true 10 nearest neighbours that the index returned, averaged over a query set. It is a set overlap, so order within the ten doesn’t affect it. Because the index returns exactly 10, recall@10 equals precision@10 here, which is why only recall gets quoted. **Follow-up:** how do you get the true 10? Exact search — compare the query against every vector. In Qdrant that’s `search_params=SearchParams(exact=True)`, done once and stored.

2. Your benchmark says recall 1.0 at every ef. What’s wrong?

Answer Almost certainly no index is being used. Under `indexing_threshold` (default 10,000 per segment) no graph is built, and under `full_scan_threshold` an existing graph isn’t used — either way every query is brute force, which is exact by definition. Check `indexed_vectors_count` against `points_count`. The other candidate is a corpus so small or so cleanly clustered that the search cannot go wrong, which is a real result but won’t survive contact with production data. **Follow-up:** how would you make the benchmark meaningful? Get above the thresholds with real vectors from your own model, and use held-out real queries.

3. Why not just use a public ANN benchmark dataset?

Answer They’re excellent for comparing *algorithms*, because everyone runs the same data and the ground truth ships with it. They’re poor for choosing *your* parameters, because recall depends on the geometry of the embeddings — dimensionality, how clustered they are, intrinsic dimensionality — and a public set of image descriptors has different geometry from your text embeddings. Use them to sanity-check that your harness is sane, then measure on your own vectors to pick settings. **Follow-up:** what if you have no production queries yet? Hold out real documents and use them as queries. It’s imperfect — real queries are shorter and phrased differently — but it’s far closer than random vectors.

4. How would you compare m=16 against m=32 fairly?

Answer Same corpus, same query set, same ground truth, same machine, one variable changed. Then **build each configuration more than once**, because HNSW’s construction is multithreaded and non-deterministic, so a single build of each confounds the parameter with the build. Report the recall/latency curve for each across an `ef` sweep, not a single point — `m=32` may look worse at one `ef` and better across the useful range. **Follow-up:** what else differs between them that you should report? Memory. Doubling `m` roughly doubles the graph, and build time goes up too.

5. Recall went down after a deploy. How do you find out why?

Answer First establish that the measurement is comparable: same query set, same ground truth, and ground truth regenerated if the corpus changed — stale ground truth after an ingest looks exactly like a recall regression. Then check the collection: is it fully indexed, did segment layout change, did anyone alter `ef` at the call site, did the embedding model version change (which invalidates everything). Only then look at index parameters. **Follow-up:** which of those is most common? A corpus change with stale ground truth, and an embedding model change. Both are upstream of the database.

6. Is recall the right metric for a RAG system?

Answer It’s necessary and not sufficient. Recall tells you the vector search returned the same candidates exact search would — that the *index* is working. It cannot tell you whether those candidates answer the user’s question, because the true nearest neighbours may all be irrelevant. [Part 2](/qdrant-first-collection/) has an example: a query about surviving power loss returned a sentence about CPU register allocation as its top hit, with perfect recall. The index did its job flawlessly and the answer was wrong. For end-to-end quality you need labelled relevance judgements and a metric like nDCG or answer accuracy, which is Part 14. **Follow-up:** so when is recall the metric you want? When you’re tuning the index, and you want to separate index error from retrieval-approach error.

7. How many queries should a recall benchmark use?

Answer Enough that the differences you care about are bigger than the noise. A few hundred held-out queries is a normal working point; a dozen shows you the shape and supports no fine comparisons. Report the count alongside the number, always. “Recall 0.94” and “recall 0.94 over 12 queries” are very different claims, and the second is honest about being an anecdote. **Follow-up:** how would you know if you have enough? Split the query set in half and compare the two halves. If they disagree by more than the effect you’re chasing, you need more.

8. What would make you distrust a recall number immediately?

Answer No latency beside it; no query count; no statement of where the queries came from; recall exactly 1.0 across a whole sweep; or a number that can’t be reproduced by re-running the measurement. Also a comparison where two things changed, and any latency quoted as a maximum or as a difference between two measurements without a spread. **Follow-up:** what’s the single cheapest check? Run it twice. Most bad benchmarks don’t survive being repeated.

Sources

  • Malkov, Y. A. and Yashunin, D. A., Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs (arXiv:1603.09320) — the recall/latency framing used throughout, and the parameters being swept.
  • Qdrant documentation, https://qdrant.tech/documentation/concepts/search/ — exact, hnsw_ef, and https://qdrant.tech/documentation/concepts/indexing/ for indexing_threshold and full_scan_threshold. The default of 10,000 for both was read from a collection on Qdrant 1.19.1.
  • This series’ own failures, which supply most of the examples above: Part 1 on query distribution, data shape, and the difference of two noisy latencies.

What to remember

Recall is the fraction of the true nearest neighbours your index actually returned. Computing it needs ground truth, ground truth needs exact search, and exact search is the thing you bought an index to avoid — so you do it once, offline, and store it.

The number on its own is meaningless. Recall without latency is free; latency without recall is free. The result is the curve, and the interesting point on it is the knee.

And most bad recall numbers aren’t wrong about the index at all. They’re wrong about the query set, the indexing state, the shape of the data, or the fact that two things changed at once.

Before you tune anything, make the ruler and check it’s straight. A benchmark you haven’t tried to break is a guess with decimal places.

How useful was this post?

Click on a heart to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.