Blog

A Complete System: Retrieval Over a Real Corpus, Measured

Written by

Putting the whole series together — chunking, embedding, indexing, hybrid retrieval, reranking and the evaluation loop that tells you whether any of it worked. The part where the decisions stop being independent.

Thirteen parts of decisions, each considered on its own. This one puts them in a line and runs data through them, because the decisions interact and the interactions are where systems go wrong.

The through-line: retrieval quality is not a property of your vector database. Qdrant is one stage of a pipeline where the stages before it — chunking, embedding — and after it — reranking, generation — matter at least as much. The database’s job is to not be the bottleneck and to give you the instrumentation to find out which stage is.

Try this first

Your RAG system gives a wrong answer.

List every stage that could be responsible, in the order you’d check them.

Most people start at the vector search. It is almost never first in the list, and knowing why is the point of this part.

The pipeline

documents
   ↓  chunk            ← decides what a "result" even is
   ↓  embed            ← decides what similar means
   ↓  index            ← Parts 2–5: collection, HNSW, quantization
   ↓  retrieve         ← Parts 6–9: filter, hybrid, fusion
   ↓  rerank           ← Part 9: cross-encoder over candidates
   ↓  assemble context ← what actually reaches the model
   ↓  generate

Every arrow can lose the answer, and only one of them is the database.

Where the answer got lost chunk embed index retrieve rerank assemble generate

Schematic — the method, not a measurement. The point is that five of the seven stages can lose the answer without the vector database doing anything wrong, and that they are indistinguishable from the outside. Only per-stage instrumentation tells them apart.

Building it

Chunk, with the parent kept

def chunk(doc_id, text, target_words=200, overlap_words=40):
    """Split on paragraphs, accumulate to a target size, overlap slightly."""
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, current = [], []
    for para in paragraphs:
        current.append(para)
        if sum(len(c.split()) for c in current) >= target_words:
            chunks.append(" ".join(current))
            tail = " ".join(current).split()[-overlap_words:]
            current = [" ".join(tail)]
    if current:
        chunks.append(" ".join(current))
    return [{"doc_id": doc_id, "chunk_index": i, "text": c} for i, c in enumerate(chunks)]

Paragraph boundaries rather than a fixed character count, because a chunk spanning two topics embeds as the average of both and matches neither (Part 8). doc_id goes in the payload so you can group results by document or fetch the whole thing to show.

Index, with everything the later parts need

client.create_collection(
    "kb_v1",
    vectors_config={"dense": models.VectorParams(size=384, distance=models.Distance.COSINE)},
    sparse_vectors_config={"lex": models.SparseVectorParams()},
    hnsw_config=models.HnswConfigDiff(m=16, ef_construct=200, payload_m=16),
    quantization_config=models.ScalarQuantization(
        scalar=models.ScalarQuantizationConfig(
            type=models.ScalarType.INT8, quantile=0.99, always_ram=True)
    ),
    optimizers_config=models.OptimizersConfigDiff(indexing_threshold=0),   # bulk load
)
for field, schema in (("tenant", models.PayloadSchemaType.KEYWORD),
                      ("doc_id", models.PayloadSchemaType.KEYWORD),
                      ("updated_at", models.PayloadSchemaType.DATETIME)):
    client.create_payload_index("kb_v1", field_name=field, field_schema=schema)

Every choice there came from an earlier part: named plus sparse vectors for hybrid (Part 9), payload_m for tenant filtering (Part 7), scalar quantization pinned in RAM (Part 5), a generous ef_construct because it’s free at query time (Part 3), payload indexes for everything filtered on (Part 6), and indexing disabled for the load (Part 10).

Then load, restore indexing_threshold, wait for the index, and point an alias at it (Part 11) so you can replace it later.

Retrieve

def retrieve(query_text, tenant, limit=50):
    dense = embed_dense(query_text)
    sparse = embed_sparse(query_text)
    tenant_filter = models.Filter(must=[
        models.FieldCondition(key="tenant", match=models.MatchValue(value=tenant)),
    ])
    return client.query_points(
        "kb",                                   # the alias, never the collection
        prefetch=[
            models.Prefetch(query=dense, using="dense", limit=limit,
                            filter=tenant_filter),
            models.Prefetch(query=sparse, using="lex", limit=limit,
                            filter=tenant_filter),
        ],
        query=models.FusionQuery(fusion=models.Fusion.RRF),
        limit=limit,
        query_filter=tenant_filter,
        search_params=models.SearchParams(
            hnsw_ef=128,
            quantization=models.QuantizationSearchParams(rescore=True, oversampling=2.0),
        ),
        with_payload=True,
    ).points

Note limit=50 rather than 5. With a reranker downstream, retrieval’s job is to not miss the answer, so you retrieve wide and let the reranker narrow. That single change is often worth more than any index tuning.

The evaluation loop

Here is the part that makes the rest improvable, and the part almost everyone skips.

You cannot tune a pipeline you cannot measure, and the measurement has to be end to end, because the stages interact. A chunking change alters what recall even means. A reranker hides a retrieval problem until retrieval degrades past its reach.

Build a query set with known answers

Thirty to a hundred real questions with the document (or chunk) that should answer each. Sources: your support tickets, your search logs, your own team writing down what they actually ask.

Deliberately include the hard cases from across this series:

  • paraphrases with no shared vocabulary — dense search’s strength;
  • exact identifiers and error codes — Part 9’s sparse case;
  • questions whose answer is spread across two chunks — the chunking case;
  • questions with a filter attached — Part 7’s case.

A query set of only easy questions tells you nothing, because everything passes.

Measure each stage, not just the end

def evaluate(query_set, tenant="acme", k_retrieve=50, k_final=5):
    stats = {"retrieved": 0, "reranked": 0, "n": len(query_set)}
    for q in query_set:
        candidates = retrieve(q["question"], tenant, limit=k_retrieve)
        ids = [c.payload["doc_id"] for c in candidates]
        if q["answer_doc_id"] in ids:
            stats["retrieved"] += 1                     # did retrieval find it at all?
        top = rerank(q["question"], candidates)[:k_final]
        if q["answer_doc_id"] in [t.payload["doc_id"] for t in top]:
            stats["reranked"] += 1                      # did it survive to the final set?
    return {
        "recall_at_retrieve": stats["retrieved"] / stats["n"],
        "recall_at_final": stats["reranked"] / stats["n"],
    }

Two numbers, and the gap between them is the diagnosis:

  • Low recall_at_retrieve — the answer never came back. The problem is upstream: chunking, embedding, the filter, or the index. No reranker can fix it.
  • High retrieve, low final — retrieval found it and reranking dropped it. The problem is the reranker or the final k.
  • Both high, bad answers — retrieval is fine and the problem is downstream, in how you assemble context or in the generation itself.

That decomposition is why you measure per stage. A single end-to-end score tells you something is wrong and not where.

The order to check things

Back to the opening question. When a RAG answer is wrong, this is the order:

  1. Was the answer in the corpus at all? Astonishingly often it isn’t, and everything downstream is blameless.
  2. Was it in a single chunk? If the answer spans a boundary, no retrieval strategy assembles it. This is a chunking bug and it looks like a search bug.
  3. Did retrieval return it in the top 50? If not, the problem is upstream of ranking.
  4. Did reranking keep it? If not, that’s the reranker.
  5. Did it reach the model? Context assembly truncates, and the thing you truncate is often the answer.
  6. Did the model use it? Now, finally, it’s a generation problem.

Vector search parameters — ef, m, quantization — only enter at step 3, and only after Part 4’s measurement says the index is losing things. Most retrieval problems are chunking problems wearing a costume.

Explain it like I’m ten

Imagine asking a librarian a question and getting a bad answer.

Before blaming the librarian, check a few things. Does the library even own a book with that answer? Is the answer split across two books, so no single one contains it? Did she bring back twenty books including the right one, and then pick the wrong five? Did she hand you the right book but open it at the wrong page?

Each of those is a different problem with a different fix, and they all look identical from where you’re standing: you asked, and the answer was wrong.

So you write down thirty questions you already know the answers to, and check where each one goes wrong. Now you can tell which part to fix — and, more usefully, whether a change actually helped, instead of guessing.

Where the analogy breaks: you can ask a librarian why she chose those books. The pipeline can’t explain itself. The only way to know which stage failed is to instrument each one and look — which is why the evaluation loop isn’t optional tidiness, it’s the only sense organ the system has.

The precise version

The pipeline is a composition of stages fchunk ∘ fembed ∘ fretrieve ∘ frerank ∘ fgenerate, and end-to-end accuracy is bounded above by every stage’s own accuracy. In particular:

Retrieval recall at kretrieve is a hard ceiling on everything downstream. A reranker reorders a candidate set; it cannot introduce a document that isn’t in it. So the quantity to optimise in retrieval is recall@50 or recall@100, not recall@10 — a distinction that changes which index parameters matter.

Chunking sets the ceiling above that. If the evidence for a question spans chunks ci and ci+1 and neither alone entails the answer, no retrieval function over single chunks can return a sufficient result. That is a property of the representation, not of the search, and no amount of ef addresses it.

Note also that these measurements are not the index recall of Part 4. Part 4’s recall compares the index against exact search over the same vectors — it measures the index. This part’s compares the pipeline against human-labelled answers — it measures the system. Index recall of 1.0 is perfectly compatible with system recall of 0.4, and Part 2’s write-ahead-log failure is exactly that case: flawless index, wrong answer.

Trade-offs

Chunk size. Larger means fewer points, less memory, blurrier matches, more context per hit. Smaller means sharper matches, more points, and results that may lack the context to be useful.

Retrieve wide against latency. A bigger candidate set raises the ceiling for reranking and costs retrieval time and reranking time, the latter linear in candidates.

Reranking against not. Usually the largest single quality gain available, at a real latency cost and an extra model to operate.

Hybrid against dense alone. A second index to build and maintain, for a large improvement on identifiers and rare terms.

Evaluation effort against everything else. Building a labelled query set is a day of unglamorous work that makes every subsequent decision measurable instead of speculative. It is the highest-return day in the project.

Common mistakes

Tuning the vector index first. It’s step 3 of 6 and rarely the problem. Check the corpus and the chunking first.

Retrieving 5 candidates and reranking 5. The reranker can only reorder what it’s given. The point of a funnel is that the wide end is wide.

Optimising recall@10 with a reranker downstream. The relevant metric becomes recall@50 or recall@100 — don’t miss it, rather than rank it first.

Measuring only end to end. You learn that something is wrong and not which stage. Measure per stage and read the gaps.

A query set of easy questions. Everything passes and you learn nothing. Include the cases you know are hard.

Confusing index recall with system recall. Part 4’s number says the index matches exact search. It says nothing about whether the results answer the question.

Changing chunking without re-measuring everything. It alters what a result is, so every downstream number changes meaning.

Never re-running evaluation. Corpora drift, models get upgraded, and quality degrades silently. It’s a regression test, not a one-off.

Interview questions

1. A RAG system gives a wrong answer. What do you check, in order?

Answer Is the answer in the corpus at all. Is it within a single chunk. Did retrieval return it in the candidate set. Did reranking keep it. Did it survive context assembly. Did the model use it. Vector index parameters enter at step three, and only if measurement says the index is losing things. Most retrieval problems are chunking problems — an answer split across a boundary cannot be retrieved by any strategy over single chunks. **Follow-up:** how do you find out which stage without guessing? Instrument each one against a labelled query set and look at where the answer disappears.

2. Why optimise recall@50 rather than recall@10?

Answer Because with a reranker downstream, retrieval’s job is to not miss the answer rather than to rank it first. The reranker reorders the candidate set and cannot add to it, so retrieval recall at the candidate size is a hard ceiling on the whole system. That changes which index settings matter: you want a configuration that reliably includes the answer somewhere in fifty, not one that nails the top ten. **Follow-up:** what’s the cost of going wider? Retrieval cost, and reranking cost that’s linear per candidate with a large constant. A few hundred is the usual interactive ceiling.

3. What’s the difference between index recall and system recall?

Answer Index recall — Part 4 — compares the index’s results against exact search over the same vectors. It measures approximation error in the index alone. System recall compares the pipeline’s results against human-labelled correct answers. It measures whether the thing works. They’re independent. Index recall of 1.0 with system recall of 0.4 is entirely possible, and Part 2 has the example: the index returned exactly the right nearest neighbours, and the top result was about register allocation instead of write-ahead logging. **Follow-up:** which do you optimise? System recall. Index recall is a diagnostic that tells you whether the index is the stage at fault.

4. How would you build an evaluation set from nothing?

Answer Thirty to a hundred real questions with the document that should answer each. Take them from support tickets, search logs, or the team writing down what they actually ask — real phrasing matters, because synthetic questions are phrased like the documents and are therefore too easy. Deliberately include hard cases: paraphrases with no shared words, exact identifiers, answers spanning two chunks, and queries with filters. A set of easy questions passes everything and teaches nothing. **Follow-up:** isn’t a hundred too few to be statistically meaningful? For fine distinctions, yes. For catching the large regressions that actually happen, it’s enough — and it’s a hundred times better than nothing, which is the real alternative.

5. Retrieval finds the answer in the top 50 but the final answer is wrong. Where’s the problem?

Answer Downstream of retrieval, and the two-number measurement localises it: if `recall_at_final` is much lower than `recall_at_retrieve`, reranking is dropping it. If both are high and answers are still wrong, the problem is context assembly or generation. Context assembly is the underrated one — truncating to a token budget frequently removes exactly the chunk that mattered, and nothing logs that. **Follow-up:** how would you check context assembly specifically? Log what actually reaches the model, for a sample of queries, and check whether the answer chunk is in it.

6. Which decision in this pipeline has the biggest effect on quality?

Answer Chunking, usually — and it’s the one with no setting in the database. It decides what a result is, whether an answer can be retrieved at all, and what “similar” is being computed over. After that, adding a reranker, then hybrid retrieval. Index parameters like `ef` and `m` matter least of the list, which is the opposite of the order most people work in. **Follow-up:** why do people start with the index? Because it has knobs and the knobs have documentation. Chunking has neither, so it looks like it isn’t a decision.

7. How does this change if the corpus updates constantly?

Answer Two things. Your evaluation set decays — the labelled answer document may be superseded or deleted, so it needs periodic review or it starts reporting failures that are actually corpus changes. And ground truth for index recall (Part 4) expires whenever the corpus changes, so it has to be regenerated rather than reused. A stale ground truth looks exactly like a recall regression. **Follow-up:** what about re-embedding when the model changes? That’s a full rebuild — Part 11’s alias swap, with the evaluation set run against the new collection before the swap rather than after.

8. What would you put in place on day one of a new RAG project?

Answer An alias in front of the collection, so any rebuild is a pointer move. A labelled query set, even a small one, so every decision afterwards is measurable. Per-stage instrumentation, so you can tell which stage lost the answer. And the memory arithmetic, so you know what the thing will cost at target scale. None of those is about search quality directly, and all of them determine whether you can *improve* search quality later. **Follow-up:** what would you deliberately not do on day one? Tune index parameters. You have no measurement yet, so any change is a guess, and the defaults are reasonable.

Sources

  • The pipeline configuration shown draws on the whole series: Part 3 for m and ef_construct, Part 5 for quantization and rescoring, Part 6 for payload indexes, Part 7 for payload_m, Part 9 for prefetch and RRF, Part 10 for the bulk-load pattern, and Part 11 for the alias.
  • Qdrant documentation, https://qdrant.tech/documentation/concepts/hybrid-queries/ for the Query API used in retrieve.
  • Lewis, P. et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401) — the RAG formulation this pipeline implements.
  • The evaluation decomposition is the standard staged-recall argument; the specific failure it is illustrated with comes from Part 2’s own measured result.

What to remember

The pipeline is chunk, embed, index, retrieve, rerank, assemble, generate. Every stage can lose the answer and only one of them is the vector database.

Retrieve wide and rerank, because retrieval’s job with a reranker downstream is to not miss the answer rather than to rank it first. And measure per stage, because a single end-to-end number tells you something is broken without telling you what.

Above all: index recall and system recall are different things. A perfect index can return perfectly wrong answers, and the only thing that distinguishes the two is a set of questions whose answers you already know.

The database’s job is to not be the bottleneck and to let you prove which stage is. Most retrieval problems are chunking problems wearing a costume.

How useful was this post?

Click on a heart to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.