Blog

Payloads and Filtering: Indexes, Conditions and What They Cost

Written by

Most real queries are not “find similar” but “find similar, where the tenant is this and the date is after that”. How payloads work, what a payload index is for, and the conditions Qdrant actually supports.

Every part so far has asked one question: find me vectors near this one. Almost no production system asks that question on its own.

Real questions have conditions attached. Find similar documents that this user is allowed to see. Similar products in stock, under £50. Similar incidents from the last 30 days.

The conditions live in the payload, and this part is about making them work. Part 7 is about what happens when they fight the index, which is a hard enough problem to deserve its own part.

Everything here is Qdrant 1.19.1, and the field types and defaults are read from a live collection rather than from documentation.

Try this first

You have ten million documents across a thousand customers, and every search must be scoped to one customer.

Two obvious designs:

  1. Search all ten million by vector, then throw away anything from the wrong customer.
  2. Somehow look only at that customer’s documents in the first place.

Write down what’s wrong with design 1. Then write down what’s wrong with design 2 — because there is something wrong with it too, and it’s less obvious.

The payload

A payload is arbitrary JSON attached to a point. You saw it in Part 2 holding the source text. It also holds everything you want to filter on.

client.upsert("docs", points=[
    models.PointStruct(
        id=1,
        vector=vector,
        payload={
            "tenant": "acme",
            "title": "Q3 planning notes",
            "price": 4200,
            "tags": ["internal", "finance"],
            "published_at": "2026-09-18T10:00:00Z",
            "location": {"lat": 12.97, "lon": 77.59},
        },
    )
])

Two properties worth knowing before you design around it.

Payloads are stored on disk by default. A fresh Qdrant 1.19.1 collection reports on_disk_payload: true. That is a sensible default — payloads are often much larger than vectors and are only read for results you actually return — but it means payload access is a disk read unless you change it. It also means filtering on an unindexed payload field is a disk-bound scan, which is the main reason indexes exist.

Updating a payload doesn’t touch the vector. set_payload changes fields without re-embedding anything, and overwrite_payload replaces the lot. Recall from Part 2 that plain upsert replaces the whole payload, which is how people accidentally wipe fields.

client.set_payload("docs", payload={"price": 3900}, points=[1])          # merge
client.overwrite_payload("docs", payload={"tenant": "acme"}, points=[1]) # replace

Payload indexes

Without an index, answering “which points have tenant = acme” means looking at every point’s payload. With one, Qdrant keeps a structure mapping values to the points that have them, and the answer is a lookup.

client.create_payload_index(
    "docs",
    field_name="tenant",
    field_schema=models.PayloadSchemaType.KEYWORD,
)

Qdrant 1.19.1 accepts these field schemas:

schema for typical condition
KEYWORD exact strings — ids, tenants, categories, tags match
INTEGER whole numbers match, range
FLOAT real numbers range
BOOL true/false match
DATETIME timestamps range
GEO {lat, lon} objects radius, bounding box, polygon
TEXT free text, tokenised for substring and word matching match with text
UUID UUID-valued fields, stored compactly match

Choosing the right one matters. KEYWORD treats the whole string as one atomic value, so it answers “is this exactly acme” very fast and cannot answer “does this contain the word acme”. TEXT tokenises, so it can do the second and is a poor fit for the first.

You can check what a collection has indexed:

info = client.get_collection("docs")
print(info.payload_schema)
# {'tenant': PayloadIndexInfo(data_type=<PayloadSchemaType.KEYWORD>, points=6), ...}

The points count in there is worth watching. It tells you how many points the index actually covers. If it’s far below your point count, most of your data is missing the field entirely, and your filter is excluding points you didn’t mean to exclude.

Conditions

Filters are built from three lists, and the names are the semantics:

  • must — every condition has to hold. This is AND.
  • should — at least one contributes. This is OR.
  • must_not — none may hold. This is NOT.
from qdrant_client import models

flt = models.Filter(
    must=[
        models.FieldCondition(key="tenant", match=models.MatchValue(value="acme")),
        models.FieldCondition(key="price", range=models.Range(gte=1000, lte=5000)),
    ],
    should=[
        models.FieldCondition(key="tags", match=models.MatchAny(any=["finance", "legal"])),
    ],
    must_not=[
        models.FieldCondition(key="status", match=models.MatchValue(value="archived")),
    ],
)

hits = client.query_points("docs", query=vector, limit=10, query_filter=flt).points

The conditions you will use most:

  • MatchValue(value=...) — exact equality.
  • MatchAny(any=[...]) — equality against any of a list. This is IN.
  • MatchExcept(**{"except": [...]}) — everything but those values.
  • Range(gt=, gte=, lt=, lte=) — numeric or datetime bounds, any subset of the four.
  • IsEmptyCondition — the field is missing or null.
  • IsNullCondition — the field is explicitly null.
  • HasIdCondition — restrict to a set of point ids.

Nested fields use dotted keys. key="location.lat" reaches into an object. For arrays of objects, where you need conditions to apply to the same array element rather than any of them, there is a dedicated nested filter — worth reaching for the moment you model anything like line items.

IsEmpty versus IsNull catches people out: a field that was never set is empty; a field explicitly set to null is null. They are different conditions and a filter that uses the wrong one silently matches nothing.

Answering tenant = acme

Schematic — this part runs no lab, so no timings are shown. The answer is the same on both routes; what changes is how many points have to be looked at to produce it, which is why create_payload_index is the highest-value call in this part. Switch scenario to compare.

Filtering and searching are one operation

This is the thing to internalise, and it is why Part 7 exists.

A filtered vector search is not “search, then filter”. It is also not “filter, then search”. Qdrant is given both at once and decides how to satisfy them together — and what it decides depends on how selective the filter is.

You can see the two extremes by thinking about them:

A filter that keeps almost everything (say, status != archived, excluding 1%). Walking the HNSW graph is fine; nearly every neighbour you encounter is eligible, so the search behaves as if the filter weren’t there.

A filter that keeps almost nothing (one tenant out of ten thousand). Now the graph is mostly a trap. You walk it looking for near neighbours and almost everything you find is ineligible. Better to ignore the graph entirely, fetch the tenant’s points from the payload index, and compare all of them directly — which is exactly what full_scan_threshold is for.

Between those extremes is where it gets interesting, and that is Part 7.

For now, the operational point: a filter is not free, and an unindexed filter is expensive. Create the payload index for anything you filter on. It is one call and it changes the strategy available to the planner.

Explain it like I’m ten

Imagine a huge lost property office where everything is arranged by what it looks like — all the black bags together, all the red coats together.

That’s great for “find me something that looks like this”. It’s useless for “find me something handed in last Tuesday”, because Tuesday’s things are scattered all over the room.

So the office also keeps a notebook: a page per day, listing where each item from that day is shelved. That’s the payload index. Now “last Tuesday” is one page-flip instead of a walk around the whole room.

When someone asks for “a black bag handed in last Tuesday”, the clerk has a choice. If hundreds of things came in on Tuesday, walk the black-bag shelves and check tags as you go. If only three things came in on Tuesday, forget the shelves — read those three off the notebook page and look at them.

Where the analogy breaks: the office’s shelves are arranged in a way a person can see and navigate. The vector “shelves” are a graph of remembered neighbours in hundreds of dimensions, so there is no aisle to walk down and no way to glance at a nearby shelf. Which is why the choice between the two strategies is so much sharper than it would be in a real room.

The precise version

A payload index is a secondary index over an attribute, mapping values to the set of point ids carrying them. Qdrant maintains one per field per segment.

Given a filter F with selectivity s = |{p : F(p)}| / N, a filtered k-NN query can be answered by:

  • graph traversal with filtering, where the HNSW search proceeds but ineligible nodes are skipped; the effective branching factor falls roughly with s, so as s → 0 the walk degenerates and may fail to reach eligible regions at all; or
  • retrieve-then-compare, where the payload index produces the eligible id set and exact distances are computed over it; cost is proportional to s·N and it is exact.

The crossover is where s·N falls below the cost of a degraded graph walk. Qdrant exposes this as full_scan_threshold — default 10,000 — meaning that when the filtered cardinality is estimated below that, it takes the second route. Cardinality here is an estimate from the payload index, not a count, which is why the behaviour can surprise you on skewed data.

Boolean structure matters to that estimate: must intersects and lowers cardinality, should unions and raises it, must_not subtracts. A should over a high-cardinality field can push a query out of the cheap regime entirely.

Trade-offs

Indexing a field against write cost. Every payload index is a structure to maintain on every write. Index what you filter on; don’t index everything.

KEYWORD against TEXT. KEYWORD is exact and compact and cannot do substring matching. TEXT tokenises, costs more, and answers a different question. Choosing wrong gives you a filter that silently matches nothing.

On-disk payloads against RAM. The default on_disk_payload: true keeps memory for vectors and the graph. If you filter heavily on unindexed fields, you are paying disk reads for it; the fix is usually an index rather than moving payloads to RAM.

Storing data in the payload against storing a key. A payload is convenient and becomes a second copy of your data, with the consistency problem that implies. A key plus a lookup in your real database is leaner and costs a round trip. Part 8 goes into this properly.

Precise filters against usable ones. Every extra must narrows the candidate set, which sounds good and eventually pushes you into the regime Part 7 is about.

Common mistakes

Filtering on a field with no payload index. It works, and it scans. This is the single most common cause of “filtered search got slow”.

Using KEYWORD and expecting substring matching. KEYWORD matches whole values. Word and substring matching needs TEXT.

Confusing IsEmpty with IsNull. Missing and explicitly-null are different states, and the wrong condition matches nothing while looking correct.

Post-filtering in your application. Fetching 100 results and discarding the ones that don’t match means you asked for the wrong thing: you may end up with fewer than you needed, and you did the work anyway. Pass the filter to the database.

Forgetting should is OR, not “nice to have”. A should clause widens the match set. It does not boost ranking — scores come from the vector, not the filter.

Ignoring the points count on a payload index. If it’s well below your total, most of your data lacks the field and your filters are excluding more than you think.

Assuming a filter makes the search faster. Sometimes it does. Sometimes it makes it much slower, which is Part 7’s entire subject.

Interview questions

1. How does filtering interact with the vector index?

Answer They’re one operation, not two. Qdrant takes the vector and the filter together and chooses a strategy based on how selective the filter is estimated to be. If the filter keeps most points, it walks the HNSW graph and skips ineligible nodes. If the filter keeps very few, it uses the payload index to get those points and compares them directly, which is exact and cheap at small cardinality. The switch point is `full_scan_threshold`, default 10,000. **Follow-up:** why not always pre-filter? Because at low selectivity the eligible set is enormous, and comparing against all of it is the brute-force search you built an index to avoid.

2. What’s the difference between KEYWORD and TEXT?

Answer `KEYWORD` treats the value as one atomic string: fast exact matching on ids, tenants, categories. `TEXT` tokenises the value so you can match words or substrings within it. Using `KEYWORD` for a field you want to search within gives you a filter that matches nothing; using `TEXT` for an identifier wastes space and lets partial matches through. **Follow-up:** which would you use for a tenant id? `KEYWORD` — or `UUID` if it is one, which Qdrant stores more compactly.

3. You added a filter and search got much slower. Why might that be?

Answer Most likely the field has no payload index, so evaluating the condition means reading payloads — and payloads are on disk by default. The other possibility is that the filter is selective enough to make the graph walk inefficient without being selective enough to trigger the direct route: you’re walking a graph in which most neighbours are ineligible. That’s Part 7’s problem, and the fixes there are different. **Follow-up:** how would you tell the two apart? Check `payload_schema` for the index first — it’s one call, and it’s the cheap explanation.

4. must, should and must_not — what do they do to the result set?

Answer `must` is AND: every condition holds. `should` is OR: at least one holds. `must_not` is NOT: none hold. The thing people get wrong is expecting `should` to affect ranking. It doesn’t — it widens which points are eligible. Scores come from vector similarity, and the filter only decides who is allowed to be scored. **Follow-up:** how would you boost rather than filter? You wouldn’t do it with a filter at all. You’d retrieve candidates and re-rank, which is Part 9.

5. Where are payloads stored, and why does it matter?

Answer On disk by default — Qdrant 1.19.1 reports `on_disk_payload: true`. Payloads are often much bigger than vectors and are usually only needed for the results you return, so keeping them out of RAM leaves memory for vectors and the graph. It matters because filtering on an *unindexed* field then means reading payloads from disk for candidate points. With an index, the condition is answered from the index structure instead. **Follow-up:** when would you move payloads to RAM? Rarely, and only after confirming the index route isn’t available — for example a field whose values are too varied to index usefully.

6. How do you filter on a nested field?

Answer Dotted keys for plain objects: `key=”location.lat”`. For arrays of objects it’s different, because you usually need several conditions to match *the same element* rather than any elements. Qdrant has a nested filter for that. Using dotted keys on an array silently gives you “any element matched condition A and any element matched condition B”, which is not the same question and is a bug that produces plausible results. **Follow-up:** what would you do instead if the nesting gets deep? Flatten at write time into fields you can index directly. Filters are cheaper on flat data.

7. Should the source text live in the payload?

Answer Usually yes for convenience, because a vector cannot be turned back into text and you need something to display. It’s the normal pattern, and it costs you a second copy of your corpus to keep consistent. The alternative is storing only a key and fetching from the system of record, which is leaner, always consistent, and costs a round trip per result set. At large scale with frequently changing documents, that trade tilts. **Follow-up:** what belongs in the payload regardless? Anything you filter on — tenant, dates, permissions, status — because those need to be where the filter runs.

8. A filter on tags should match “finance” or “legal”. How do you write it?

Answer `MatchAny(any=[“finance”, “legal”])` as a single condition, which is an `IN` over the array field — Qdrant matches if any array element equals any listed value. Two `should` conditions with `MatchValue` each would also work, but `MatchAny` says the intent more directly and keeps the filter’s structure simpler, which matters because the boolean shape feeds the cardinality estimate that picks the search strategy. **Follow-up:** and to exclude a set? `MatchExcept`, which matches everything except the listed values.

Sources

What to remember

The payload is where everything you filter on lives, and it is on disk by default. A payload index turns a condition from a scan into a lookup, and creating one for every field you filter on is the single highest-value thing in this part.

Filters are must, should and must_not — AND, OR and NOT — and they decide who is eligible to be scored, never how they score.

And filtering is not a step that happens before or after the search. It happens with it, and how selective your filter is changes which algorithm runs.

Index the fields you filter on. Then find out what your filter’s selectivity does to the search, because that is the part nobody warns you about.

How useful was this post?

Click on a heart to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.