RAG similarity threshold: let retrieval come back empty

A developer building a chatbot for a water billing office asked on Stack Overflow why pgvector always returns five rows even when the question has nothing to do with the documents. The query was ORDER BY embedding <=> $1 LIMIT 5, and it did what it says: sort every chunk by distance and hand back the first five. Ask that bot about the weather and it will still pass five chunks about water rates to the model, which will try to make something of them.

Two hours later someone else asked why their Spring AI pipeline answers "How many active users do I have?" but refuses "How many users do I have?". The author lists the two explanations they suspect: the search returned nothing useful for the first wording, or the model refused because its context was weaker. Either way, whether the bot answers is being decided by whatever retrieval happened to return.

I think both questions come from treating retrieval as a step that always succeeds. A pipeline needs a defined empty result, and a definition of what the model does with it. The similarity threshold is the first half of that.

Top-k search has no empty result

Nearest-neighbor search orders candidates by distance, and LIMIT k cuts the list at k whatever the distances are. In SQL the fix is a distance condition next to the ordering, a pattern the pgvector README documents for getting rows within a certain distance:

SELECT id, texto, fuente, 1 - (embedding <=> $1) AS similarity
FROM fragmentos
WHERE embedding <=> $1 < 0.5
ORDER BY embedding <=> $1
LIMIT 5;

<=> is cosine distance, so < 0.5 keeps chunks whose cosine similarity is above 0.5. Whether 0.5 is right depends on the embedding model and the documents. I don't trust thresholds copied from someone else's post, including this one, so here is what the scores looked like when I measured them.

Scores from an eight-document billing corpus

I created a collection with eight short help-center articles for an imaginary SaaS: annual refunds, monthly refunds, cancelling an annual plan, VAT on invoices, seat billing, SSO, data export and API rate limits. Then I ran five questions through AIVAX search with minScore at 0, once without a reranker and once with Reflex, the default. Similarity here is cosine similarity from the embedding model AIVAX uses for every collection, so these numbers say nothing about OpenAI or Cohere embeddings.

Question	Closest document	Similarity	Best Reflex score	Answerable from the corpus
Can I get my money back on a yearly plan after three weeks?	refund-window	0.723	0.623	Yes
how do i stop being charged every year	cancel-annual	0.639	0.518	Yes
How long do you keep my old invoices?	invoice-vat	0.485	0.399	No
Do you offer a student discount?	refund-window	0.391	0.282	No
What is the boiling point of water at high altitude?	refund-window	0.188	0.070	No

The boiling-point question is the easy case. Its best match is below 0.2, the AIVAX default, so nothing from the collection reaches the model. The student discount question worries me more: it sounds like billing, its closest documents are refund articles at 0.39, and it clears the default with room to spare.

The invoice question is a different kind of miss. Its closest document really is about invoices, so 0.485 is a correct retrieval of an article that never says how long invoices are kept. A similarity threshold cannot separate "on topic" from "contains the answer", because embeddings measure only the first.

On this corpus, the lower of the two answerable best matches was 0.639 and the highest unanswerable one was 0.485, a gap of about 0.15. A cutoff of 0.5 sits inside it, 0.015 above the invoice question. On a corpus of a few thousand chunks I would expect that gap to shrink or close, because every additional article is another chance for a near-miss to land close to something.

The threshold also decides less than it seems for the questions that pass it. At 0.5, the yearly refund question still brought back three documents, and one of them was the monthly-refund article at 0.639, which does not apply to an annual plan. Picking the right chunks among those that pass is the reranker's job, and in this run Reflex moved monthly-refund from second to third.

A reranker reorders the pool while the cutoff stays on similarity

Once a reranker is in the pipeline, it is natural to assume the threshold should move to the reranker's score. In AIVAX search and in AI gateways, minScore is applied to vector similarity before reranking. Documents below it never enter the candidate pool, and the reranker reorders whatever is left. Those paths have no second cutoff on the rerank score.

I think that is the right place for it. Similarity is computed for every candidate anyway, and cutting there sends fewer documents to the reranker. Reranker scores also sit on their own scale: Reflex gave the answerable questions 0.62 and 0.52 where similarity said 0.72 and 0.64. A threshold tuned on one does not carry over to the other, which the reranker docs also point out. Reflex did separate the groups a little more clearly in this run, 0.518 against 0.399 for the best scores, but five questions are not enough for me to move the cutoff onto it. If you are still choosing a reranker, I compared Reflex with Cohere, Voyage, Jina, Qwen and NVIDIA and wrote about when a reranker is worth adding at all.

The exception is ranking documents you already hold through POST /api/v1/generations/rerank. There, min_score applies to the reranker's relevance_score and defaults to 0.

An empty retrieval needs an instruction to go with it

To see what the threshold changes end to end, I attached the collection to two AI gateways that differed only in knowledgeBaseMinimumScore, 0.2 and 0.5. Both injected at most three documents into a gpt-5.6-luna prompt, with a one-sentence system instruction: answer only from the retrieved context, and say you don't know otherwise.

I sent each gateway three questions: the yearly refund question, the student discount and the invoice retention one. Both gateways got the refund right, with the prorated refund and the 10% fee. Both declined the other two. The difference was in the prompt. At 0.2, the two unanswerable questions carried three loosely related articles each, for 254 and 256 prompt tokens. At 0.5 they carried nothing, for 51 and 53. The refund question used 261 tokens on both gateways, because its top three documents cleared 0.5 anyway.

With one model and two unanswerable questions, the instruction did the abstaining and the threshold kept irrelevant text out of the prompt. I would not read much into a model behaving well with three short, obviously unrelated articles in front of it. With five chunks of 800 tokens each, the same miss would put about 4,000 tokens of plausible text in the prompt. I did not test that case, and it is the one where I would expect a model to start assembling an answer from neighboring articles.

When nothing passes the threshold, an AIVAX gateway adds no retrieval block to the prompt, and nothing tells the model that a search ran and came back empty. The instruction then carries the whole decision, so write it for that case: what the assistant should say, whether it offers a ticket or a handoff, and whether it may fall back on general knowledge.

What similarity threshold should I use for RAG?

There is no portable number; 0.5 worked on eight short articles and one embedding model. For a new collection, I would derive it with this procedure:

1. Collect real questions from logs or support tickets, and make sure a good share of them are ones the corpus cannot answer, including near-misses that use the domain's vocabulary.
2. Run them with the threshold at 0 and record the similarity of every returned document, marking each one relevant or not.
3. Find the weakest relevant score and the strongest score among unanswerable questions, and put the cutoff in the gap, closer to the unanswerable side if you would rather miss an answer than inject noise.
4. Repeat whenever you change the embedding model or the chunking, because both move every score.

If there is no gap, the threshold can only filter noise, and the decision to answer has to come from the instruction or from a separate check on the retrieved text. Raising the cutoff until the near-misses disappear starts cutting real answers. Those failures are harder to notice than wrong answers, since they show up only as a rising share of "I don't know" replies, so track that share after every change.

On AIVAX, the RAG observability page in the Console lists each search with its original similarity and its reranked score, which is where I would look for questions that keep landing just under the cutoff. When the right documents are a known subset, filters often beat a threshold: pre-filtering by metadata, tags and date narrows the corpus before any score is computed, and a filter that matches nothing returns empty without charging for the query.

The cutoff has a different name on each AIVAX surface

The threshold's field name, range and default depend on how retrieval is called:

• POST /api/v2/query takes minScore, from 0 to 0.99, default 0.2. /api/v2/query/similar accepts the same field.
• AI gateways take knowledgeBaseMinimumScore, from 0 to 1, default 0.2, for both automatic retrieval and the built-in search tool the model can call.
• The collections MCP server reads the X-Mcp-Min-Score header, above 0 and up to 1, default 0.4.
• POST /api/v1/generations/rerank takes min_score on the reranker's score, from 0 to 1, default 0.

A query with a threshold looks like this, and when nothing clears it the response is a normal 200 with an empty result list:

{
  "collections": [ "<collection-id>" ],
  "terms": "Do you offer a student discount?",
  "minScore": 0.5,
  "limit": 5
}

My reading of the higher MCP default is that an agent calling search on its own initiative should get less noise back than an application that inspects every result. Whatever value you choose belongs with the rest of the retrieval pipeline you own, which is most of what separates a RAG system from a vector database.

---

The scores and answers above come from runs against the live AIVAX API on November 10, 2026, using an eight-document test collection that was deleted afterwards. Parameter names, ranges and defaults were checked against the implementation on the same date.