Rerankers Explained: The Cheapest Fix for Bad RAG
By Ergini, Software & AI Developer
TL;DR
If your RAG system retrieves the right document at rank 12 but your prompt only sees the top 5, the problem is ranking, not embeddings, and a reranker fixes it for a fraction of the effort of changing models. A reranker is a cross-encoder: it reads the query and each candidate document together rather than comparing two independently-computed vectors, which is far more accurate and far too slow to run over a whole corpus. So you retrieve the top 50 to 100 with embeddings and rerank those. Expect a meaningful jump in ranking quality, roughly 100 to 300ms of added latency, and about an hour of integration work.
The symptom that means you need this
Your RAG system answers most questions well and then, on a question you know the answer exists for, produces something vague or wrong. You dig in, and the correct chunk is sitting at rank 12. Your prompt only includes the top 5. Retrieval found it. Ranking buried it.
That specific failure is what a reranker exists to fix, and it is worth recognising because the instinctive response to bad RAG results is to change the embedding model, which in this case would be an expensive migration that does not address the problem. Measure recall at 5 and recall at 50 first. A large gap between them is a ranking problem. A low recall at 50 is a retrieval or chunking problem, and no reranker will rescue it.
What a reranker is
A reranker is a cross-encoder model that takes a query and a candidate document together as one input and returns a single relevance score. It reads both at once, so it can attend to the relationship between the specific words in the query and the specific words in the document, rather than comparing two summaries produced in isolation.
The contrast with embeddings is the whole point. An embedding model is a bi-encoder: it turns the query into a vector and each document into a vector separately, and then compares them. Because documents are encoded without knowing the query, their vectors can be computed in advance and stored in an index, which is what makes searching millions of documents possible in milliseconds. A cross-encoder can precompute nothing, because it needs the pair.
| Bi-encoder (embeddings) | Cross-encoder (reranker) | |
|---|---|---|
| Input | Query and document encoded separately | Query and document read together |
| Precomputable | Yes, documents are indexed ahead of time | No, every pair must be scored at query time |
| Scales to | Millions of documents | Tens of candidates |
| Relative accuracy | Good | Better, materially so on the final ordering |
| Role in the pipeline | Find the candidates | Order the candidates |
So the two are not competitors. They are two stages of one pipeline, and the standard 2026 architecture uses both: embeddings retrieve the top 50 to 100 from the whole corpus, and the reranker sorts those survivors.
The pipeline, concretely
- Retrieve wide. Query your vector index for the top 50 to 100 chunks instead of the top 5. This is a parameter change, not an architecture change, and it is the step people forget: reranking a top-5 list accomplishes nothing, because the whole value is promoting something from rank 12.
- Rerank. Send the query and the 50 candidates to the reranker. It returns them scored.
- Cut. Take the top 3 to 5 by reranker score and build your prompt from those.
- Log both orderings. Store the pre-rerank and post-rerank ranks. This is how you find out whether the reranker is earning its latency, and it costs one extra column.
That last step matters more than it sounds. Reranking is easy to add and easy to keep forever without evidence. Logging both orderings turns "we added a reranker" into "the reranker moved the correct chunk into the top 3 on 40 percent of queries where it was not already there", which is a number you can defend or act on.
Which reranker to use
Three realistic options, and the right first move is the easiest one.
Cohere Rerank is a hosted API with strong multilingual coverage and long-context support. Pricing is per search unit, where a unit is one query against up to 100 documents, which makes the cost easy to reason about. It is the default I reach for when finding out whether reranking helps at all, because integration is a single call.
Voyage rerank is a closed API in the same shape, and tends to test slightly better on code and technical documentation. If your corpus is a codebase or developer documentation, it is worth including in the comparison, the same way Voyage embeddings are worth comparing on those corpora.
BGE-reranker-v2-m3 is BAAI's open Apache-licensed model, built on the same base as BGE-M3, multilingual, and small enough to serve from a single mid-range GPU. It is the default starting point for self-hosted setups and the obvious choice when the query text cannot leave your infrastructure, which is a constraint that applies to the reranker exactly as much as to the embedding model. That is usually part of a broader self-hosted deployment.
The costs, honestly
Latency. Expect roughly 100 to 300 milliseconds for 50 candidates. This lands directly on time-to-first-token, which is the latency users perceive, so it is not free. If it hurts, rerank 25 candidates instead of 100: the accuracy loss is usually small and the time roughly halves.
Money. Per-query cost on a hosted reranker is small compared to the generation call that follows it, so reranking rarely moves your bill noticeably. It is one of the few quality improvements in a RAG stack that is genuinely cheap. The relevant comparison is against your total cost per query, where it is usually a rounding error.
Engineering time. About an hour for a hosted reranker into an existing pipeline, most of which is widening the retrieval call and adding the logging. A self-hosted reranker is a day or two, depending on whether you already have GPU serving infrastructure.
When a reranker will not help you
Three cases where adding one is wasted effort, stated so you can rule yourself out cheaply.
- Recall at 50 is already low. The reranker only reorders what retrieval found. If the correct chunk is not in the candidate set, nothing downstream can rescue it. Fix chunking or retrieval first, using the approach in the RAG architecture guide.
- Your corpus is tiny. If you have 200 chunks total and can fit the relevant subset in context, ranking is not your bottleneck and you are adding a stage for nothing.
- Relevance depends on reasoning, not similarity. If the right document is the one with the most recent date or the one matching a policy rule, that is filtering and business logic, not semantic ranking. Do it in code before the reranker ever sees the candidates.
Frequently asked questions
What is a reranker in RAG?
A second-stage cross-encoder that re-sorts a shortlist of retrieved documents by reading the query and each candidate together and scoring the pair. It runs on the top 50 to 100 candidates, never on the whole corpus.
How is it different from an embedding model?
Embeddings encode query and document separately so documents can be indexed in advance, which is what makes search fast. A cross-encoder needs the pair together, which makes it more accurate and impossible to precompute.
How much does reranking improve quality?
Published benchmarks report roughly five to fifteen NDCG@10 points. Your actual gain tracks the gap between your recall at 50 and recall at 5.
What latency does it add?
Around 100 to 300ms for 50 candidates, applied to time-to-first-token. Rerank fewer candidates if that hurts.
Hosted or self-hosted?
Start hosted with Cohere Rerank or Voyage to find out whether it helps. Move to BGE-reranker-v2-m3 if volume justifies it or if query text cannot leave your infrastructure.
Bottom line
A reranker is the highest ratio of retrieval quality to engineering effort available in a RAG stack, and it is routinely skipped in favour of an embedding model migration that costs ten times as much and fixes a different problem. Measure the gap between recall at 50 and recall at 5. If it is large, spend the hour.