What is reranking?
Your RAG system retrieves the right document, ranks it fourth, and the model only reads the top three.
Reranking
Cross-encoder reranking
Reranking is a second pass over retrieved passages. It reorders them by relevance to the question, using a slower, more accurate model than the one that fetched them. Only the passages that survive this cut reach the model.
Vector search finds passages that sit close to the question in embedding space. Close is not the same as relevant. A passage can share the topic and the words of a question and still fail to answer it. Meanwhile, the one that does answer it can sit a little further away, and miss the first cut.
A reranker reads the question and each candidate passage together, as a pair. It does not compare two separate vectors the way a first search does. That pairwise read scores relevance better, but it is too slow to run against a whole document set. It only sees the shortlist a first search already narrowed down.
The usual shape: retrieve 50 to 100 candidates cheaply, then rerank down to the 5 or so that go to the model. Retrieval's job shifts too. It no longer has to find the exact right answer, only to put it somewhere in the pile the reranker sees.
It is usually the cheapest accuracy fix available
When a RAG system retrieves fine but answers wrong, the fault usually sits in the order, not the search. The correct passage came back. It just did not make the top three passed to the model, because vector similarity ranked three others above it.
Reranking fixes that without touching the index, the embeddings or the chunking strategy. It sits between retrieval and generation as one extra step. That makes it the first thing worth adding when accuracy stalls and a full pipeline rebuild is not yet justified.
It costs latency, not architecture. A reranking call adds tens to a few hundred milliseconds depending on how many candidates it scores. For most support and internal-search use cases that delay is invisible against the time the model itself takes to generate an answer.
- 01Keep the first-stage retrieval wide. A reranker can only promote a passage that made the shortlist.
- 02Rerank before you touch chunk size or embedding model. It is the cheaper change to test first.
- 03Log the reranker's scores, not just the final order, so a bad call is visible later.
- QuestionThe user's query, embedded.
- RetrieveFast search, wide net: 50 to 100 candidates.
- RerankSlow, pairwise scoring of each candidate.
- CutKeep the top 5, drop the rest.
- GenerateModel sees only what survived the cut.
Retrieval decides what is in the running. Reranking decides what the model actually reads.
Common questions
01Does reranking improve RAG accuracy?
Yes, in the common case where the correct passage is retrieved but ranked too low to reach the model. Reranking will not help if the passage was never retrieved at all. That is a search or chunking problem. Reranking cannot promote a passage that never made the shortlist.
02Is a reranker the same as an embedding model?
No. An embedding model turns text into a vector once, ahead of time, so similarity can be compared cheaply at query time. A reranker reads the question and each candidate together at query time and produces a relevance score, which is slower but more accurate per pair.
03Do I need reranking if I already use hybrid search?
Often yes. Hybrid search improves what gets retrieved by combining keyword and vector matching. Reranking improves the order of what was already retrieved. The two solve different failures and are commonly used together.
04How many candidates should go into a reranker?
Start around 50 to 100 from the first-stage search, reranked down to the 3 to 10 that go to the model. Measure against real questions from your own data rather than copying a default. The right count depends on how similar your documents are to each other.
