The Vector Search Was Right. The Reranker Was Wrong.

I was asked to inspect a RAG pipeline that wasn’t consistently returning the right context.

The architecture looked normal. Documents were embedded and stored in PostgreSQL with pgvector. A similarity search retrieved candidate chunks, then a reranker reordered them before the final context was sent to the LLM.

Nothing unusual there.

Query
  ↓
Embedding
  ↓
pgvector similarity search
  ↓
Candidate documents
  ↓
Reranker
  ↓
Top documents
  ↓
LLM

The obvious assumption was that retrieval needed improvement.

It didn’t.

Once I started looking at the pipeline stage by stage, I found that pgvector was already retrieving the correct documents.

We were finding the right documents, then losing them.

Looking at retrieval before looking at the LLM

When a RAG system produces a bad answer, it’s tempting to start with the final prompt.

Maybe the model ignored the context. Maybe the system prompt isn’t strict enough. Maybe the context window contains too much noise.

But before touching any of that, I wanted to know what documents actually reached the model.

So I started logging the ranking at different points in the retrieval pipeline.

For some queries, the result looked roughly like this:

| Document | Vector search | After reranking | | ---------------- | ----------------- | ------------------- | | Correct document | #1 | #4 | | Document B | #2 | #1 | | Document C | #3 | #2 | | Document D | #4 | #3 |

That changed the direction of the investigation.

The retriever knew about the correct document.

The candidate set contained it.

The problem appeared after retrieval.

The reranker was taking a good candidate set and making it worse.

A reranker isn’t automatically an upgrade

It’s easy to think about a reranker as a second, smarter retrieval stage.

Retrieve 20 or 50 candidates with vector search, give them to a cross-encoder, and let the reranker produce a better ordering.

Conceptually:

Fast retrieval
      ↓
Top-K candidates
      ↓
More expensive relevance model
      ↓
Better Top-N candidates

And often that works.

But there’s an assumption hidden inside this architecture:

The reranker needs enough information to make a better decision than the first-stage retriever.

That wasn’t consistently true in this pipeline.

The two stages weren’t seeing the same document

This was the first important issue I found.

The embedding used for similarity search represented more information about the document than the text being evaluated during reranking.

That distinction matters.

It would be inaccurate to say that pgvector itself “had more context.” pgvector doesn’t understand the document. It compares vectors.

The important question is:

What text was used to create the embedding, and what text is later passed to the reranker?

Imagine that the embedding represents something like:

Title
Topic
Section
Chunk content
Additional document context

The resulting vector encodes information from all of that text.

But later the reranker receives something closer to:

Query
+
Chunk content

Now the two stages aren’t evaluating exactly the same representation.

The similarity search might rank a chunk highly because the title, topic and surrounding information make its relevance obvious.

The reranker doesn’t necessarily see those signals.

And because reranking happens later in the pipeline, its ranking wins.

At that point, adding a “smarter” model doesn’t necessarily improve retrieval.

You’re asking two models to rank documents using different evidence.

Then I found another problem: language

There was a second issue hiding in the reranking stage.

The system had multilingual content, but the reranker that had been selected wasn’t a good fit for the languages it was expected to rank.

That’s an easy detail to overlook.

A RAG pipeline may have a multilingual embedding model that successfully places semantically related content close together in vector space.

The first-stage retrieval can therefore look good.

Then those candidates are passed to another model.

If that reranker doesn’t have comparable multilingual capability, you’ve introduced a language bottleneck after successful retrieval.

The embedding model had already done the difficult part.

The reranker was undoing some of it.

This is why checking only the final retrieval output can be misleading. You see bad Top-N results and assume vector search isn’t finding the right documents.

It was.

The documents were being reordered incorrectly afterward.

Truncation makes this even easier to miss

Cross-encoder rerankers don’t receive infinite text.

They have input limits, and the effective input usually contains both the query and candidate document.

Something conceptually like:

[query] + [document]

If the document is long, something has to be truncated.

And if the information that distinguishes one candidate from another happens to be outside the retained portion, the reranker never gets to evaluate it.

This becomes particularly important when your retrieval chunks already contain structured context:

Title
Topic
Section
Metadata
Chunk

Adding those fields gives the reranker more useful signals, but it also consumes more tokens.

So simply saying “give the reranker more context” isn’t enough.

There is a tradeoff between context richness and truncation.

I changed what the reranker was actually evaluating

Instead of treating the reranker input as an implementation detail, I started treating it as part of the retrieval architecture.

The representation passed to the reranker needed to preserve the signals that made the document relevant in the first place.

Conceptually, the change was closer to:

Before

Query
+
Chunk

versus:

After

Query
+
Title
+
Topic
+
Section
+
Chunk

I also increased the maximum input length available to the reranking stage so that adding those fields didn’t immediately result in more aggressive truncation.

That improved the results.

But it didn’t magically solve every ranking regression.

And I think that’s the more useful observation.

More candidates weren’t necessarily helping either

The pipeline was initially sending a relatively large candidate set into the reranker.

The usual argument for this makes sense: retrieve broadly first, then let the more precise model decide.

But every additional candidate has a cost.

For a cross-encoder reranker, each query-document pair has to be scored.

Roughly:

1 query × 50 candidates

means 50 relevance evaluations.

Reducing the candidate pool significantly reduced reranking latency in my tests while keeping retrieval quality close to the vector-search baseline.

That was another reminder that RAG optimization isn’t just about maximizing retrieval quality.

It’s a balance between retrieval quality, reranking quality, context quality, latency and cost.

A configuration that improves one metric while adding a large amount of latency isn’t automatically a better production configuration.

I stopped evaluating only the final result

One of the most useful changes during debugging was comparing rankings at each stage.

Looking only at:

Query → RAG → Answer

hides too much.

I wanted to know what happened to the documents between those points.

Now the failure becomes obvious.

Document A didn’t disappear because the LLM ignored it.

It never reached the LLM.

This distinction matters because the fix is completely different.

No amount of prompt engineering can make the model use a document that your retrieval pipeline removed before generation.

Measure ranking changes, not just final RAG answers

I also found it useful to explicitly compare the ranking before and after reranking.

For every evaluation query, I could classify the reranker result as:

Better
Same
Worse

Then calculate retrieval metrics such as:

MRR
Recall@1
Recall@3
Recall@5

This gives you a much clearer picture than looking at a few generated answers manually.

If the correct document moves:

#4 → #1

the reranker helped.

If it stays:

#1 → #1

it didn’t change anything.

But if it moves:

#1 → #4

that’s a regression worth investigating.

And when those regressions cluster around a particular language, document type, chunk structure or query pattern, you suddenly have something concrete to debug.

The reranker wasn’t really the problem

Or at least, not by itself.

The actual problem was the assumption that inserting a reranker after vector search automatically made the retrieval pipeline better.

There were several things that needed to line up:

Embedding model
      ↓
Document representation
      ↓
Vector retrieval
      ↓
Candidate count
      ↓
Reranker model
      ↓
Reranker language support
      ↓
Reranker input representation
      ↓
Input length / truncation
      ↓
Final Top-N context

Changing any one of these can change the final ranking.

That’s why I now think about reranking less as a component that sits after retrieval and more as another retrieval model with its own assumptions.

It has its own training distribution.

Its own language capabilities.

Its own input limits.

And, most importantly, its own view of the document.

What I’d check first when a reranker makes RAG worse

If I encountered the same problem again, I wouldn’t start by swapping models.

I’d first compare the raw vector-search results with the reranked results.

If the correct documents are already present near the top before reranking, the first-stage retriever may not be the problem at all.

Then I’d inspect four things:

Document representation. Is the reranker seeing the same important information that contributed to the embedding?

Language support. Does the reranker actually support the languages appearing in queries and documents?

Truncation. Is important content disappearing because the query-document pair exceeds the model’s effective input length?

Candidate count. Are we reranking more candidates than we need, paying additional latency without improving the final ranking?

Only after that would I start experimenting with another reranker.

The part I nearly missed

The most interesting part of this debugging session wasn’t that a particular reranker performed poorly.

Models can be replaced.

The important part was realizing that the pipeline was successfully retrieving the information and then destroying that signal in the next stage.

That’s a very different failure from “RAG can’t find my document.”

And it’s one reason I don’t treat retrieval as a single black box anymore.

When a RAG system gives me the wrong context, I want to see the ranking before reranking, the ranking after reranking, what representation each model actually received, and what finally entered the prompt.

Because sometimes your vector search already has the right answer.

The next stage just needs to stop getting in its way.