Trying Out RAG Evaluation With RAGAS, Pinecone and Cohere

RAGAS scores retrieval and generation separately, and the split matters most on an agentic pipeline: when the agent decides not to search at all, the contexts field comes back empty and both retrieval metrics fail the row regardless of whether the answer was actually good. I built a small tool-calling agent over Pinecone with Cohere embeddings and scored it end to end with RAGAS to find exactly where that breaks.

• RAGAS scores retrieval and generation separately, and that split is the whole point. It tells you which half to go fix instead of handing you one number.
• The setup is straightforward: Cohere for embeddings, Pinecone for the index, an OpenAI agent that calls retrieval as a tool. Nothing exotic.
• The interesting problem is not the metrics. It is that when the agent decides not to search at all, there is nothing to put in the contexts field and the retrieval scores go strange.

The setup

Deliberately small, because the point was the measurement and not the pipeline. A public dataset of pre-chunked ArXiv papers, filtered down to two of them, Llama 2 and Mixtral of Experts. Cohere embeds the chunks, Pinecone stores them, and an OpenAI agent gets one tool: search that index.

The two papers are not a random choice. The evaluation questions I used are about those papers, and retrieval can only look good if the answers are actually in the index. Obvious once you say it out loud, easy to get wrong when you are assembling a demo.

# the agent decides when to call this. that turns out to matter a lot
@tool
def search_arxiv(query: str) -> str:
    """Search the indexed ArXiv papers for passages relevant to a query."""
    vec = embed.embed_query(query)
    hits = index.query(vector=vec, top_k=5, include_metadata=True)
    return "\n\n".join(h["metadata"]["text"] for h in hits["matches"])

Cosine similarity for the index, since it compares direction rather than length and embedding magnitude often just reflects how long the text was.

Full code, including the evaluation script, is on GitHub: agentic-rag-evaluation-ragas.

What RAGAS is actually looking at

This is the part I found genuinely useful, and it is why I would use RAGAS again rather than writing my own checks.

It scores two things separately. Retrieval: did the search find the right material. Generation: given what it found, is the answer any good. Those fail for different reasons and have different fixes, so a single combined score would be nearly useless.

On the retrieval side, context precision asks how much of what came back was actually relevant, and context recall asks how much of what was needed actually came back. Noisy versus incomplete.

On the generation side, faithfulness breaks the answer into individual claims and checks each against the retrieved text, which is the hallucination check. Answer relevancy asks whether it answered the question at all, regardless of whether it was right. And answer correctness compares against the ground truth directly.

One thing worth knowing before you run it: most of these use a model as judge, so evaluation is a lot of API calls. It is not a free operation you sprinkle on every commit.

What a low score on each metric probably means, and what to try:

• Context recall — probably missing from the index, or top_k too small. Try smaller chunks, more chunks, a wider corpus.
• Context precision — probably the retrieval is noisy. Try re-ranking, or a stricter threshold.
• Faithfulness — probably the model going beyond its context. Try tightening the prompt, lowering temperature.
• Answer relevancy — probably a prompt problem, not retrieval. Try rewriting the instructions.
• Answer correctness only — probably substance, or the ground truth itself. Read the row yourself.

Then the agent decided not to search

Here is where it got interesting, and where I stopped following the tutorial in my head.

The evaluation format everybody uses is question, retrieved contexts, answer, ground truth. Fine for a fixed pipeline where you always retrieve then generate. But my agent gets to choose. Sometimes it just answers, because the model already knows enough about Llama 2 to say something reasonable without searching.

When that happens the contexts field is empty, so both retrieval metrics score it as a failure. On a question that might have been answered perfectly well. Average those rows into everything else and your retrieval numbers say something is broken when nothing actually ran.

Same thing in a different shape when the agent searches twice with different queries. Do you score everything it found, or the last search, or each one separately? Three answers, three different numbers, and nothing tells you which is correct.

And a smaller one I did not expect: the agent rewrites the question before searching. So context precision is judging retrieval against the user's wording, when the retriever actually saw the agent's version. If precision is low, is the search bad or is the rewrite bad? The metric cannot tell you.

What I ended up doing is simple and not very satisfying. Record whether the tool got called on each question, and how many times, then look at those groups separately instead of averaging everything into one row of numbers.

What the numbers said

Two views, and you want both:

metric_cols = ["context_precision", "context_recall",
               "faithfulness", "answer_relevancy", "answer_correctness"]

result_df[["user_input"] + metric_cols]   # every question, one row each
result_df[metric_cols].mean()             # the summary you show people

Average scores across the run:

• context_precision — 0.22
• context_recall — 0.52
• faithfulness — 0.75
• answer_relevancy — 0.48
• answer_correctness — 0.48

Not good numbers, and the shape of them is more interesting than the level.

Faithfulness is the highest at 0.75, and the two retrieval metrics are the lowest. That already tells you the generation side is not the problem: when the model was given something to work with, it stayed inside it. Most rows have context_precision at exactly 0.

Finding where a problem actually lives

The averages say retrieval. The rows say something more specific.

Several questions came back clean: recall 1.0, faithfulness 1.0, relevancy around 0.87. All of them are about what Mixtral of Experts does to model size and token generation. When the material was in the index, the whole pipeline worked.

Then there is a group where answer_relevancy is exactly 0.00. An exact zero there is not a slightly-off answer, it is a non-answer, the model saying it cannot find the information. Every one of those rows also has context_precision at 0. So the agent searched, got nothing useful, and said so.

Reading what those questions ask about is where it clicked. Expert LRU caching. Speculative expert loading. Offloading on an A100. Those are not in Llama 2 and they are not in Mixtral of Experts. They come from a different paper on MoE offloading, and my index only holds the two I filtered for.

So the low retrieval scores are mostly corpus coverage, not retrieval quality. The retriever was asked for material that was never indexed.

That distinction matters, because the two have opposite fixes. Bad retrieval quality means re-ranking, better embeddings, tuning top_k. Missing coverage means none of that helps and you go index the paper.

One row is a real failure worth naming: context_recall 0.10, faithfulness 0.96, correctness 0.27. The model answered confidently and stayed faithful to almost nothing, which is faithfulness working exactly as documented and also being deeply unhelpful on its own. Faithful to a bad context is still wrong.

Another is the opposite: everything near zero including faithfulness at 0.125. A quantization question, nothing retrieved, and the model answered anyway from what it already knew.

None of this is new thinking. It is the same two-level habit predictive ML always needed: the aggregate tells you something changed, the individual examples tell you what.

Small things that cost me time

Pin your versions. LangChain changes constructor signatures and import paths between minor releases and RAGAS moves faster than that. As a taste of how fine-grained it gets, in RAGAS 0.3.x the results frame calls the question column user_input rather than question, so older code indexes a column that is not there anymore.

Chunk size is the biggest lever in the whole thing and in my case somebody else had already pulled it, since the dataset came pre-chunked. Too large and each embedding blends several topics so search gets fuzzy. Too small and no single chunk holds a full answer. Worth owning that decision yourself if you can.

FAQ

What does RAGAS actually measure?Retrieval and generation separately. Context precision and recall look at whether the search returned relevant and complete material. Faithfulness and answer relevancy look at whether the answer stayed inside that material and addressed the question. Answer correctness compares the answer to a ground truth.

How do you evaluate a RAG agent that might not retrieve?Track whether the retrieval tool was called on each question and group those rows separately. With no tool call the contexts field is empty, so retrieval metrics fail the row regardless of whether the answer was good, and mixing them into the average makes retrieval look worse than it is.

Do RAGAS scores cost money to produce?Yes, most of the metrics use a model as judge, breaking answers into claims and checking each one. An evaluation run is a lot of calls, so it is not something to attach to every commit without thinking about the bill.

Is a small evaluation set worth running?For finding rows to read, yes. For a number you quote to somebody, not really. Judge-based metrics vary between runs and on a small set a single bad row moves the average visibly.

Conclusion

Wiring it up was the easy part. Cohere, Pinecone and an agent with one tool is an afternoon of work.

Deciding what counts as a fair score took considerably longer, and the thing I would still like a good answer for is how to grade the decision to search at all. An agent that skips retrieval and gets lucky looks identical, in these metrics, to one that skipped it and was right to. That feels like the actual question and none of the five metrics touch it.