How to Evaluate RAG Agent Accuracy and Retrieval Quality

Measure Context Precision and Recall

Context precision and recall evaluate the retrieval stage in isolation, answering the question: did the retrieval system find the right documents?

Context recall measures what fraction of the relevant documents in your corpus were actually retrieved. If three documents in your knowledge base contain information needed to answer a question and retrieval returns two of them, context recall is 67%. Low recall means the answer will be incomplete or wrong because the generator never sees critical information. To measure recall, you need a ground truth mapping of which documents are relevant for each test question. Build this by having domain experts annotate your test set with the document IDs that contain the answer.

Context precision measures what fraction of retrieved documents are actually relevant to the question. If retrieval returns five documents but only two are relevant, context precision is 40%. Low precision means the generator's context window is cluttered with irrelevant information, which increases cost (more tokens processed), introduces distraction (the model may attend to irrelevant passages), and can actively mislead the generator if irrelevant documents contain superficially related but incorrect information.

The standard measurement approach: for each test case, retrieve documents using your production retrieval settings (same embedding model, same chunk size, same top-k parameter). Compare the retrieved set against the annotated ground truth relevance labels. Calculate precision and recall. Most RAG systems target context recall above 85% (you should find most relevant documents) and context precision above 60% (majority of retrieved documents should be useful).

When precision is low, the fixes are: better embeddings that distinguish relevant from superficially similar content, tighter retrieval thresholds that reject low-similarity results, reranking models that push relevant documents to the top after initial retrieval, and better chunking strategies that create more semantically coherent passages.

When recall is low, the fixes are: adding keyword search alongside semantic search (hybrid retrieval) to catch cases where important terms do not embed closely, adjusting chunk overlap to prevent relevant information from being split across chunk boundaries, expanding the top-k retrieval count, and improving the query expansion or reformulation step to match relevant documents that use different terminology.

Score Faithfulness to Retrieved Context

Faithfulness measures whether every factual claim in the generated answer is supported by the retrieved context. An answer with high faithfulness makes only claims that can be traced back to specific passages in the retrieved documents. An answer with low faithfulness introduces information that does not appear in the context, which is hallucination by definition in a RAG system.

The measurement approach: decompose the generated answer into individual claims (statements of fact). For each claim, check whether the retrieved context contains supporting evidence. Calculate faithfulness as the fraction of claims supported by context. A faithfulness score of 0.9 means 90% of claims are grounded in retrieved documents and 10% are hallucinated or inferred beyond what the context states.

Tools like RAGAS and DeepEval automate this decomposition using an LLM: one call extracts claims from the answer, another call checks each claim against the context. The total cost is approximately $0.01-0.02 per evaluated response due to the multi-step LLM calls required.

Target faithfulness above 0.9 for factual/informational RAG agents. For creative or advisory agents where some inference beyond literal context is acceptable (e.g., "based on these product reviews, customers seem to prefer X"), you might accept 0.8 faithfulness, but clearly define which types of inference are acceptable in your rubric.

Common causes of low faithfulness: the model's parametric knowledge overriding the retrieved context (it "knows" something from training and states it even when context does not support it), the model confusing information between multiple retrieved passages (combining facts from document A with facts from document B in invalid ways), and the model making logical inferences that go beyond what the context actually states.

Assess Answer Relevancy and Completeness

Answer relevancy measures whether the generated answer actually addresses what the user asked. A response can be faithful to the context (everything it says is supported by documents) yet irrelevant (it discusses a tangential topic instead of answering the question). This happens when retrieval returns documents that are topically related but do not address the specific question asked.

Measure answer relevancy by generating hypothetical questions from the answer and checking whether they match the original question semantically. If the answer could be a response to a very different question, it has low relevancy. RAGAS implements this by using an LLM to generate questions that the answer would be suitable for, then computing semantic similarity between those generated questions and the original question. High similarity means the answer is on-topic.

Completeness measures whether the answer covers all aspects of the question. A user who asks "What are the pros and cons of approach X?" expects both pros and cons. An answer that only covers pros has low completeness even if everything stated is accurate and relevant. Measure completeness by decomposing the question into sub-questions or requirements and checking whether the answer addresses each one.

For multi-part questions, track per-component completeness. If a user asks three related questions in one message, measure whether the response addresses all three rather than just the first or most prominent one. RAG agents commonly fail on completeness when the retrieved context only covers some aspects of a multi-faceted question, causing the generator to address only the aspects it has context for without acknowledging the gaps.

Test End-to-End with Ground Truth Answers

End-to-end evaluation measures the full pipeline from question to answer, comparing the final output against known correct answers. This is the ultimate quality measure because it integrates retrieval quality, generation quality, and their interaction into a single score.

Build an end-to-end test set by creating question-answer pairs from your knowledge base. For each question, write the ideal answer that a perfect RAG system would produce given the available documents. Include the source documents that contain the information, so you can also measure retrieval quality on the same cases. Aim for 100-500 end-to-end cases covering your most important query categories.

Score end-to-end accuracy using a combination of metrics: semantic similarity between the generated answer and the reference answer (using embeddings), factual overlap (do they contain the same key facts), and LLM-judge scoring on a rubric that assesses accuracy, completeness, and clarity. Use multiple metrics rather than relying on a single one because different metrics catch different failure modes.

Track end-to-end scores over time as you modify the retrieval pipeline, update the knowledge base, or change the generation model. Any component change should improve end-to-end scores or at minimum not degrade them. If a retrieval improvement increases context recall but end-to-end accuracy stays flat, the improvement is not translating into better answers, possibly because the generator was not limited by context recall in the first place.

Monitor Retrieval and Generation Independently

The most actionable diagnostic approach separates retrieval evaluation from generation evaluation so you can identify which component is responsible for any quality issue.

Retrieval-only evaluation: Run your retrieval pipeline against test queries and measure precision, recall, MRR (mean reciprocal rank), and NDCG (normalized discounted cumulative gain) without ever invoking the generator. This tells you how well your search works independently. When end-to-end quality drops, check retrieval metrics first. If retrieval metrics are stable but end-to-end dropped, the problem is in generation. If retrieval metrics dropped, the problem is in search, and fixing generation will not help.

Generation-only evaluation: Provide the generator with perfect context (manually curated relevant passages) and measure output quality. This tests generation capability in isolation, removing retrieval as a variable. If the generator produces poor answers even with perfect context, the problem is in the generation prompt, model capability, or output formatting, not in retrieval.

Interaction effects: Some problems only manifest when retrieval and generation interact. The generator might handle three relevant documents well but fail when given seven (context too long, loses focus). Retrieval might work well for simple queries but return conflicting documents for complex ones, confusing the generator. Test with varying numbers of retrieved documents and varying relevance distributions to understand these interaction effects.

Set up dashboards that display retrieval and generation metrics side by side with end-to-end metrics. When end-to-end quality changes, the component dashboards immediately indicate where to investigate. This eliminates the guesswork that teams without separated metrics experience when debugging RAG quality issues.