Deploying a Retrieval-Augmented Generation (RAG) prototype in a demo environment is simple; maintaining high accuracy in production is notoriously difficult. Without automated, quantitative evaluation metrics, engineering teams flying blind suffer from silent hallucinations, irrelevant vector chunk retrieval, and degraded customer trust.
The RAG Triad and Ragas evaluation framework provide empirical mathematical formulas to measure every stage of the RAG pipeline independently.
```
+-----------------------------------------------------------------------------+
| THE RAG EVALUATION TRIAD |
| |
| [ USER QUERY ] |
| / \ |
| / \ |
| Context Relevance Answer Relevance |
| / \ |
| v v |
| [ RETRIEVED CONTEXT ] <==== Faithfulness ====> [ GENERATED ANSWER ] |
| |
| +-----------------------------------------------------------------------+ |
| | 1. Context Precision: Did we retrieve only relevant chunks? | |
| | 2. Faithfulness: Is the answer 100% grounded in retrieved context? | |
| | 3. Answer Relevance: Does the generated answer answer the query? | |
| +-----------------------------------------------------------------------+ |
+-----------------------------------------------------------------------------+
```
The Core Ragas Metrics Matrix
| Ragas Metric | Pipeline Component Evaluated | Mathematical Definition & Focus | Target Production SLA |
| :--- | :--- | :--- | :--- |
| Faithfulness (Groundedness) | Generation (LLM) | $\frac{\text{Claims in Answer Grounded in Context}}{\text{Total Claims in Answer}}$ | $\ge 0.95$ (Zero Hallucination) |
| Answer Relevance | Generation (LLM) | Mean cosine similarity of generated hypothetical questions vs. original user query | $\ge 0.88$ |
| Context Precision | Retrieval (Vector DB) | $\text{Mean Average Precision@K}$ of ground-truth chunks in retrieved ranks | $\ge 0.85$ |
| Context Recall | Retrieval (Vector DB) | $\frac{\text{Ground Truth Sentences Attributed to Context}}{\text{Total Sentences in Ground Truth}}$ | $\ge 0.90$ |
Mathematical Breakdown of Faithfulness (Hallucination Detection)
To compute Faithfulness automatically without manual human labeling:
1. Deconstruct the Answer into Atomic Claims: An evaluation LLM breaks the generated output $A$ into a set of discrete, factual propositions $S = \{s_1, s_2, \dots, s_n\}$.
2. Verify Claims Against Retrieved Context: For each proposition $s_i$, the evaluation model verifies whether the proposition is logically entailed by the retrieved context chunks $C$.
3. Calculate the Groundedness Ratio:
$\text{Faithfulness Score} = \frac{|S_{\text{verified}}|}{|S_{\text{total}}|}$
If a generated response contains 5 factual claims and 1 claim cannot be substantiated by the retrieved document chunks, the Faithfulness score is $4 / 5 = \mathbf{0.80}$ (Flagged for hallucination review).
Production Evaluation Stack: Integrating Ragas with Qdrant & Gemini
```python
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevance, context_precision, context_recall
from datasets import Dataset
Construct evaluation dataset from production logs
eval_data = {
'question': ['What is the penalty for late Form 5472 filing?'],
'contexts': [['Under IRC Section 6038A, the statutory penalty for failing to timely file Form 5472 is $25,000 per entity.']],
'answer': ['The IRS assesses a mandatory $25,000 late-filing penalty under IRC Section 6038A.'],
'ground_truth': ['A $25,000 penalty is assessed by the IRS under Section 6038A.']
}
dataset = Dataset.from_dict(eval_data)
results = evaluate(dataset, metrics=[faithfulness, answer_relevance, context_precision, context_recall])
print(results)
```
Actionable RAG Quality Assurance Checklist
[x] Build Golden Test Dataset: Curate 100+ verified question-context-answer ground truth pairs across your documentation.
[x] Implement Automated CI/CD Regression Evaluation: Run Ragas evaluation on pull requests before modifying chunking strategies or embedding models.
[x] Monitor Real-Time Production Faithfulness: Sample 5% of live customer queries for asynchronous hallucination scoring.
[x] Tune Chunk Sizes with Precision Benchmarks: Optimize chunk character counts (e.g., 512 tokens with 50-token overlap) based on empirical context recall scores.
To optimize your production retrieval infrastructure, read our guide on Self-Hosting Qdrant & Vector DBs on Hetzner and explore Building Autonomous Customer Support Agents.
For real-time voice latency benchmarks, consult AI Voice Agents for Inbound Sales and automate engineering checks in AI Code Review Actions.