Ibravra is the decision engine and digital knowledge hub for modern founders, operators, and cross-border businesses. We publish research-backed incorporation guides, interactive financial tools, and head-to-head software reviews.

Are Ibravra business tools and calculators free to use?

Yes, all interactive financial calculators, invoice generators, and business launch checklists are 100% free with no registration or credit card required.

How does Ibravra verify software reviews and guides?

All IBRAVRA content follows our strict Editorial Constitution and Post-AI Value Rule, requiring real practitioner experience, primary source statutory checks (IRS/FinCEN), and verifiable benchmarks.

What are the three parts of an evaluation?

Data, task, and evaluators: the examples you evaluate, the behavior you evaluate, and the scorers and classifiers that judge the results — the anatomy Braintrust documents in its evaluation guide.

How many examples does a golden dataset need?

Start with 5 to 10 hand-curated examples per critical component, grow the working set to 10 to 20 high-quality examples per behavior, and size the gate set with the sample-size formula — 385 cases for a 5-point margin of error at 95% confidence.

How does the Ragas faithfulness metric work?

It divides the number of claims in the response supported by the retrieved context by the total number of claims, on a 0-to-1 scale — so an answer with 21 of 24 claims supported scores 0.875.

What does an evaluation harness cost per month?

At 500 cases, 4 scorers, and 20 runs a month (40,000 scores), Braintrust Pro costs $249 with no score overage, Braintrust Starter costs $75 in overage, Langfuse Core costs $29 at the 100,000-unit inclusion, and LangSmith Plus costs $39 per seat at the 10,000-trace inclusion — before judge-model token costs.

AI Tools • 12 min read

AI Agent Evaluation Harness: The Accuracy Playbook

Golden datasets, accuracy and groundedness metrics, regression gates, failure-mode triage, and reproducible eval cost math at stated volumes — from the vendors' own docs.

By Enow A. Jovial • Published 2026-11-13

AI Agent Evaluation Harness: The Accuracy Playbook

An AI agent ships the day its outputs stop being inspected and start being trusted. The evaluation harness stands between those two days: a golden dataset of known inputs and expected outputs, scorers that judge each run against it, a gate that blocks deployment when scores regress, and a loop that feeds production failures back into the dataset.

Our editorial analysis found that most teams evaluate their agents the way they test a website: once, by hand, before launch, and never again. That fails because LLM outputs are non-deterministic — the LangSmith docs state the same input can produce different outputs, so quality is hard to assess without a measurement framework. This playbook builds the harness from the vendors' own documentation — LangSmith, Langfuse, Braintrust, Ragas, and Arize Phoenix — with eval cost from their pricing.

What the harness is: data, task, evaluators

Braintrust's evaluation guide gives the cleanest anatomy of an evaluation, in three parts. Data: the examples you evaluate, including inputs and optional expected outputs. Task: the behavior you evaluate — a single LLM call, a retrieval pipeline, or a multi-step agent. Evaluators: the scorers and classifiers you judge the results with — scorers measure quality with numeric scores, classifiers assign labels for grouping and analysis.

The harness runs in five stages: iterate in playgrounds, promote to an experiment that locks in an immutable snapshot, automate in CI/CD so evals run on every pull request, score in production, and feed back by pulling production traces into datasets.

LangSmith draws the line the whole harness turns on: offline versus online evaluations. Offline evaluations test pre-deployment on curated datasets with reference outputs — benchmarking, regression testing, unit testing, and backtesting. Online evaluations monitor live traffic without reference outputs: real-time monitoring, anomaly detection, and production feedback. Offline-only means a silent production regression can run for weeks; online-only means every deploy is a gamble.

The harness is a loop, not a test. Offline evals gate the deploy; online evals watch production; production failures become new offline cases.

Golden datasets: the asset the harness protects

The dataset is the most valuable artifact in the harness, because every score it ever produces is measured against it. LangSmith's guidance is to start by defining what "good" looks like through manually curated examples: 5 to 10 examples of good output for each critical component — good retrievals and answers for a RAG system; tool selection, argument formatting, and trajectory for an agent. The working set needs 10 to 20 high-quality examples.

Four sources feed a dataset: manually curated examples; historical traces; user feedback on failed runs; and heuristics that flag long-latency or error cases. Synthetic data generates additional examples from existing ones and works best from several high-quality templates. Synthetic-only datasets are a trap: they test the agent against the same assumptions that built it.

LangSmith evaluation documentation on offline versus online evaluations and dataset building

Dataset organization decides whether the harness stays trustworthy as it grows. LangSmith documents three mechanisms. Splits segment examples: ML-style splits divide them into training, validation, and test sets to avoid overfitting; category-based splits evaluate input types separately; staged rollout splits keep exploratory examples isolated until ready. Versions are created automatically when examples change, and targeting a specific version in CI ensures dataset updates do not break workflows. Annotation queues collect structured human feedback. One rule governs all of it: the gate set and the tuning set must be different splits.

Metric selection: pick scorers that match the failure

LangSmith's evaluator taxonomy is the right starting point: human evaluators, code evaluators, LLM-as-a-judge evaluators, and decision-model evaluators. Pairwise evaluators compare two outputs head to head. Reference-based evaluators compare against an expected answer; reference-free evaluators judge without one. The choice follows from what can go wrong.

Failure mode	Scorer type	Metric	Authority
Wrong tool or arguments	Code scorer	Tool Call Accuracy, Tool Call F1	Ragas agent metrics
Answer contradicts retrieved documents	LLM-as-a-judge	Faithfulness: supported ÷ total claims	Ragas Faithfulness
Retriever ranks irrelevant chunks first	LLM-as-a-judge	Context Precision: mean precision@k	Ragas Context Precision
Agent wanders off task	LLM-as-a-judge	Agent Goal Accuracy, Topic Adherence	Ragas agent metrics
Harmful or injected content	Guardrail scorer	Block rate on adversarial set	Ibravra guardrail playbook
RAG answer misses question	LLM-as-a-judge	Response Relevancy, Context Recall	Ragas RAG metrics

The RAG and guardrail rows above are single rows, not sections — each has its own treatment on this site. For RAG depth, see our RAG evaluation guide. For filtering depth, see our LLM guardrail playbook.

Two Ragas metrics deserve their formulas, because they are the two most misused. Faithfulness measures how factually consistent a response is with the retrieved context, on a 0-to-1 scale: a response is faithful when all its claims can be supported by the retrieved context. Ragas documents the calculation — identify every claim, check each against the context, divide supported by total:

\text{Faithfulness} = \frac{\text{Number of claims supported by the retrieved context}}{\text{Total number of claims in the response}}

A worked example makes the threshold concrete. An agent answer makes 24 claims; the judge finds 21 supported and 3 unsupported. Faithfulness is 21 ÷ 24 = 0.875. If the gate floor is 0.90, that run fails.

Context Precision evaluates the retriever's ability to rank relevant chunks above irrelevant ones: the mean of precision@k, weighted by relevance, divided by the relevant items in the top K:

\text{Context Precision@K} = \frac{\sum_{k=1}^{K} \left( \text{Precision@k} \times v_k \right)}{\text{Total number of relevant items in the top } K \text{ results}}

Regression gates: the math that makes a gate honest

A gate is a rule: deploy only if metric X stays above floor Y on dataset Z. Three numbers make that rule honest: the sample size, the confidence interval, and the floor.

Sample size first:

n = \frac{z^2 \times p \times (1 - p)}{E^2}

where $z$ is 1.96 for 95% confidence, $p$ is the expected pass rate, and $E$ is the margin of error. Using the conservative $p = 0.5$, an error margin of 0.05 needs $n = 3.8416 \times 0.25 \div 0.0025 = 384.16$, rounded up to 385 cases.

Second, the confidence interval. A gate must judge the lower bound, not the point estimate. For an observed pass rate $\hat{p}$ over $n$ cases:

\hat{p} \pm z \times \sqrt{\frac{\hat{p} \times (1 - \hat{p})}{n}}

Take a run that scores 0.90 over 200 cases with a floor of 0.88. The standard error is $\sqrt{0.90 \times 0.10 \div 200} = 0.0212$, the margin is $1.96 \times 0.0212 = 0.0416$, and the interval is $[0.858, 0.942]$ — the point estimate clears the floor but the lower bound does not, so the gate fails. The same 0.90 over 1,000 cases gives $[0.881, 0.919]$, which clears it.

Third, the floor: a fixed distance below the measured baseline. If baseline faithfulness is 0.90, a floor of 0.87 allows a 3-point regression before blocking; a floor of 0.95 blocks every deploy forever. Recompute the baseline after every improvement, and record the floor with the dataset version.

Langfuse makes the gate mechanical: add the langfuse/experiment-action GitHub Action to a pull-request workflow and raise RegressionError when a score violates its threshold, and the action fails the job. A gate a developer can skip is a suggestion, not a gate.

Price the harness on the free tier first: Langfuse Hobby includes 50,000 units a month and Braintrust Starter includes 10,000 scores, so the first gated deploy costs only judge tokens.

Never gate on a point estimate. A 0.90 observed on 200 cases has a lower bound of 0.858, which fails a 0.88 floor.

Cost of eval: reproducible math at stated volumes

Evaluation is not free, and the vendors price it three different ways. LangSmith prices by seat plus traces: Developer is $0 per seat with up to 5,000 base traces, Plus is $39 per seat with up to 10,000, and usage beyond the base is metered in LangChain Standard Units at $1.00 per LSU. Langfuse prices by units: Hobby is free with 50,000 units a month for 2 users, Core is $29 with 100,000 included and $8 per additional 100,000, Pro $199, Enterprise $2,499. Braintrust prices by scores: Starter is $0 with 10,000 included and $2.50 per 1,000 over, Pro is $249 with 50,000 included and $1.50 per 1,000 over, plus $4 per GB processed data on Starter and $3 on Pro.

Our own arithmetic on stated assumptions, using those published prices: Take a 500-case dataset scored by 4 scorers per case, run 20 times per month: 2,000 scores per run, 40,000 per month. On Braintrust Pro, 40,000 sits inside the 50,000 included, so the platform cost is the $249 fee. On Starter, the 30,000 over the 10,000 included costs $30,000 \div 1,000 \times 2.50 = \$75$ per month in score overage. On Langfuse, assuming 10 units per trace, the workload is 500 × 20 × 10 = 100,000 units — exactly Core's inclusion, so $29. On LangSmith Plus, 500 × 20 = 10,000 traces is exactly the included base, so $0 beyond the $39 seat fee.

The formula generalizes. Monthly eval cost is the platform fee plus the overage on the meter you exceed:

C_{\text{month}} = C_{\text{platform}} + \max(0, V - V_{\text{included}}) \times r

where $V$ is monthly volume on the metered unit, $V_{\text{included}}$ is the included volume, and $r$ is the overage rate. Judge-model token cost sits outside these platform prices: 500 cases judged 20 times a month is 10,000 judge calls, often a larger bill than the platform fee.

Founder & CEO Insight — Enow A. Jovial

"An eval suite you cannot afford to run is the same as an eval suite you do not have. Price the harness at your real run volume before you build it, not after the first invoice." — Enow A. Jovial, Founder & CEO

Failure-mode triage: reading a red gate

Triage a red gate in three passes. First, the dataset: did a version change, or did a synthetic case with a wrong expected output slip in? Second, the judge: did the model, the rubric, or the calibration drift? Third, the agent: did the prompt, model, retrieval, or tool change? Only the third pass is an agent regression.

Classify what the third pass finds. Trajectory failure: the agent called the wrong tool or the tools in the wrong order — fix the tool descriptions or the planner prompt. Retrieval failure: the right documents were not in the context — fix chunking, ranking, or the query rewrite. Grounding failure: the answer added claims the context does not support — add a faithfulness scorer and a stricter instruction to answer only from context. Format failure: the output broke the JSON schema or API contract — add a code scorer that validates structure first. Cost failure: the agent burned its token or tool-call budget — OWASP's 2025 Top 10 lists this as LLM10 Unbounded Consumption; fix it with a hard budget. Agency failure: the agent acted when it should have asked first — OWASP LLM06 Excessive Agency, fixed with an approval boundary. Misinformation failure: a fluent, confident, wrong answer — OWASP LLM09 Misinformation, fixed with grounding plus a human review queue. Log every triage outcome in the dataset, or the failure will recur.

Braintrust evaluation guide on the anatomy of an evaluation and the evaluation cycle

The build sequence: Step 1 to Step 8

Step 1: Name the three behaviors the agent must never regress on, and write 5 to 10 hand-curated examples for each. Step 2: Pick one scorer per failure mode from the metric table, and write each judge rubric plainly. Step 3: Run the 50-case human calibration set and confirm judge-human agreement of at least 0.80. Step 4: Split the dataset into tuning and gate sets, version it, and tag the gate version. Step 5: Compute the sample size for your error margin, and grow the gate set to that size. Step 6: Set each floor as baseline minus 3 points, judged on the lower confidence bound, and wire the gate into CI/CD. Step 7: Price the harness at real run volume. Step 8: Ship with online evals running; route flagged failures into the annotation queue.

Who should NOT use this

If the agent answers fewer than 100 requests a month and a human reads every output, a harness is overhead: read the outputs, keep a list of failures, and revisit this playbook when volume grows. If no one can own the dataset, do not build a harness — an unowned dataset rots and blocks good work. If the agent's outputs are never acted on, a weekly manual review is enough. If the organization cannot accept a deploy being blocked by a number, do not install a gate.

The domain terms here are not decoration. The harness runs as code: an SDK call per trace, an API that posts scores, a CI/CD job that reads a JSON report and a CSV export of failures, and a decision rule expressible as an F1 score, a ROC curve and its AUC, or a pass-rate interval. GPU and TPU capacity matters if the judge runs on local hardware; most teams judge through a vendor API and pay per token. PII in datasets is a real liability: redact PII before it enters a dataset, and confirm the vendor's handling meets GDPR and SOC 2 expectations — Langfuse documents SOC 2 and ISO 27001 reports and a HIPAA-ready region on Pro.

Pre-flight checklist: before the first gated deploy

[x] Three behaviors named, with 5 to 10 hand-curated examples each.
[x] One scorer per failure mode chosen, with a written rubric for every judge scorer.
[x] Judge calibrated on 50 human-labeled cases, agreement of at least 0.80.
[x] Dataset split into tuning and gate sets, versioned, and tagged.
[x] Gate set grown to the sample size the error-margin formula requires.
[x] Floors set as baseline minus 3 points, judged on the lower confidence bound.
[x] Gate wired into CI/CD so a violation fails the pull request.
[x] Monthly eval cost priced at real run volume and approved.
[x] Online evals running, with failures feeding back into the dataset.
[x] PII redacted from every dataset example; vendor data handling confirmed in writing.

An agent without a harness is a product whose quality is a rumor. The harness turns the rumor into a number, and the number into a deploy decision. For the production architecture the harness protects, read our autonomous agent architecture guide. To price the eval budget against your unit economics, run the numbers through the break-even calculator.