LLM Evaluation for Systems That Must Be Right

AI Engineering

I do not treat an LLM evaluation as a leaderboard. I treat it as a versioned argument—with evidence—about which failures a particular system can tolerate before release.

Begin with the decision, not the metric

“Is model B better?” is too vague to evaluate. A useful question names the task, users, consequence of error, operating configuration, and release decision. For a support copilot, the decision might be: can version 7 replace version 6 for English and Arabic refund questions without increasing unsupported policy claims or p95 cost beyond an agreed budget?

That framing matters because language-model quality is multidimensional. The HELM research framework evaluates models across scenarios and metrics including accuracy, calibration, robustness, fairness, toxicity, and efficiency rather than collapsing quality into one universal number. A production system adds more variables: system prompt, retrieval corpus, tools, sampling parameters, safety policy, and post-processing. Change any of them and the evaluated object has changed.

Create an evaluation dossier with immutable identifiers for the model, prompt, tools, retrieval snapshot, code commit, dataset, rubric, and judge. A score without that configuration is not reproducible evidence.

Build a test portfolio around risk

I start with a task taxonomy, then sample cases by consequence and frequency. The portfolio I would take to a release review contains:

• representative cases from real, consented, and de-identified workflows;
• boundary cases where policy, dates, currencies, languages, or permissions interact;
• adversarial cases such as prompt injection and unsupported instructions;
• regression cases distilled from incidents and user corrections;
• counterfactual pairs that vary one sensitive attribute or one critical fact;
• explicit abstention cases where the correct action is to refuse or escalate.

Protect a holdout set from prompt tuning and routine developer inspection. Otherwise the team optimizes to a familiar test rather than the task. Track provenance and licenses; remove secrets and personal data; document synthetic cases as synthetic. Repeated trials are necessary when sampling or tool execution is nondeterministic. Report the run count and dispersion, not only the mean.

Slice results by risk, language, workflow, document freshness, and tool path. A 94% aggregate pass rate can conceal a 60% pass rate on high-value transfers or Arabic policy questions. Confidence intervals are especially important for small slices: “zero failures in ten cases” is not evidence of zero risk.

Use an evaluation stack, not one judge

Different checks answer different questions.

Deterministic assertions should verify what code can know exactly: JSON schema, required citations, tool arguments, arithmetic, forbidden actions, latency, and cost. Reference-based metrics can compare structured outputs or retrieval results. Model graders can assess qualities such as relevance or groundedness at scale, but they introduce their own bias, instability, and blind spots. Human experts remain necessary for ambiguous, novel, or high-impact cases.

Calibrate a model grader against independently labeled examples. Publish its agreement, false-pass rate, false-fail rate, and ambiguous region by slice. Do not let the same model family silently grade its own preferred style. NIST’s work on evaluating machine-generated reports emphasizes dimensions such as completeness, accuracy, verifiability, and claim-to-source support—useful reminders that fluent prose is not a factuality metric.

Route disagreement and high-risk failures to human review. Measure inter-rater agreement and improve a rubric when qualified reviewers interpret it differently. “Human reviewed” is not a method unless the reviewer role, sampling rule, rubric, and resolution process are recorded.

Turn scores into release policy

I set thresholds before examining the candidate result; otherwise the goal tends to move around the preferred release. Here is how I would model a practical gate:

1. zero violations of non-negotiable controls, such as unauthorized tool calls;
2. no statistically credible regression on high-risk slices;
3. a bounded false-pass rate for the calibrated grader;
4. latency and cost budgets at named percentiles;
5. documented acceptance of residual risk by the accountable owner.

Suppose a collections assistant improves answer completeness from 86% to 92% but unsupported fee claims rise from 0.5% to 1.4%. The average may improve while the product becomes less safe. The correct response could be a narrower rollout, a deterministic policy lookup, or mandatory review—not simply another prompt tweak.

In regulated financial institutions, evaluation evidence may also feed model-risk governance. The US Federal Reserve’s revised model risk management guidance published on 17 April 2026 stresses risk-proportionate governance, independent validation, ongoing monitoring, use limitations, and effective challenge. Applicability depends on the institution and use case; an evaluation suite is evidence for governance, not proof of compliance.

Monitor the gap between tests and production

Pre-release cases are a model of reality, not reality itself. In production, record privacy-safe outcome signals: escalation, correction, task completion, policy violation, user appeal, latency, and cost. Join them to the evaluated configuration. Define drift alerts for input mix and failure slices, not only token volume.

Every confirmed failure should produce three artifacts: an incident record, a minimized reproducible test, and a decision about whether the release threshold or control must change. Re-run the suite when the model, prompt, tool, retrieval data, policy, or evaluator changes. Benchmark results decay as these components evolve.

What a defensible evaluation can claim

My design rule is to claim only what the evidence supports. A strong evaluation cannot guarantee that an LLM “must be right.” It can state a narrower and more useful conclusion: under a named configuration, on a versioned portfolio, with stated uncertainty, the candidate met defined thresholds and known residual risks are controlled or accepted.

That claim is less dramatic than a leaderboard win. It is also the kind of claim an engineering owner can reproduce, challenge, monitor, and reverse.

Related reading

• AI Agents in Regulated FinTech: Control Before Autonomy
• RAG for Financial Operations: Evidence Over Eloquence

References

• Liang et al. — Holistic Evaluation of Language Models (HELM)
• NIST — On the Evaluation of Machine-Generated Reports
• Federal Reserve — Supervisory Guidance on Model Risk Management