RAG for Financial Operations: Evidence Over Eloquence

AI Engineering

I treat retrieval-augmented generation (RAG) as governed evidence retrieval with a generator attached. It is useful in financial operations only when every material claim traces to current, authorized evidence; retrieval does not guarantee truth, completeness, or permission. The operations examples below are synthetic.

Define the decision the answer may influence

“Search our procedures” is too broad. Specify the user, task, evidence set, and allowed consequence. Answering where to find a policy is lower risk than recommending whether to release a payment exception. A system that drafts an investigation note differs from one that triggers a workflow.

The original RAG paper by Lewis and colleagues combines parametric generation with retrieved non-parametric memory and provides provenance for generated text. It establishes a technique, not a control framework for financial decisions.

Write an answer contract: supported sources, maximum source age, jurisdiction or entity, required citations, prohibited data, confidence/abstention behavior, and actions the answer cannot authorize. For a material decision, keep the human or deterministic rule responsible and present the retrieved evidence separately from the model’s interpretation.

Govern ingestion as a data product

Each document needs a stable identifier, version, effective and expiry dates, owner, jurisdiction, confidentiality label, retention rule, and source URL. Preserve the original artifact and a checksum. Chunked text is a derived representation; link every chunk to its page, section, and document version.

Process updates and deletions. A superseded procedure must stop appearing in new answers while remaining available to authorized audit workflows where retention requires it. Build deletion propagation through the object store, search index, vector index, cache, and evaluation fixtures.

Do not mix incompatible embeddings during a silent model change. Version the chunking algorithm, embedding model, and index. Build a shadow index, compare retrieval quality and access behavior, then cut over atomically. Retain enough lineage to reconstruct which index produced an answer.

Enforce authorization throughout retrieval

An entitlement filter drawn as one box is insufficient. Access control begins before ingestion, is attached to each derived chunk, is enforced during candidate generation and ranking, and is checked again before context reaches the model and before an answer reaches the user.

Use allow-based policies tied to authenticated identity, role, entity, jurisdiction, case, and purpose. Avoid retrieving broadly and filtering only after ranking: unauthorized titles or snippets may leak through logs, scores, caches, or model context. Partition indexes where the security boundary warrants it, and test negative access cases continuously.

Retrieved documents are untrusted inputs. They can contain malicious instructions or obsolete advice. Treat their content as evidence, never as system instructions. Strip active content, delimit quotations, allowlist tools, and prevent a document from changing the answer contract.

Separate retrieval quality from answer quality

Evaluate retrieval first. For versioned test questions, measure whether the required evidence appears in the top results, whether forbidden evidence is absent, and whether filters, freshness, and citations resolve to the expected version. Track recall at a fixed candidate count, rank quality, no-result rate, and latency by security slice.

Then evaluate generation. NIST’s work On the Evaluation of Machine-Generated Reports emphasizes dimensions such as completeness, accuracy, verifiability, and claim-to-source support rather than fluency alone. Break answers into atomic claims and label each as entailed, contradicted, unsupported, or not requiring evidence.

A citation is not proof. The linked passage may be real but irrelevant. Add a verifier that checks quotation location, document version, user entitlement, and whether the cited passage entails the claim. High-risk or low-confidence results should abstain with a useful explanation: what could not be established, which sources were checked, and what the operator should do next.

Keep privacy and auditability in tension

The NIST Generative AI Profile addresses information integrity, privacy, confabulation, monitoring, and provenance risks. Apply data minimization: do not log full prompts, documents, or answers by default. Record identifiers, versions, policy decisions, model configuration, latency, token and cost measures, citation results, and a protected reference to content where investigation requires it.

Use synthetic examples for development and evaluation. Redact personal and account data before model calls. Define model-provider retention and training settings, data residency, encryption, incident duties, and subcontractors. Do not put secrets or access tokens in the retrieval context.

Design the operating loop

Monitor unsupported-claim rate, citation entailment, stale-source use, access denials, abstention, retrieval miss, feedback by task, and operator overrides. Segment by document family, language, entity, and risk tier. User thumbs-up is not a correctness label; collect structured reasons and review sampled evidence.

Re-evaluate after changing the model, prompt, chunker, embedding, index parameters, reranker, policy engine, document set, or provider. Keep a release record that binds all those versions to results and known limitations. A kill switch should disable generation while leaving ordinary source search available.

Test the dangerous cases

Test a user asking about another entity, a document whose permissions changed, a superseded policy, conflicting sources, a malicious instruction inside a PDF, a missing page, an unsupported numerical claim, an answer in Arabic citing an English source, and deletion of a sensitive artifact. Verify both the response and the absence of leakage in logs and caches.

The architecture decision

Build RAG as governed evidence retrieval with a generator attached—not as a chatbot with a vector database. Version sources and derived data, enforce entitlement at every boundary, evaluate retrieval and generation separately, verify claim-to-source support, and make abstention successful. The goal is not the most fluent answer. It is an answer whose scope, evidence, and uncertainty an operator can inspect before acting.

Related reading

• AI Agents in Regulated FinTech: Control Before Autonomy
• LLM Evaluation for Systems That Must Be Right

References

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• NIST — On the Evaluation of Machine-Generated Reports
• NIST — Generative AI Profile