Loop engineering is the emerging practice of designing the system that repeatedly prompts, equips, checks, and stops an AI agent. The important shift is not from one clever prompt to a longer prompt. It is from supervising every turn manually to engineering a controlled execution loop.

I find the term useful because it moves the architecture discussion to the right level. The question is no longer, “Can the model complete this task?” It becomes:

What evidence must one iteration produce before the system is allowed to continue, stop, escalate, or create a real-world effect?

That is the difference between an agent that keeps trying and an operating system for agent work.

Loop engineering is one layer above the agent run

IBM’s definition of loop engineering describes an iterative workflow organized around a goal, an action, an observation, and an adjustment. OpenAI’s description of the Codex agent loop shows the lower-level mechanism: a harness orchestrates the user, the model, and the tools the model invokes.

Those views are compatible, but they describe different design scopes.

Loop engineering does not replace the other three. A loop simply compounds their quality. A poor prompt, polluted context, or overpowered harness becomes more dangerous when it can run repeatedly.

The smallest useful loop has a contract

I would not start with multiple agents or a sophisticated orchestrator. I would begin with a contract that can be inspected without reading model reasoning.

goal:
  outcome: classify one reconciliation exception
  done_when: evidence bundle passes deterministic checks

scope:
  read: [ledger, processor_report, settlement_report]
  write: [case_note, proposed_correction]
  forbidden: [post_ledger_entry, move_money, close_case]

limits:
  iterations: 4
  wall_time: 8m
  model_cost: 1.50_USD

evidence:
  required:
    - source_completeness
    - amount_and_currency_match
    - identifier_lineage
    - verifier_result

terminal_states:
  - proposed
  - no_action
  - needs_human
  - control_failed

This is not a universal syntax. It is a design checklist expressed as configuration. A credible loop needs six things.

1. A recursive goal with a verifiable end

“Investigate the exception” is an activity. “Produce a complete evidence bundle, a classified cause, and a proposal that passes these four checks” is an outcome.

The goal must be re-evaluated on every iteration. Otherwise the agent can optimize for whatever intermediate signal is easiest: more notes, more tool calls, a plausible explanation, or a green evaluator score that does not represent the business result.

2. A bounded trigger

The loop needs a reason to start: an event, schedule, queue item, failed control, or human request. The trigger should carry an identity, priority, owner, and deduplication key.

Without an idempotent trigger, one real-world issue can create several active loops. That is inconvenient for code maintenance and dangerous for financial operations.

3. Prepared context, not accumulated conversation

Each cycle should receive a deliberately assembled evidence bundle: current records, relevant policy, tool results, prior decisions, and a compact summary of earlier attempts.

Appending the full history is not memory design. It is context inflation. The loop should preserve durable facts and decisions while discarding conversational residue.

4. Tools with authority proportional to the task

OpenAI’s practical guide to agents recommends assessing tool risk based on factors such as whether access is read-only or write-enabled, reversibility, permissions, and financial impact. That is especially important inside a loop because tool authority is exercised repeatedly.

I use three tool classes:

• Observe: query records, retrieve policy, inspect telemetry.
• Prepare: create a draft, proposal, test fixture, or case note.
• Effect: modify a customer state, deploy software, post a ledger entry, or move money.

An observe tool may be automatic. A prepare tool usually needs validation. An effect tool needs explicit authorization based on impact and reversibility.

If tools are exposed through MCP, transport authorization is only part of the design. The MCP authorization specification defines protections such as OAuth-based authorization, secure token handling, HTTPS, and scopes. The business layer must still decide whether this agent, for this case, at this point in the loop, may invoke the tool.

5. Evidence that is stronger than model confidence

A model saying “the task is complete” is a claim. The loop needs evidence that another component can inspect.

Useful evidence includes:

• test results bound to the current source version;
• schema and invariant checks;
• before-and-after snapshots;
• signed or immutable tool receipts;
• independent record counts and control totals;
• deterministic policy outcomes;
• a model-based critique with a stable rubric;
• human approval for high-impact exceptions.

Anthropic’s evaluator-optimizer pattern is valuable when success criteria are clear enough for iterative feedback to improve the result. But a second model is not automatically an independent control. Two models can share missing context, incentives, or blind spots. The evaluator should add evidence, not ceremonial agreement.

6. Terminal states that include failure

A loop without a negative terminal state is an infinite retry policy.

At minimum, I want:

• done: the requested outcome is proven;
• no action: the evidence shows that a change is unnecessary;
• needs human: ambiguity or impact exceeds delegated authority;
• control failed: a required source, check, or safeguard is unavailable;
• budget exhausted: the loop reached its time, iteration, or cost limit.

Stopping safely is a product feature. It prevents an uncertain agent from converting uncertainty into repeated action.

The maker-checker pattern needs three kinds of verification

Loop engineering discussions often reduce verification to a maker agent and a checker agent. I prefer a three-part model.

Deterministic verification

Use code for properties that code can decide: type checks, balances, thresholds, allowed transitions, completeness, signatures, uniqueness, and policy rules.

Deterministic checks are cheap, repeatable, and explainable. They should not be delegated to a model merely because a model is already present.

Model-based evaluation

Use a model when the property requires interpretation: whether an explanation is supported by the evidence, whether a proposed plan addresses the stated goal, or whether the output omits a material risk.

Anthropic’s guidance on agent evals emphasizes that multi-turn agents must be evaluated across tool use, state changes, and intermediate results—not only the final answer. I would therefore score both the trajectory and the terminal artifact.

Accountable authority

Use a named human or formally delegated policy for the consequences the system is allowed to create.

The NIST AI RMF Core calls for documented scope, responsibilities, human oversight, testing, independent assessment, and production monitoring. In practice, that means the architecture should show who owns the outer loop: risk tolerance, tool authority, exception policy, evaluation changes, and the decision to expand autonomy.

A worked use case: a reconciliation exception loop

Consider a synthetic payment operation with three independently produced records:

1. an internal ledger entry;
2. a processor or scheme report;
3. a settlement or bank statement.

The reconciliation engine finds a mismatch. A traditional workflow puts the case into an analyst queue. A poorly designed agent workflow gives an AI assistant broad access and asks it to “resolve the exception.”

A loop-engineered version is narrower.

Iteration 1: assemble facts

The loop retrieves the three records, verifies source completeness, normalizes identifiers, and builds control totals. It cannot update any financial record.

If a source is missing, the terminal state is control_failed. The agent does not infer settlement from a success message.

Iteration 2: classify and propose

The maker classifies the mismatch:

• timing difference;
• duplicate report;
• identifier mapping failure;
• fee or foreign-exchange variance;
• missing ledger posting;
• unknown.

It then proposes either no action, delayed review, a data correction, or a compensating ledger entry. The proposal includes the expected effect and the records that support it.

Iteration 3: verify independently

Deterministic controls recompute totals, validate currency and amount, check that referenced records exist, and confirm that the proposed transition is permitted.

A separate evaluator checks whether the narrative is supported by the evidence and whether a plausible alternative cause was ignored. The evaluator sees the evidence bundle and rubric, not the maker’s hidden reasoning.

Outer-loop decision: authorize or stop

No model may post a ledger entry. Low-risk data enrichment can be approved by policy if it is reversible. A financial correction requires the authorized operating role.

After approval, the system creates a new compensating entry; it never edits financial history. Reconciliation then runs again against the new state. Only that fresh result can close the case.

This is where the FinTech example clarifies the broader concept: the inner loop can accelerate investigation, but the outer loop owns financial truth.

The Basel Committee’s operational-resilience principles emphasize the ability to respond, recover, and learn while sustaining critical operations. Loop engineering should reinforce that discipline. It should not create a fast but opaque bypass around it.

Observe the loop as a production system

An agent trace is not only a transcript. The system needs correlated telemetry for:

• trigger and case identity;
• context and policy versions;
• model, prompt, and evaluator versions;
• tool calls and authorization results;
• iteration count, latency, and cost;
• evidence checks and failures;
• human decisions and reason codes;
• terminal state and later business outcome.

OpenTelemetry’s signal model provides a useful implementation vocabulary: traces show the path, metrics show aggregate behavior, logs record events, and baggage carries controlled correlation context.

The metrics I would review are not “tokens used” alone.

Every repeated stop is also design information. If the loop repeatedly lacks one identifier, that is a data-contract problem. If reviewers repeatedly reject the same proposal class, that is an evaluation or authority problem. If the same exception returns after correction, the loop is treating symptoms rather than causes.

Where loop engineering is the wrong answer

The term is new; the temptation to apply it everywhere will be strong.

Use deterministic automation when:

• the states and transitions are known;
• the inputs are structured;
• the decision is already encoded reliably;
• an agent would only imitate a rules engine;
• uncertainty adds no business value.

Use a human-led workflow when:

• the goal itself is disputed;
• the evidence cannot be made available safely;
• accountability cannot be delegated;
• the action is irreversible and rare;
• learning from the decision matters more than throughput.

Use an agent loop when the work combines interpretation, tool use, multiple steps, recoverable actions, and verifiable intermediate progress.

The test is not whether the task is difficult. It is whether the task can be bounded and proven.

A practical adoption sequence

Stage 1 — Shadow

Run the loop against historical or duplicated cases. Give it read-only tools. Compare its classifications and evidence bundles with known outcomes.

Stage 2 — Recommend

Let the loop prepare proposals for live work while humans make every consequential decision. Measure missing evidence, overrides, failure modes, cost, and latency.

Stage 3 — Delegate reversible actions

Allow narrowly scoped, reversible updates when deterministic checks pass. Keep high-impact actions behind human authority.

Stage 4 — Expand by evidence class

Increase autonomy only for the classes of work that have stable inputs, clear controls, acceptable error rates, and reliable recovery—not because the average score improved.

At every stage, version the loop itself: goal, context assembly, tool contracts, evaluator rubric, budgets, and stop conditions. Otherwise a changed prompt or tool silently changes the operating control.

My design rule

Do not optimize a loop for the number of iterations it can complete without a human. Optimize it for the quality of evidence it produces before the system changes state.

A good loop has the confidence to continue when progress is proven, the discipline to stop when controls fail, and an accountable owner for the consequences it creates.

That is loop engineering: not endless autonomy, but engineered recurrence.

Related reading

• Observability for AI: From Tokens to Business Outcomes
• Responsible AI Governance as an Engineering System
• Reconciliation by Design, Not by Spreadsheet
• How to Build an MCP Server for Your API

References

• IBM — What Is Loop Engineering?
• OpenAI — Unrolling the Codex Agent Loop
• OpenAI — A Practical Guide to Building AI Agents
• Anthropic — Building Effective Agents
• Anthropic — Demystifying Evals for AI Agents
• NIST — AI Risk Management Framework Core
• Basel Committee — Principles for Operational Resilience
• OpenTelemetry — Signals
• Model Context Protocol — Authorization

Good engineers can produce a difficult system without making an obviously bad decision. Each team responds rationally to its own deadline, service boundary, budget, and incident queue. The architecture becomes messy when those local decisions accumulate without a mechanism that protects the end-to-end system.

That is why I do not begin an architecture review by asking, “Who designed this?” I begin with a different question: What environment made this design the easiest reasonable choice?

Architecture is shaped by the environment around the team

Melvin Conway’s original paper, “How Do Committees Invent?”, connected a system’s structure to the communication structure of the organization that designed it. The point is deeper than drawing team boxes around services. Communication paths, delegation, incentives, and ownership boundaries constrain which designs are practical.

DORA’s guidance on loosely coupled teams makes the operating consequence explicit: teams perform better when they can make, test, and deploy changes independently. Independence, however, does not mean isolation. A team can deploy independently and still create hidden coupling through shared data, duplicated rules, inconsistent identities, or an undocumented operational dependency.

The distinction I use is:

• Autonomy means a team can make a bounded decision and carry its consequences.
• Isolation means a team cannot see, influence, or measure the consequences outside its boundary.

Healthy architecture needs the first and actively prevents the second.

Five forces that turn reasonable decisions into system debt

1. Local optimization rewards the visible target

A team is usually measured on the work it can directly control: roadmap delivery, service availability, sprint commitments, cloud cost, or ticket closure. Those measures are useful, but they are incomplete.

Imagine that an onboarding team is measured on conversion and lead time. Reusing the enterprise customer profile requires coordination with another domain and a migration the roadmap did not fund. The team creates a local profile store. The decision improves the local target and may even be the correct short-term choice. At system level, the organization now has two customer identities, two data-retention paths, and a reconciliation problem.

The architecture followed the scorecard. If leaders measure only local throughput, they should expect local designs.

2. Delivery pressure changes the option set

Deadlines do not merely make teams work faster. They remove options that require discovery, coordination, migration, or platform work.

Under pressure, temporary solutions are attractive:

• copy a validation rule instead of publishing a reusable contract;
• query another team’s database instead of waiting for an API;
• add a second queue because the shared event contract is difficult to change;
• keep a manual approval because automating the control does not fit the release;
• bypass a platform capability because onboarding takes longer than the feature.

The danger is not the workaround itself. The danger is a workaround with no owner, expiry condition, observable cost, or funded removal plan. “Temporary” is not a lifecycle state.

3. Unclear ownership hides at the seams

Most organization charts assign applications and teams. They do not always assign the relationship between them.

Who owns the meaning of “customer active” across onboarding, compliance, core banking, cards, and payments? Who owns replay after an event contract changes? Who decides whether a failed downstream posting can move a payment back to processing? Who explains the customer outcome when every individual service met its SLO?

If the answer is “all of us,” the operational answer is often “nobody.” Microsoft’s current cloud governance guidance similarly emphasizes explicit function, authority, scope, and interaction across teams. The useful lesson is not to create another committee. It is to make decision rights and escalation paths executable.

I assign four things at every important boundary:

1. the owner of the business meaning;
2. the owner of the technical contract;
3. the owner of the end-to-end outcome;
4. the authority that resolves a conflict.

Without those, a dependency can be documented and still remain unowned.

4. Standards fail when the compliant path is harder

An architecture standard can be correct and still be ineffective. If it is abstract, hard to discover, slow to approve, or unsupported in the delivery toolchain, teams will route around it.

“Use event-driven integration” is not an executable standard. A usable standard answers:

• Which event types and ownership rules apply?
• How are schema compatibility and sensitive data checked?
• What are the retry, ordering, and idempotency expectations?
• Which supported libraries, templates, and observability conventions exist?
• How does a team request a justified exception?
• Who upgrades the shared capability?

The strongest standards are delivered as paved roads: templates, APIs, tests, policy feedback, examples, and supported runtime capabilities. The Team Topologies material on organization dynamics frames cognitive load as an important design constraint. A standard that transfers all integration and operational complexity to the consuming team has not reduced system complexity; it has only moved it.

5. Missing system feedback allows drift to look successful

A local release can be green while the system becomes harder to change.

Traditional dashboards show latency, errors, availability, and resource use by service. System-level architecture needs additional signals:

• duplicated capabilities and business rules;
• cross-domain data ownership violations;
• synchronous dependency depth;
• change lead time across team boundaries;
• exception age and repeated exception patterns;
• contract-breaking changes and consumer recovery time;
• incidents where every component met its SLO but the journey failed;
• cost of changing one business rule across the estate.

Without these signals, architectural drift remains an opinion. By the time it appears as a major transformation programme, the organization has already paid for it through slower delivery, reconciliation, incidents, and coordination.

A synthetic example: the “simple” eligibility rule

Consider a synthetic eligibility rule used by customer onboarding, payments, lending, and card issuance. The business wants to change one threshold in six weeks.

Each domain originally implemented the rule locally because its first release had a different deadline. Every implementation is tested and owned. None is careless. Over time:

1. onboarding stores the rule in a workflow engine;
2. payments copies it into an API validator;
3. lending enriches it with a risk attribute;
4. cards evaluates it overnight in a batch;
5. reporting reconstructs the outcome from downstream data.

Now a single policy change requires five delivery plans, different effective dates, customer-state reconciliation, and a decision about historical treatment. The architecture problem is not duplicated code alone. It is duplicated authority.

I would not immediately replace every implementation with one central service. That can turn local duplication into runtime coupling. I would first establish a canonical policy owner, a versioned decision contract, common evidence, explicit latency and availability needs, and a migration sequence. Some domains may evaluate locally using the same signed policy version; others may call a shared capability. The system design follows the required authority and failure model—not a slogan about reuse.

Diagnose the environment before prescribing a target architecture

This table prevents a common failure: treating a structural problem as a training problem. Training helps when people do not know the practice. It does not repair a scorecard, funding model, or ownership gap that rewards the opposite behaviour.

The operating system I want around good teams

Make ownership end to end

Give stream-aligned teams a bounded domain and a customer or business outcome, not only a set of components. Publish the contracts they own, the decisions they may take independently, and the situations that require cross-domain resolution.

Use a named owner for journeys that cross teams. This role does not implement every component. It owns the coherence of the outcome, evidence, and unresolved trade-offs.

Turn principles into executable fitness functions

“Services must be resilient” is too vague to govern. An architecture fitness function checks a property continuously: dependency policy, schema compatibility, recovery objective, encryption, ownership metadata, deployment independence, or an allowed data flow.

Use automation for rules that can be tested. Keep human review for trade-offs that genuinely require judgement. Martin Fowler’s explanation of architecture decision records reinforces the complementary discipline: record the context, decision, alternatives, and consequences, then supersede the record when evidence changes rather than rewriting history.

Fund the shared work

Cross-team architecture work cannot survive as spare-time goodwill. Platform capabilities, migrations, contract improvements, and removal of high-cost exceptions need explicit capacity and owners.

I use an architecture investment backlog with evidence: recurring incidents, repeated exceptions, duplicated cost, long change paths, or control gaps. The backlog competes on business and operational impact, not architectural elegance.

Govern exceptions as information

An exception should record the unmet need, risk owner, compensating control, support model, expiry, and review trigger. Repeated exceptions are product discovery for the platform or evidence that the standard is wrong.

The goal is not zero exceptions. It is zero invisible exceptions.

Review outcomes, not diagrams

Architecture reviews should ask whether the environment produces the intended properties:

• Can teams deploy without coordinating routine changes?
• Can an incident owner explain an end-to-end failure?
• Can one business rule change without a programme?
• Can a team adopt the standard path without waiting for a specialist?
• Are system risks visible before they become transformation projects?
• Can an accepted exception be found, measured, and retired?

The diagram is a hypothesis. Delivery and runtime evidence decide whether it is true.

A 30-day intervention

I would start with one painful customer or operational journey, not an enterprise reorganization.

Week 1 — Map the evidence. Trace one recent change or incident across teams. Record waiting, decisions, data ownership, workarounds, and unresolved authority.

Week 2 — Name the seams. Assign business meaning, contract ownership, journey ownership, and escalation authority for the two or three boundaries creating most friction.

Week 3 — Make one standard executable. Replace one policy document with a template, automated check, working example, and exception route.

Week 4 — Change one feedback loop. Add a system-level measure—cross-team change lead time, exception age, duplicated rule count, or journey failure—and put one funded improvement on the roadmap.

This will not “fix the architecture” in a month. It will prove whether the organization can change the environment that keeps reproducing the same architecture.

My design rule

Good teams do not need more slogans. They need a technical environment where the locally easy decision and the systemically healthy decision are usually the same decision.

When they are not, the organization must make the trade-off visible: name the owner, preserve the rationale, measure the consequence, fund the repair, and learn from the exception. Architecture becomes coherent when the operating model produces coherence repeatedly—not when one architect draws a cleaner target state.

Related reading

• Architecture Decision Records That Teams Keep Using
• Platform Engineering Golden Paths Without Golden Cages
• Modernizing Core Banking Without a Big-Bang Rewrite

References

• Melvin Conway — How Do Committees Invent?
• DORA — Loosely coupled teams
• Team Topologies — Organization Dynamics mini-book
• Microsoft Cloud Adoption Framework — Build a cloud governance team
• Martin Fowler — Architecture Decision Record

A familiar target-state diagram puts channels at the top, APIs and orchestration in the middle, core banking and payments underneath, then data, AI, governance, and observability across the bottom. It is a useful inventory. It shows that a digital bank needs more than a mobile app and a core ledger.

But it is not yet an architecture a team can execute.

My test is simple: point at a customer action and ask who is authorised to decide, which system creates the financial fact, what happens after a timeout, which evidence proves the outcome, and how the bank recovers the critical service. If the diagram cannot answer those questions, its layers are categories—not contracts.

An AI-enabled bank needs two connected paths. The first turns customer intent into an authorised, durable outcome. The second turns observed outcomes into governed data and model proposals. AI may improve the decision, but it must not blur the boundary between a proposal and a permitted financial effect.

Start with the operation, not the platform

“Real-time, intelligent, scalable, secure, resilient” sounds right because no one wants the opposite. It is still too vague to guide design.

The Basel Committee’s operational-resilience principles start from critical operations and a bank’s ability to withstand disruptive events. That changes the architecture discussion. Instead of asking whether Kafka, Kubernetes, or a service mesh is highly available, I ask:

• Which customer or market outcome must continue?
• What level of disruption can the bank tolerate?
• Which people, processes, technology, data, facilities, and third parties support it?
• How does the bank detect that the operation is outside tolerance?
• What degraded mode is safe?

Take a card-limit increase requested through a conversational assistant. The critical outcome is not “the chatbot replies.” It is that an authenticated customer receives a correct, authorised decision; any approved limit is recorded once; the change is visible to fraud, servicing, statements, and audit; and the bank can explain or reverse the effect under the applicable rules.

That one operation crosses the channel, identity, product, decision, core-record, event, data, AI, and control concerns shown in a layer diagram. Designing each layer independently will not guarantee the outcome.

Replace boxes with six enforceable contracts

I use six contracts to turn the capability inventory into a delivery architecture.

1. The authority contract

The authority contract answers who or what may request an action, for which resource, under which purpose, scope, limit, and time window.

In the UAE, the CBUAE Open Finance Regulation makes the separation tangible. The framework includes an API Hub and Trust Framework; data sharing and service initiation are subject to explicit user consent, appropriate authentication, and secure communication. These are not decorative features of an API gateway. They are part of the business authority for an action.

The same reasoning applies inside the bank. NIST’s Zero Trust Architecture rejects implicit trust based only on network location. A request from an “internal” service is not authorised merely because it crossed the service mesh. User identity, workload identity, protected resource, requested operation, policy, and current context still matter.

An API gateway can validate a token and rate limit traffic. It cannot decide every product entitlement, financial limit, segregation-of-duties rule, or exceptional approval. The target state must show where those decisions live and how their evidence travels with the command.

2. The capability-ownership contract

A box labelled “microservices” says nothing about ownership. A banking capability needs a bounded vocabulary, commands it owns, invariants it enforces, records it controls, and service outcomes for which one team is accountable.

BIAN’s banking service landscape is useful here because it gives teams a banking-specific vocabulary for decomposing capabilities and service domains. It should inform the map, not become a reason to generate hundreds of services. The deployable boundary should follow a stable business responsibility and its consistency needs—not one rectangle per reference-model term.

For the limit-increase example, the assistant does not own the limit. A product or credit capability owns the command, validates the account state, applies the policy decision, enforces idempotency, and records the approved change. The model may help interpret the request or propose an amount. Ownership remains with the domain.

3. The financial-truth contract

Layer diagrams often use “single source of truth” beside a lake, warehouse, or customer-360 platform. In banking, that phrase becomes dangerous when it collapses different kinds of truth.

The ledger records balanced financial facts. A product system records contractual state. A payment rail supplies external settlement evidence. An event log records that a producer announced something. A warehouse provides governed analytical views. None is a universal replacement for the others.

The BCBS 239 principles require more than central storage: integrated taxonomies and architecture, metadata, identifiers, ownership, quality, and the ability to support risk-data aggregation during stress. A data lake icon does not satisfy those responsibilities.

For every important number, the architecture should identify its authoritative producer, business meaning, effective time, correction mechanism, lineage, and reconciliation control. “Customer balance” alone is not precise enough. Available balance, ledger balance, settled balance, and an analytical daily snapshot can be correct at the same time while answering different questions.

4. The event contract

An event-streaming layer can decouple producers from consumers, but it can also become a shared ambiguity bus.

An event needs a named owner, versioned schema, business key, ordering assumption, event time, publication guarantee, retention policy, privacy classification, and a replay contract. Consumers need to know whether the event reports an accepted command, a committed domain fact, or an observation that may later be corrected.

I avoid using an event as the only proof of a financial outcome. The domain first commits its durable state and an outbox record within the appropriate transaction boundary. Publication can then be retried. Consumers remain idempotent. Reconciliation compares the source record, emitted event, downstream record, and—where relevant—external rail evidence.

This is the difference between “event driven” as a transport preference and event-driven architecture as an explicit consistency model.

5. The AI-decision contract

Putting an “AI/ML layer” below the data platform suggests every model is a shared utility. The risk and latency profile actually depend on the decision.

Fraud scoring before authorisation, document classification during onboarding, a next-best-action recommendation, and a generative servicing assistant do not have the same authority, evidence, or failure mode. Each needs a decision contract:

The NIST AI Risk Management Framework treats trustworthiness as a lifecycle concern spanning design, development, use, and evaluation. The Financial Stability Board’s June 2026 report on sound practices for responsible AI adoption similarly proposes organisation-wide governance across the AI lifecycle. That FSB document is a consultation, not a final binding rule, but its framing is useful: adding a model endpoint is not the same as governing an AI-enabled capability.

My default boundary is therefore: the model proposes; deterministic controls and authorised domain capabilities create effects. A low-risk use case may automate within narrow delegated limits. A material or uncertain case may require human approval. Both paths should record the same decision evidence.

6. The evidence-and-resilience contract

Governance and observability should not be drawn only as foundations at the bottom. They must intersect the live path.

For one request, I want to correlate customer intent, consent, authentication, policy decision, domain command, idempotency key, model and prompt version, retrieval evidence, financial or business record, emitted events, downstream processing, and customer-visible outcome.

OpenTelemetry’s observability model describes traces, metrics, and logs as signals used to understand internal system state from its outputs. In a bank, technical telemetry needs business context without leaking sensitive data. A trace that proves an HTTP call completed is useful; a control record that proves which authorised command created a financial effect is different and equally necessary.

The contract also defines failure behavior. If the model is unavailable, does the bank fall back to deterministic rules, route to a person, or stop the operation? If an event stream is delayed, can the source capability continue safely? If the customer loses the connection after submission, can the channel retrieve the durable outcome without creating the command twice?

Walk the request from intent to evidence

Return to the limit-increase example.

1. The channel captures the customer’s intent and presents the required disclosure.
2. Identity, consent, session risk, and product scope establish what may be requested.
3. The assistant extracts a structured proposal. It does not call a database or mutate an account directly.
4. The decision service evaluates governed data and records the model, feature, rule, and evidence versions.
5. Deterministic policy checks eligibility, exposure, delegated limits, and whether human approval is required.
6. The product capability accepts one idempotent command, enforces its invariant, and records the result.
7. Events publish the committed outcome for fraud, servicing, notification, analytics, and regulatory processes.
8. Telemetry and control evidence show the end-to-end result; later outcomes return to a governed data product for evaluation and improvement.

Notice what is not in this sequence: an LLM with broad credentials choosing arbitrary tools. Tool access should be narrow, typed, policy-controlled, and auditable. The AI context should not contain reusable credentials or final authority.

A target state also needs a transition state

A beautifully layered end state can hide the hardest question: how does the bank move without breaking a critical operation?

I would not begin with a programme to “build the API layer,” “move to microservices,” or “create the AI platform.” I would choose one bounded operation with measurable pain and build a vertical slice.

For example:

1. Map the current limit-change journey, systems of record, approvals, failure modes, and control evidence.
2. Define the future authority, capability, truth, event, AI, and resilience contracts for that operation.
3. Put a stable API and idempotent command boundary around the current product system.
4. Add end-to-end identity, policy, telemetry, and reconciliation before changing the implementation behind the boundary.
5. Introduce the AI proposal path in shadow mode and compare it with approved outcomes.
6. Automate only the cases that meet evaluation and delegated-authority thresholds.
7. Prove recovery and degraded operation, then expand to the next capability.

This approach produces reusable platform capabilities, but each one is earned by a real journey. It also exposes whether the supposed target architecture supports delivery or merely reorganises the technology catalogue.

The architecture review I would run

I would ask every team to complete this scorecard for one critical operation:

If a row is answered with only a product name, the design is not finished.

The layer cake still has a place

Layered reference diagrams remain useful. They create a shared vocabulary, expose missing capabilities, and help explain investment areas. A central API, data, or AI platform can also reduce duplicated controls and improve consistency.

The trade-off is concentration. A universal orchestration layer can become the owner of no business outcome and the failure domain of every outcome. A customer-360 platform can become a stale copy mistaken for a system of record. A shared model gateway can standardise access while hiding use-case-specific evaluation and authority.

The answer is not to reject platforms. It is to make their contracts explicit and keep domain accountability visible.

My decision rule

An AI-enabled digital bank is not defined by the presence of an LLM box, an event bus, or a data lake. It is defined by how safely and observably it turns intent into a permitted, durable outcome—and how it learns from that outcome without surrendering authority or financial truth.

Use the layer diagram as the capability map. Use critical operations, contracts, sequences, failure modes, and evidence to create the architecture.

The model may propose. The bank must still authorise, execute, record, reconcile, explain, and recover.

Related reading

• Responsible AI Governance as an Engineering System
• Designing an Event-Driven Ledger That Auditors Can Trust
• Platform Engineering Golden Paths Without Golden Cages

References

• CBUAE — Open Finance Regulation: Introduction and Scope
• CBUAE — Open Finance Regulation
• Basel Committee — Principles for operational resilience
• Basel Committee — Principles for effective risk data aggregation and risk reporting
• NIST — Zero Trust Architecture, SP 800-207
• NIST — AI Risk Management Framework
• Financial Stability Board — Sound Practices for Responsible Adoption of AI, consultation report
• BIAN — Banking Industry Architecture Network
• OpenTelemetry — Observability primer

An API becomes useful to an AI system when the model can discover a small capability, supply valid arguments, and receive a result it can reason about. That does not mean handing an OpenAPI document and an administrator token to the model. I prefer a narrow MCP adapter that preserves the API's existing authorization and domain rules.

This guide builds that adapter with the official TypeScript MCP SDK, connects it to Codex or another MCP host, and shows the tool loop needed when Ollama or an open Kimi model is the model layer. The example API is synthetic, but the boundary is the one I use for real systems: the model proposes; the application validates and decides.

Start with the boundary, not the SDK

MCP defines three server capabilities: resources, tools, and prompts. A resource is readable context, a tool performs an action or query, and a prompt is a reusable interaction template. For an existing API, I normally begin with tools because they give the model a named operation with an explicit input schema.

The first design exercise is not “How do I convert every endpoint?” It is “Which business questions should the model be allowed to ask?” An internal API with 180 operations may need only three initial tools:

• orders.get_status reads one authoritative order;
• orders.search searches within the caller's tenant and a fixed time window;
• orders.prepare_cancellation validates a cancellation proposal without executing it.

I do not expose raw URLs, arbitrary HTTP methods, SQL, access tokens, or a generic call_api tool. A narrow tool gives me a stable contract, a smaller authorization surface, better descriptions for the model, and meaningful audit events.

Scaffold the MCP server

The current official TypeScript SDK uses separate server and client packages. A local stdio server needs Node.js 20 or later:

mkdir api-mcp && cd api-mcp
npm init -y
npm pkg set type=module
npm install @modelcontextprotocol/server zod tsx
mkdir -p src/plugins

Create src/index.ts. The API URL and token come from the process environment; they are never arguments the model can choose.

import { McpServer } from '@modelcontextprotocol/server';
import { serveStdio } from '@modelcontextprotocol/server/stdio';
import * as z from 'zod/v4';

const API_BASE = new URL(process.env.API_BASE ?? 'https://api.example.com');
const API_TOKEN = process.env.API_TOKEN;

if (!API_TOKEN) throw new Error('API_TOKEN is required');

const Order = z.object({
  id: z.string(),
  status: z.enum(['pending', 'confirmed', 'cancelled']),
  updatedAt: z.string()
});

function createServer(): McpServer {
  const server = new McpServer({ name: 'orders-api', version: '1.0.0' });

  server.registerTool(
    'orders.get_status',
    {
      title: 'Get order status',
      description: 'Read the authoritative status of one order visible to the caller.',
      inputSchema: z.object({
        orderId: z.string().regex(/^ord_[a-zA-Z0-9]+$/)
      }),
      outputSchema: Order,
      annotations: { readOnlyHint: true, idempotentHint: true }
    },
    async ({ orderId }) => {
      const controller = new AbortController();
      const timeout = setTimeout(() => controller.abort(), 5000);

      try {
        const url = new URL(`/v1/orders/${encodeURIComponent(orderId)}`, API_BASE);
        const response = await fetch(url, {
          headers: { Authorization: `Bearer ${API_TOKEN}` },
          signal: controller.signal
        });

        if (response.status === 404) {
          return { content: [{ type: 'text', text: 'Order not found.' }], isError: true };
        }
        if (!response.ok) {
          return {
            content: [{ type: 'text', text: `Upstream API failed with HTTP ${response.status}.` }],
            isError: true
          };
        }

        const order = Order.parse(await response.json());
        return {
          content: [{ type: 'text', text: JSON.stringify(order) }],
          structuredContent: order
        };
      } finally {
        clearTimeout(timeout);
      }
    }
  );

  return server;
}

void serveStdio(createServer);
console.error('orders-api MCP server running on stdio');

One Zod schema produces the JSON Schema advertised to clients and validates arguments before the handler executes. The output schema gives clients a machine-readable contract as well. I still parse the upstream response because an API returning malformed data should fail at the adapter boundary, not become confident model context.

The readOnlyHint and idempotentHint annotations improve client presentation; they are not security controls. The server and the upstream API must enforce identity, tenant scope, role, current state, rate limits, and domain invariants.

Add tools as explicit plugins

As the server grows, I keep each business capability in a small module. “Plugin” here means application code registered deliberately at startup—not arbitrary packages downloaded and executed because a model requested them.

// src/plugins/types.ts
import type { McpServer } from '@modelcontextprotocol/server';

export type ToolPlugin = {
  name: string;
  register(server: McpServer): void;
};

// src/plugins/index.ts
import { orderStatusPlugin } from './order-status.js';
import { customerSummaryPlugin } from './customer-summary.js';

export const plugins = [orderStatusPlugin, customerSummaryPlugin];

const server = new McpServer({ name: 'operations-api', version: '1.0.0' });
for (const plugin of plugins) plugin.register(server);

This layout gives every plugin an owner, tests, schemas, endpoint allowlist, authorization policy, and version. A plugin registry can load configuration, but I avoid scanning an untrusted directory and importing whatever file appears there.

Test before connecting a model

The official MCP Inspector runs through npx and lets you list tools, inspect schemas, and call handlers directly:

API_BASE=https://sandbox.example.com \
API_TOKEN=replace-locally \
npx @modelcontextprotocol/inspector npx tsx src/index.ts

Test valid calls, malformed identifiers, unauthorized records, timeouts, unexpected upstream JSON, and concurrent requests. Verify that logs contain trace identifiers and decisions but never raw bearer tokens or excessive personal data.

For stdio, stdout belongs to JSON-RPC. A single console.log can corrupt the protocol stream, so operational logging goes to stderr or a separate sink.

Register it in Codex or another host

Codex can register the same local command and pass environment variables to the child process:

codex mcp add orders-api \
  --env API_BASE=https://sandbox.example.com \
  --env API_TOKEN=replace-locally \
  -- npx tsx /absolute/path/api-mcp/src/index.ts

Do not commit the real token or place it in an article, shell history, or model prompt. In production, prefer a short-lived workload identity or secret manager and bind authorization to the authenticated user rather than one shared super-token.

VS Code uses the same command in .vscode/mcp.json:

{
  "servers": {
    "orders-api": {
      "type": "stdio",
      "command": "npx",
      "args": ["tsx", "src/index.ts"]
    }
  }
}

The host performs tools/list, gives the names, descriptions, and schemas to the model, then sends the selected tools/call to the MCP server. The model does not call the REST API directly.

Where Ollama fits

Ollama exposes a local chat API at http://localhost:11434/api/chat and supports tool definitions and tool-call responses. Ollama is the model runtime, not the MCP server. A host or a small bridge must translate MCP tools into the function schema sent to Ollama, execute selected calls through the MCP client, append tool results to the conversation, and ask the model to continue.

The core loop looks like this:

import { Client } from '@modelcontextprotocol/client';
import { StdioClientTransport } from '@modelcontextprotocol/client/stdio';

const mcp = new Client({ name: 'ollama-host', version: '1.0.0' });
await mcp.connect(new StdioClientTransport({
  command: 'npx', args: ['tsx', 'src/index.ts']
}));

const { tools } = await mcp.listTools();
const ollamaTools = tools.map(tool => ({
  type: 'function',
  function: {
    name: tool.name,
    description: tool.description,
    parameters: tool.inputSchema
  }
}));

const response = await fetch('http://localhost:11434/api/chat', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({
    model: process.env.OLLAMA_MODEL ?? 'qwen3:8b',
    messages: [{ role: 'user', content: 'What is the status of ord_1042?' }],
    tools: ollamaTools,
    stream: false
  })
});

const turn = await response.json();
for (const call of turn.message.tool_calls ?? []) {
  const result = await mcp.callTool({
    name: call.function.name,
    arguments: call.function.arguments
  });
  // Append the assistant turn and this tool result, then call Ollama again.
}

A production host adds the complete message loop, cancellation, maximum tool-call depth, per-tool approval, result-size limits, telemetry, and a hard allowlist. Never dispatch call.function.name into eval, a shell, or a dynamic import.

Where open Kimi fits

Moonshot AI publishes Kimi model code and weights under a Modified MIT licence. Kimi K2 is designed for tool use, but its published architecture has one trillion total parameters with 32 billion activated parameters. Open weights do not mean laptop-scale. Running the full model locally requires serious inference infrastructure; Moonshot documents engines such as vLLM and SGLang.

If an Ollama installation exposes a Kimi model with tool support, the bridge above changes only OLLAMA_MODEL. If Kimi is served through an OpenAI-compatible endpoint, the same MCP client can sit behind an OpenAI-compatible host loop. The MCP server itself does not change because model selection belongs above the tool boundary.

For a genuinely local developer workstation, start with a smaller Ollama model marked as tool-capable. Use Kimi when its serving footprint and operating model fit the environment. Do not market a cloud-routed model as locally hosted simply because the command starts with ollama.

What this architecture actually costs

MCP itself does not add a licence fee. The official SDK is open source, and the adapter in this guide is a small process. The material costs sit around it: remote compute, model inference, logs, secrets, identity, testing, and the engineering time needed to keep tool contracts aligned with the API.

The following prices were checked on 21 July 2026 and are reference points, not quotations. They exclude tax, storage, observability, managed identity, network charges, support, and staff time.

The choice is therefore not “free versus paid.” It is where the cost and operational responsibility land. A local Ollama prototype can have no new invoice and still consume a workstation. A $4 VM can run a lightweight adapter, but not a large open-weight model. A remote MCP endpoint also needs TLS, workload identity, secret rotation, rate limiting, telemetry, patching, and an availability design.

A cost model I can defend

For a production estimate, I separate fixed platform cost from usage-driven model cost:

monthly cost = adapter compute
             + identity, secrets, logs, and storage
             + (input tokens / 1,000,000 × input rate)
             + (output tokens / 1,000,000 × output rate)
             + operational ownership

Consider an illustrative workload of 50,000 tool-assisted turns per month, averaging 2,000 input tokens and 500 output tokens per turn. That is 100 million input tokens and 25 million output tokens. At hypothetical rates of $0.60 per million input tokens and $2.50 per million output tokens, inference would be:

Those token prices are deliberately illustrative so the method remains valid when providers change their rates. Replace them with the current rate for the exact model and region, then measure real prompts rather than trusting a demo average. Tool descriptions, schemas, conversation history, retries, and repeated agent loops all consume tokens. Cache discounts and batch rates should be modelled separately, not assumed.

For self-hosted inference I use a different denominator: cost per successful, policy-compliant task, not cost per token. I include amortized hardware or GPU rental, electricity, average and p95 latency, concurrency, utilization, failed tool loops, on-call work, and the spare capacity required during maintenance. Open weights remove a usage invoice; they do not remove capacity planning.

Measure value and failure together

A research-grade pilot needs a baseline and a holdout. I compare the MCP-assisted workflow with the current API or manual workflow on the same task set and record:

I would not approve a production rollout from answer-quality scores alone. The evidence package should include the task set, model and prompt versions, tool-schema versions, raw outcome categories, cost assumptions, excluded costs, failure examples, and the date on which provider prices were retrieved.

Production controls I would not skip

Before exposing a real API, I require:

1. A capability allowlist. No generic URL, method, SQL, shell, or unrestricted search tool.
2. Two validation layers. MCP schema validation at the adapter and domain validation in the API.
3. Caller-bound authorization. Tenant, role, legal entity, record scope, and purpose remain explicit.
4. Separate read and write tools. A read tool cannot become a write through a hidden query parameter.
5. Preparation before material effects. High-impact changes produce a reviewable proposal before execution.
6. Idempotency and state checks. Retries cannot create duplicate effects, and stale proposals fail closed.
7. Output minimization. Return only fields needed for the model's task; redact secrets and unnecessary personal data.
8. Bounded execution. Deadlines, concurrency limits, response-size limits, and maximum tool-call depth.
9. Evidence. Record authenticated subject, tool, schema version, decision, trace ID, result class, and timing.
10. Versioned contracts. A breaking domain change creates a new tool or contract version.

The patterns in AI Agents in Regulated FinTech, Responsible AI Governance as an Engineering System, and Resilient Payment APIs apply even when the example is not financial. MCP gives the model a clean interface; it does not absolve the system from engineering discipline.

The design test

I consider the adapter ready when I can replace Ollama with Kimi—or replace both with another host—without changing authorization, endpoint mapping, domain rules, or audit evidence. That is the practical value of MCP: the model layer can evolve while the capability boundary stays deliberate.

References

• Model Context Protocol — Introduction
• Model Context Protocol — Build an MCP Server
• Official MCP TypeScript SDK
• MCP TypeScript SDK — Tool Contracts
• Model Context Protocol — Inspector
• Ollama API — Tool Calling
• Ollama — Pricing
• Moonshot AI — Kimi K2 Repository
• Moonshot AI — Kimi K2 Tool-Calling Guidance
• DigitalOcean — Droplet Pricing
• Cloudflare Workers — Pricing

UAEFTS is easy to describe as “the UAE’s RTGS.” That description is correct, but it is not enough to design a production integration. The engineering problem begins when a bank must turn a customer or treasury instruction into an individually settled transfer, manage liquidity while the instruction is in flight, distinguish settlement from beneficiary posting, and prove the outcome later.

My rule is simple: never let one generic SUCCESS hide the boundaries between instruction acceptance, queueing, settlement in central-bank money, participant confirmation, and account posting.

UAEFTS is settlement infrastructure, not a payment API

The Central Bank of the UAE describes UAEFTS as the country’s Real Time Gross Settlement system, owned, hosted, and managed by CBUAE. It has operated since 2001 and transfers funds between participating banks and financial institutions through their accounts at the central bank.

“Real time” means instructions are processed continuously during the system’s operating window. “Gross” means transfers are settled individually rather than offsetting debits and credits into a net position. Those properties define a financial-market infrastructure. They do not mean every channel, compliance decision, core-banking posting, or beneficiary notification completes in one synchronous request.

CBUAE classifies UAEFTS as a Large-Value Payment System, while its other infrastructures serve retail-payment purposes. Yet UAEFTS activity is not limited to a small set of exceptional corporate payments. The CBUAE Annual Report 2025 reports 114.9 million retail transfers worth approximately AED 9.9 trillion and 865,708 institutional transfers worth approximately AED 14.5 trillion through UAEFTS in 2025. At that scale, operational correctness matters as much as raw throughput.

Follow one transfer through the boundaries

I model an outbound UAEFTS transfer as a sequence of explicit proofs.

1. The participant accepts the instruction

The originating institution authenticates the caller, validates the account and mandate, applies sanctions and financial-crime controls, checks product and participant rules, and reserves a durable internal operation identifier. Customer acceptance proves that the institution has taken responsibility for processing the request. It does not prove that UAEFTS has received or settled it.

The operation should become immutable enough to retry safely: amount, currency, debtor, creditor, value date, purpose and permitted remittance data, initiating channel, approvals, and the relevant external references. Corrections should create a controlled replacement or cancellation path rather than silently changing a submitted instruction.

2. The gateway submits through the certified channel

CBUAE states that UAEFTS supports straight-through processing using prescribed formats over SFTP. Participants can also enter transactions through the restricted Transactions and Enquiry Portal, or TEP. I therefore keep the payment domain separate from the transport adapter.

The adapter serializes the approved instruction into the required market format, signs or packages it as required, sends it through the certified connection, and records a content hash, transmission time, file or message reference, and technical acknowledgement. A successful upload is transport evidence—not settlement evidence.

3. UAEFTS validates, orders, or queues the transfer

The rail may reject an invalid request or queue an otherwise valid instruction for processing. The public CBUAE description says participant requests are queued for immediate action and settled continuously according to the governing rules. The bank’s state model should therefore preserve at least submitted, technically_acknowledged, accepted_for_processing, queued, settled, rejected, and unknown as distinct concepts, even if the exact external codes come from the applicable UAEFTS specification.

Queueing deserves special treatment because an RTGS system consumes liquidity transfer by transfer. It is not a harmless synonym for “pending.” A growing queue can indicate a participant liquidity problem, a counterparty pattern, an operational incident, or an approaching cut-off risk.

4. Settlement moves balances at CBUAE

The decisive interbank event is the individual debit and credit across participant accounts held with CBUAE. Once the rail reports settlement under its rules, the originating institution must stop treating the instruction as a retryable outbound command. A second submission is a second financial instruction unless the scheme explicitly identifies it as the same operation.

That is why idempotency must span the business command, the generated UAEFTS instruction, the outbound artefact, and the returned evidence. A timeout after submission is an unknown outcome. The correct response is to query or reconcile the original reference—not create a replacement payment because the caller did not receive an answer. The same principle appears in my article on resilient payment APIs.

5. Confirmation and beneficiary posting complete different obligations

CBUAE says the system sends confirmation to affected participants after processing. The receiving institution still has its own work: correlate the incoming transfer, run the controls required for that context, post the beneficiary account, notify the customer, and expose the entry through statements or transaction history.

Interbank settlement and beneficiary credit should therefore have separate timestamps and references. Customer support needs to know whether funds settled to the receiving institution but remain unmatched or unposted. Finance needs the same distinction to reconcile settlement-account movement against customer-account entries.

Liquidity is part of the application design

RTGS integrations fail when teams treat liquidity as somebody else’s dashboard. Each individually settled debit changes the participant’s available position. A payment service needs visibility into queued value, settlement-account capacity, urgent-payment priority, expected incoming funds, cut-off exposure, and aged instructions.

CBUAE’s Intraday Liquidity Facility gives eligible UAEFTS participants access to collateralised dirham funding to manage unforeseen disruption or gridlock. That facility is a liquidity-management tool, not an automatic application retry. The bank still needs an operating decision: which queued instructions are urgent, which collateral and approvals are available, who may invoke a facility, and how every decision is recorded.

I expose liquidity as an explicit dependency to orchestration. The payment engine may request priority, hold a lower-priority instruction, or escalate an aged queue item, but it must not manufacture settlement capacity or hide the reason for delay.

Build an evidence ledger beside the money ledger

For every UAEFTS instruction, I retain an append-only evidence chain:

• the approved business instruction and its payload fingerprint;
• the market-format artefact actually transmitted;
• technical acknowledgements and delivery attempts;
• UAEFTS processing status with the raw code and interpretation version;
• settlement reference, timestamp, amount, currency, and affected participant accounts;
• receiving-side posting or return evidence when available;
• operator actions, investigations, and corrections.

The money ledger records financial postings. The evidence ledger explains why those postings exist and which external event justified each one. They are linked, but one should not overwrite the other.

This makes reconciliation deterministic. At minimum, I reconcile four populations: approved outbound instructions, artefacts sent, UAEFTS outcomes, and settlement-account entries. On the inbound side, I reconcile rail confirmations to beneficiary postings and exceptions. A perfectly matched set can still be incomplete, so control totals must cover count, value, currency, business date, source sequence, and delivery completeness. This is the same discipline described in reconciliation by design.

Treat ISO 20022 as a controlled migration

The CBUAE Rulebook includes an in-force annex on the transition from SWIFT to the ISO 20022 payment standard, effective 7 November 2025, and places UAEFTS within the UAE payment-system landscape affected by the change.

That does not justify replacing one file parser and declaring the platform “ISO 20022 ready.” The canonical model must preserve party identities, account data, agent roles, purpose, remittance information, identifiers, and status reasons without truncating them into a legacy schema. The adapter must then apply the exact UAE market profile and version in force.

I version three things independently:

1. the internal payment model;
2. the UAEFTS usage profile and validation rules;
3. the transport and security configuration.

That separation lets a bank introduce a new message version without rewriting its ledger or customer journey. It also prevents the common mistake of treating syntactically valid XML as a valid, authorised, fundable payment. ISO 20022 is a domain model, but the scheme profile remains the executable contract.

Controls I would require before production

• Every submission has a stable business idempotency key and a unique scheme instruction identity.
• Transport acknowledgement, processing acceptance, settlement, and beneficiary posting are stored separately.
• A timeout becomes unknown, followed by enquiry and reconciliation—not blind resubmission.
• Queue age and value are monitored by priority, participant, and cut-off exposure.
• Operators cannot edit a settled instruction; corrections use governed return or adjustment flows.
• Settlement-account movements reconcile to UAEFTS evidence and internal ledger entries.
• Manual TEP activity is imported into the same control population as straight-through traffic.
• Message-profile versions and mapping changes are tested with golden files and negative cases.
• Production access satisfies CBUAE licensing, network, application-certification, and participant-rule requirements.

The architecture decision

I would not hide UAEFTS behind a method called sendMoney() that returns a boolean. I would build a payment state machine around durable evidence and place settlement at its own boundary.

The originating instruction, secure delivery, rail acceptance, liquidity queue, CBUAE settlement, receiving-bank posting, and reconciliation result each answer a different question. When those facts remain separate, the platform can recover from timeouts, manage liquidity, explain delays, prevent duplicates, and prove exactly where the money is.

Related reading

• ISO 20022 Payment Lifecycle: From pain.001 to camt.054
• How a Payment Moves Through ISO 20022
• Reconciliation by Design, Not by Spreadsheet
• Resilient Payment APIs: Timeouts, Retries, and Unknown Outcomes

References

• Central Bank of the UAE — UAE Fund Transfer System
• Central Bank of the UAE — Payments and Settlements
• Central Bank of the UAE — Annual Report 2025
• Central Bank of the UAE — Domestic Market Operations and Intraday Liquidity Facility
• CBUAE Rulebook — Transition from SWIFT to ISO 20022
• CBUAE Rulebook — Large Value and Retail Payment Systems Regulations

When I compare pacs.008 with pacs.009, I do not start with their XML. I start with the financial reason for moving value. A pacs.008 carries a customer credit-transfer instruction between financial institutions. An ordinary pacs.009 carries a credit transfer in which the debtor and creditor are financial institutions. A pacs.009 COV is a specific cover use of pacs.009: it moves funds for one customer transfer already sent separately in pacs.008.

That distinction matters when a platform must choose a route, screen the right parties, correlate two message legs, or explain why an instruction arrived before its cover funds. A wrong choice changes the represented business function and can create routing, compliance, liquidity, and reconciliation failures.

The short answer

I use this decision table before discussing fields:

The official ISO 20022 message catalogue names pacs.008 FIToFICustomerCreditTransfer and pacs.009 FinancialInstitutionalCreditTransfer. The CPMI harmonised requirements include both in the core cross-border credit-transfer set and make “use the appropriate message for a particular business function” the first fundamental requirement.

The rule is therefore not “customer-facing channel means pacs.008” or “bank network means pacs.009.” Both messages travel between financial institutions. The difference is whose underlying credit transfer the message represents and, for cover, whether the message is the customer instruction or the funding leg.

Name the version and profile

“We support pacs.008” is incomplete. As of this review, the global catalogue lists pacs.008.001.14 and pacs.009.001.13. The February 2026 CPMI technical annex models pacs.008.001.13 and pacs.009.001.12. The April 2026 PMPG cover guidance discusses the CBPR+ pacs.009.001.08 COV usage guideline.

These are different layers and release cycles: the global message definition, a harmonised cross-border data model, and a network market-practice profile. The ISO 20022 implementation FAQ notes that implementation depends heavily on the adopting community, so schemes can constrain the same base message differently.

For every integration, I record:

• message identifier and namespace;
• market-practice or scheme usage guideline and release;
• permitted settlement methods and agents;
• validation rules beyond XSD;
• supported identifiers, code-list snapshot, and character rules;
• bilateral restrictions and activation date.

Schema validity proves structure against one XSD. It does not prove that the message is valid for the selected route or profile. The deeper implementation model is covered in ISO 20022 as a Domain Model; here the important point is that routing must select a versioned business contract, not only a serializer.

Serial: pacs.008 follows the customer-payment chain

Consider a synthetic EUR 250,000 supplier payment. A debtor instructs Bank A to credit a supplier at Bank C. Bank A has no direct settlement relationship with Bank C for the route, so Bank B acts as an intermediary.

In the serial model, Bank A sends pacs.008 to Bank B, and Bank B forwards the customer credit transfer to Bank C according to the applicable profile and account relationships. The customer parties, agents, amount, remittance, purpose, and transaction identifiers remain part of the customer-payment instruction as it advances. The Swift CBPR+ customer-payment material explicitly separates this serial pacs.008 method from the cover method.

I represent each hop as part of one end-to-end payment but not as one database update. Each hop can be received, validated, held, rejected, settled, or forwarded at a different time. The platform keeps the end-to-end identity and the current instructed/instructing agents visible while preserving prior-agent evidence required by its profile.

Serial flow is simpler to correlate because instruction and settlement follow the same chain, but it is not automatically faster or safer. Every intermediary adds a boundary for cut-off times, liquidity, screening, fees, transformation, and exceptions.

FI-to-FI: ordinary pacs.009 represents a bank transfer

Now remove the supplier. Bank A wants to transfer its own funds to Bank C for an eligible treasury purpose. The debtor and creditor are financial institutions; there is no underlying customer credit transfer to fund. This is the ordinary pacs.009 business function.

A pacs.009 can pass through intermediaries and its settlement can fail, but it does not become customer payment data because an operator attaches a customer reference. Conversely, a pacs.008 is not the right treasury message merely because its XML can carry agents and amounts.

This boundary matters in the canonical model. I keep customer_credit_transfer and financial_institution_credit_transfer as separate domain concepts with separate allowed parties, purposes, routing policies, and validations. A generic credit_transfer table can share infrastructure, but it must not erase the business type.

Cover: one payment, two linked paths

Return to the supplier payment. Bank A can send the pacs.008 customer instruction directly toward Bank C while using reimbursement agents to move the funds. The customer instruction and settlement now travel on different paths:

1. The Direct leg is the pacs.008 from the debtor-agent side toward the creditor agent.
2. The cover leg is one pacs.009 COV through the correspondent or reimbursement chain.

The Swift ISO 20022 Programme User Handbook documents the pacs.008 settlement method and uses COVE to indicate settlement through a covering pacs.009. The current PMPG Cover Payments guideline is stricter about purpose: pacs.009 COV covers one separately sent pacs.008 customer credit transfer. It is not an alternative to pacs.008, and an ordinary pacs.009 must not be used to disguise that customer-cover function.

The COV usage is identified by the underlying customer credit-transfer component. That component copies selected information from the pacs.008, including the customer parties required by the profile, and the guidance says those elements must remain unchanged through the cover chain. Intermediaries need that underlying information for screening; replacing it with placeholders or unrelated data defeats the transparency the variant was designed to provide.

The Direct and cover legs need deterministic correlation. PMPG guidance describes the UETR as linking pacs.008 and pacs.009 COV and says the related camt.054 credit notification should reference it. I also correlate message identifiers, amount, currency, agents, value date, and the immutable customer-data fingerprint. The UETR is evidence, not permission to ignore contradictory fields.

Model two states, then derive the customer outcome

A cover payment is where weak state models become dangerous. Receiving the Direct pacs.008 does not mean the cover funds have settled. Receiving a pacs.009 COV does not prove that the beneficiary was credited. A transport acknowledgement is not a business acceptance, and a credit advice is not the same thing as end-of-day reconciliation.

I keep at least two linked state machines:

• instruction state: received, validated, held, accepted, rejected, cancelled, or completed under the adopted profile;
• cover state: expected, received, screening, settled, delayed, rejected, returned, or unmatched.

The customer-facing state is derived from policy and evidence across both. Whether Bank C credits before cover receipt is not a universal ISO 20022 rule; it depends on legal interpretation, scheme rules, account relationships, credit policy, and the selected service. The platform must record that decision instead of burying it inside an adapter.

PMPG guidance calls for delay or rejection of the cover leg to be communicated quickly and identifies pacs.002, tracking updates, cancellation, return, and investigation paths depending on the situation. For the API side, I apply the same uncertainty rule described in Resilient Payment APIs: a timeout means the caller does not know, not that money did not move.

Failure modes I test before launch

The happy path is easy to draw. These are the cases that reveal whether the model is correct:

1. Ordinary pacs.009 used as cover. The funds arrive without the required underlying customer context. Stop and investigate; do not infer a COV variant from an operator note.
2. pacs.009 COV has no matching pacs.008. Hold it as unmatched evidence, alert operations, and follow the applicable status or investigation process.
3. Direct and cover data disagree. Compare UETR, amount, currency, agents, value date, and the underlying customer fingerprint. Never silently overwrite one leg with the other.
4. Direct arrives before cover. Represent “instruction received, settlement pending” explicitly. Apply the institution's approved credit policy.
5. Cover settles after Direct rejection or cancellation. Route the case into governed exception handling; do not post it as a new unrelated FI transfer.
6. A retry creates duplicate legs. Deduplicate each message in its defined scope and enforce the one-pacs.008-to-one-pacs.009-COV relationship for this use case.
7. A router switches serial to cover after submission. Reject the mutation. A different settlement model creates new instructions and evidence; it is not an invisible adapter fallback.

Ledger postings should follow confirmed financial events, and corrections should be new entries rather than edits to history. Designing an Event-Driven Ledger explains that accounting boundary in more detail.

My implementation checklist

Before enabling a route, I ask:

• Is the underlying movement a customer transfer or an FI's own transfer?
• If customer-related, are instruction and settlement serial or separated into Direct and cover paths?
• Which exact message version and usage guideline govern the route?
• Does pacs.008 declare the permitted settlement method for that profile?
• If cover is used, is there exactly one corresponding pacs.009 COV with preserved underlying customer data?
• Which identifiers and financial fields must match across both legs?
• Which system owns instruction state, cover state, beneficiary-credit policy, and reconciliation evidence?
• What happens when either leg is delayed, rejected, duplicated, returned, or unmatched?
• Can operations explain the payment without reading raw XML across several systems?

The durable design rule is simple: choose by business function, then make the route a first-class domain decision. pacs.008 is the customer instruction. pacs.009 is the FI transfer. pacs.009 COV is the linked cover leg for one separately sent pacs.008. Once those meanings are explicit, mapping, status processing, reconciliation, and incident handling become testable instead of interpretive.

References

• CPMI — Harmonised ISO 20022 data requirements, updated report
• CPMI — Harmonised ISO 20022 data models, technical annex
• ISO 20022 — Message Definitions catalogue
• ISO 20022 — Frequently asked questions
• PMPG — Cover Payments Market Practice Guideline, version 4.1
• Swift — CBPR+ Customer Payments
• Swift — ISO 20022 Programme User Handbook

When a corporate payment reaches a bank, the dangerous question is, “Has it been processed?” That word can mean the file arrived, the instruction passed validation, the debtor account was reserved, the interbank message was accepted, settlement completed, the creditor posted the beneficiary account, or the customer received a notification. Those are different events, owned by different systems.

I prefer to follow one synthetic payment end to end. A treasury team sends a supplier payment to its bank. The bank validates it, creates an interbank instruction, sends it through a clearing or RTGS service, and the creditor agent posts the beneficiary account. Reporting then returns enough evidence to reconcile the original intent with the financial outcome.

The familiar shorthand is pain.001 → pacs.008 → camt.054. It is useful, but it is not a universal recipe. The adopted scheme, market infrastructure, message version, and usage guideline determine the actual sequence and rules. My design rule is stricter: every transition needs its own proof, authority, and timestamp.

The lifecycle is an evidence ledger

I do not let one mutable payment_status column carry this journey. I keep an append-only evidence record and derive the customer-facing state from policy. That distinction is what prevents a late callback from turning “settled” back into “processing,” or a transport acknowledgement from becoming “paid.”

The table is deliberately scheme-neutral. A usage guideline may use a different message, status vocabulary, or event order. The proof boundary remains useful because it asks the same operational question at every step: which system is authoritative for this claim?

ISO 20022 names contracts, not one payment rail

The ISO 20022 catalogue separates Payments Initiation, Payments Clearing and Settlement, and Cash Management. That structure is important. pain.001 is a customer-to-bank initiation message. pacs.008 is an FI-to-FI customer credit-transfer instruction. pacs.002 reports processing status. camt.054 is a bank-to-customer debit or credit notification.

These messages describe business communication. They do not create one global settlement system. The ISO implementation FAQ explains that an adopting community defines how it uses the standard. A domestic instant-payment scheme, an RTGS service, and a cross-border correspondent route can therefore constrain the same message family differently.

Before building an adapter, I write down the executable contract:

• exact message identifier and namespace;
• scheme or market-practice release;
• mandatory and forbidden elements beyond the XSD;
• code-list and character-set rules;
• duplicate scope and identifier lifetime;
• accepted status transitions;
• cut-off, settlement, return, and investigation rules.

That is why ISO 20022 as a Domain Model matters more than a library that serializes XML. A valid document can still be invalid for the selected route.

Step 1: pain.001 captures customer intent

In the synthetic case, a corporate treasury submits a payment initiation to its debtor agent. The pain.001 carries the requested execution, debtor and creditor information, amount, currency, remittance data, and references permitted by the bank’s channel profile.

Receipt is the first proof, not the outcome. The channel should assign an immutable intake identifier, retain the original payload or its governed evidence, authenticate the sender, and make retries idempotent. If a client times out after submission, the system must let it query the original request instead of encouraging a second payment.

The bank then validates more than XML. It checks entitlement, mandate, account state, funds or limits, party data, sanctions and compliance controls, cut-off policy, and route availability. Some profiles return customer-payment status through a pain.002; others expose equivalent status through an API or channel workflow. Whatever the transport, I model the result as received, accepted, held, or rejected with reason and timestamp.

Step 2: acceptance creates an interbank obligation

After acceptance, the debtor agent maps the customer instruction into its canonical payment model and selects a route. For a customer credit transfer between financial institutions, that route can produce a pacs.008 under the adopted clearing or network profile.

This mapping is not a field-copy exercise. The service must preserve end-to-end references, party roles, amount and currency meaning, charge arrangement, remittance, purpose, and agent chain. It must also record which source value produced each outbound value. That lineage is what makes an exception explainable later.

The CPMI’s updated 2026 harmonisation requirements address interbank payments, clearing and settlement, returns, and investigations, while preserving the need for consistent implementation across communities. I take the same approach internally: one payment aggregate can connect the journey, but every message and financial event keeps its own identity, version, and provenance.

Step 3: the rail validates, orders, and settles

The clearing or RTGS service validates the message against its own usage guideline. It can reject the instruction, accept it for later processing, queue it for liquidity, order it, or settle it according to its rules. A pacs.002 or rail-specific event may report part of this progression.

The key engineering rule is simple: a processing status is evidence of exactly the state it names. It is not automatically proof that central-bank money moved, that correspondent accounts were posted, that the beneficiary was credited, or that the customer statement reconciles.

I store the rail message identifier, business status, reason, settlement reference, value date, timestamp, source system, and usage-guideline version separately. Then I derive a payment-level state from an explicit policy. This avoids the common anti-pattern where the last arriving callback overwrites a more authoritative financial event.

A compact implementation can model the evidence like this:

create table payment_evidence (
  payment_id          uuid        not null,
  evidence_type       text        not null,
  source_system       text        not null,
  source_reference    text        not null,
  business_status     text        not null,
  occurred_at         timestamptz not null,
  recorded_at         timestamptz not null default now(),
  payload_hash        text        not null,
  profile_version     text        not null,
  primary key (source_system, source_reference, evidence_type)
);

The primary key is illustrative, not universal. The adopted route defines the real duplicate scope and identifier lifetime. The important property is that a retry either finds the same evidence or adds a new, separately identifiable event; it never silently edits financial history.

Payment infrastructure documentation makes this profile boundary concrete. The Fedwire ISO 20022 technical FAQ, for example, directs participants to the service-specific format and validation documentation. Supporting “ISO 20022” without naming the market profile is not a testable claim.

Step 4: beneficiary credit is another financial event

Once the creditor agent receives the instruction and the settlement evidence required by its operating model, it validates the beneficiary account and applies its posting policy. The beneficiary-credit timestamp can differ from the interbank-settlement timestamp. Legal finality, availability of funds, screening, account restrictions, and local operating rules all matter.

The platform therefore needs separate facts for:

1. instruction accepted by the debtor agent;
2. interbank instruction accepted by the rail;
3. settlement completed under the rail or account model;
4. creditor agent received the instruction;
5. beneficiary account posted or rejected;
6. customer notification produced;
7. reconciliation completed.

Collapsing these into SUCCESS is convenient until the first dispute. After that, the missing distinctions become an operational and accounting problem.

Step 5: camt.054 reports a debit or credit

A camt.054 can notify a customer of a debit or credit entry. In this example, the creditor agent uses it to report the beneficiary credit, while a debtor-side notification can report the debit under the applicable service.

The European Payments Council recommendation on customer reporting emphasizes ISO 20022 cash-management reporting and reference continuity for reconciliation. That is the real value of structured reporting: the notification should carry enough stable identity to match the booked entry to the instruction and interbank evidence.

camt.054 is not a magic close signal. A notification can be duplicated, delayed, corrected, or unmatched. The reconciliation service checks identifiers, account, amount, currency, debit or credit indicator, booking and value dates, and any scheme-specific references. It closes the payment only when the expected financial evidence agrees.

The broader control pattern is explained in Reconciliation by Design: exceptions must remain visible, and corrections must produce new evidence rather than rewrite history.

Failure paths I test before the happy path

I design these cases before launch:

• Duplicate pain.001: return the existing payment identity within the idempotency scope; do not create another obligation.
• Channel timeout after acceptance: expose queryable state and do not tell the client to resend blindly. The uncertainty model in Resilient Payment APIs applies here.
• Business rejection after transport acceptance: retain both events. The transport worked; the payment did not pass the business rule.
• pacs.008 accepted but settlement delayed: show accepted / settlement pending, not paid.
• Settlement recorded but beneficiary posting fails: open a governed exception with the creditor-side reason and ownership.
• camt.054 arrives twice: deduplicate the notification without suppressing a genuine later correction.
• References disagree: stop automatic closure and investigate; never force a match using amount alone.

Each failure has an owner, a deadline, a safe retry rule, and a customer-facing explanation. A payment platform is not reliable because every message succeeds. It is reliable because uncertainty cannot silently become a second debit or a false completion.

My implementation checklist

Before enabling a payment route, I ask:

• Which business event does each message represent?
• Which exact usage guideline and version apply?
• What proof authorizes the next transition?
• Which system owns receipt, validation, settlement, posting, notification, and reconciliation?
• Which identifiers survive every transformation?
• How long does each idempotency and duplicate key remain valid?
• Can a late event move state backward or overwrite stronger evidence?
• What does the customer see during a timeout, hold, or partial completion?
• Can operations explain the payment without reconstructing it from raw XML?

The durable design is not a straight line of acronyms. It is a chain of explicit business contracts and evidence. pain.001 captures intent, pacs.008 carries the interbank customer transfer, status messages report defined processing outcomes, the settlement system records its financial result, the creditor agent records beneficiary posting, and camt.054 supports customer reporting. Reconciliation joins those facts. Only then should the platform say the payment is complete.

References

• ISO 20022 — Message Definitions
• ISO 20022 — Frequently asked questions
• CPMI — Harmonised ISO 20022 data requirements, updated 2026
• European Payments Council — Customer-reporting recommendation
• Federal Reserve Financial Services — Fedwire ISO 20022 technical FAQs
• Swift — ISO 20022 for payment instructions

I define a golden path as a supported, observable route through common engineering work. It succeeds when teams choose it because it removes friction—not because an internal platform makes every alternative impossible.

Start with a journey, not a portal

Teams do not wake up wanting a developer portal. They want to create a service, ship a change, rotate a secret, expose an API, diagnose an incident, or retire a workload. A golden path should make one of those journeys faster and safer from request to operation.

The CNCF Platforms White Paper frames a platform as a product that curates capabilities for internal users. That product framing changes the first question from “Which tools should we centralize?” to “Which recurring user problem should we solve, for whom, and how will we know?” The white paper is living guidance; teams should record when they adopt a particular version or interpretation.

I start with user interviews and observation of the work. I measure current lead time, handoffs, failure/rework, cognitive load, and support demand, then pick a frequent, painful journey with enough commonality to standardize. A rare specialist workload is a poor first candidate.

Separate the path into contracts

A maintainable path combines several layers without hiding them:

• Experience: portal, CLI, documentation, examples, and discoverable ownership.
• Scaffolding: a template that creates a repository, metadata, tests, deployment definitions, and an initial service record.
• Capabilities: identity, secrets, build, deployment, networking, observability, data, and policy exposed through stable interfaces.
• Guardrails: automated checks and secure defaults with actionable feedback.
• Operations: runtime service levels, on-call ownership, upgrades, incident support, and decommissioning.

Backstage Software Templates provide one official example of controlled inputs and actions that scaffold and publish standardized project skeletons. A template is only the entry point. If generated services cannot receive platform updates, diagnose failures, or exit safely, the organization has created a one-time code generator rather than a supported path.

Version capability contracts independently from templates. Prefer composition: a team may use the standard build and observability capability while choosing a justified data store. Publish compatibility and deprecation windows. Test migrations on representative services before changing defaults.

Make safety the easiest choice

Good defaults remove decisions that add little product value: encrypted transport, baseline telemetry, ownership metadata, dependency scanning, deployment health checks, resource limits, and rollback hooks. Guardrails should explain the risk, show the failing evidence, and provide a repair path.

Do not confuse safety with uniformity. A single runtime, repository shape, or release process can become a constraint when workloads differ. The CNCF Platform Engineering Maturity Model describes progression toward self-service and standardized interfaces while warning that early approaches may offer too few customization options. Maturity is not the number of mandatory tools; it is the platform’s ability to meet user needs reliably through product practices.

Define a documented exception route. Ask for the use case, unmet requirement, risk owner, alternative controls, duration, support model, and review date. Time-box temporary exceptions. Feed repeated exceptions into the roadmap: they may reveal a missing capability or a path whose assumptions are too narrow.

An escape hatch without ownership becomes unmanaged risk. A path without an escape hatch becomes a golden cage.

Measure outcomes, not enrolment

Portal visits and template executions are activity metrics. A platform can have high enrolment because it is mandatory while users remain slow and frustrated.

Use a balanced scorecard:

• journey lead time and waiting time;
• change failure and recovery performance;
• reliability and security-control coverage;
• developer satisfaction and task success;
• support tickets and time to resolve;
• adoption by eligible teams, not all teams;
• exception frequency, age, and common causes;
• upgrade completion and exit cost.

Segment the results. A path may work for stateless APIs and fail for data-intensive or regulated workloads. Combine telemetry with interviews; instrumentation shows where users leave, while research explains why.

Set a hypothesis before launch: “For a standard API, the path will reduce median production onboarding from ten working days to two while preserving required controls.” Define the boundary and baseline. That is more useful than declaring the portal a success after one hundred registrations.

Run the platform as an internal product

Give every path a product owner, technical owner, service level, documentation, support channel, lifecycle policy, and roadmap. Treat teams as users rather than captive consumers. Publish incidents and breaking changes. Maintain examples that run in CI so documentation cannot drift indefinitely.

Create feedback loops at three levels:

1. Inline: task-specific errors explain how to recover.
2. Operational: telemetry identifies failure, latency, abandonment, and support hotspots.
3. Product: interviews, exception reviews, and outcome metrics change priorities and contracts.

Prioritize by user outcome and systemic risk. A new dashboard may be less valuable than reducing a twenty-minute build queue or clarifying ownership during incidents.

A synthetic path: launching a standard API

Here is how I would model a synthetic “managed API” path. The user supplies ownership and data classification and receives a repository with service metadata, tests, CI, deployment descriptors, and observability. The pipeline verifies policy and deploys to a supported runtime. The catalogue shows ownership, dependency, health, and runbook links.

A team needing unusual network placement requests an exception. The platform cannot currently satisfy it, so the team uses an approved external pattern with named support and a review date. Three similar exceptions appear; product discovery confirms a reusable networking capability is warranted. The capability is added, migration support is offered, and the exception count falls.

That loop is the key: the platform learns from friction instead of treating deviation as disobedience.

The design test

My design rule is that a golden path should be easy to enter, safe to operate, inexpensive to upgrade, and possible to leave. It should expose composable contracts rather than conceal an opaque stack. Its success is measured in delivery and operational outcomes, not enforced adoption.

Build one end-to-end journey, instrument it, support it, and let evidence determine the next path. The goal is a platform product that earns trust—not a golden cage with better branding.

Related reading

• Architecture Decision Records That Teams Keep Using
• Modernizing Core Banking Without a Big-Bang Rewrite

References

• CNCF — Platforms White Paper
• CNCF — Platform Engineering Maturity Model
• Backstage — Software Templates

I consider an architecture decision record useful when a future engineer can recover the context, understand the trade-off, and tell whether new evidence should supersede it.

The question an ADR must answer

Code shows what the system does now. It rarely preserves why one option was chosen, which constraints mattered, what was deliberately not solved, or what evidence would justify reversal. Those details disappear when a chat closes or a meeting ends.

Michael Nygard’s original “Documenting Architecture Decisions” proposed a deliberately small record: title, context, decision, status, and consequences. The format remains useful because it captures a decision at the moment the trade-off is understood, not months later as historical fiction.

An ADR is appropriate for a decision that is architecturally significant, costly to reverse, contested, or likely to surprise a future maintainer. It is not required for every library update or implementation detail. Teams should define their threshold and use examples to calibrate it.

A compact record with sharp fields

I start with a record that fits on one or two screens:

1. ID, title, date, owner, status. Use a stable sequence or identifier and a verb phrase such as “Use an outbox for payment events.”
2. Context. State the forces: scale, reliability, regulation, team capability, deadlines, existing constraints, and what is out of scope.
3. Options considered. Present credible alternatives fairly, including “do nothing.” Link experiments or measurements.
4. Decision and rationale. State the chosen option and the decisive trade-off. “Use Kafka” is not a rationale.
5. Consequences. Record benefits, costs, risks, migration work, and operational obligations.
6. Validation and review triggers. Define how the assumption will be tested and what evidence would prompt reconsideration.
7. Links. Connect code, diagrams, incidents, benchmarks, issues, and related ADRs.

The AWS ADR process guidance similarly describes concise context, options, decision, consequences, ownership, status, review, and superseding accepted records rather than editing them silently. Treat the page as living guidance and capture the access/version context when using it as an organizational standard.

A worked example: synchronous call or outbox

Weak record:

We chose events because they scale better.

This cannot be challenged. It omits the transaction boundary, delivery expectations, operational cost, alternatives, and measurement.

Here is how I would make the synthetic record stronger. It says the service must commit an order and reliably notify downstream fulfilment without a distributed transaction. It compares a synchronous downstream call, database polling, change-data capture, and a transactional outbox. It chooses the outbox because the order row and outbound record can share one local transaction, while acknowledging duplicate delivery, ordering, relay operations, and consumer idempotency. It links a failure test and states review triggers: database throughput below a threshold, relay lag above a service objective, or adoption of a platform capability that provides equivalent guarantees.

The record does not prove the design works. Tests, telemetry, and operational documentation provide current evidence. The ADR explains why this design was chosen under stated conditions.

Put ADRs in the path of work

Store ADRs close to the code or architecture repository they govern. Use a predictable directory, searchable index, and lightweight template. A pull request works well because reviewers can comment alongside the proposed implementation and the accepted record becomes versioned.

Suggested states are proposed, accepted, rejected, deprecated, and superseded. Ownership and decision date prevent an orphaned proposal from appearing authoritative. When a decision changes, create a new ADR that links supersedes and update the old record to point to it without erasing its original context.

The AWS introduction to using ADRs to streamline decisions emphasizes alignment, decision history, reduced repeated debate, and handover. Those benefits depend on discoverability and adoption; creating files alone produces no organizational memory.

Integrate simple checks: unique IDs, required fields, valid status links, and an index. More important, include an ADR link in significant design pull requests and incident reviews. If a production event invalidates an assumption, open the review trigger rather than leaving the record as dead documentation.

Review without approval theatre

Review should improve the decision while it is reversible. Ask:

• Is the problem and scope clear?
• Are alternatives represented honestly?
• Which evidence supports the decisive assumption?
• Who carries each consequence?
• How will implementation validate the choice?
• What new fact would cause reconsideration?

Avoid committees for every record. Match reviewers to impact: security for a trust boundary, data engineering for consistency, operations for on-call consequences, and product for user-facing trade-offs. Time-box discussion and name the decision owner. Consensus is useful but not always possible; accountability must remain clear.

Failure patterns that make ADRs decay

Decision archaeology. Writing the ADR after implementation turns it into justification. Draft it with the proposal.

Template inflation. A dozen mandatory sections encourages vague filler. Keep the core short and link supporting evidence.

Technology inventory. “We use PostgreSQL” is not a decision unless the context, alternatives, and consequences are present.

Mutable history. Rewriting an accepted ADR hides why earlier implementation exists. Supersede it explicitly.

No review trigger. A record based on traffic, cost, regulation, or team assumptions needs a signal that can invalidate them.

Disconnected truth. An ADR that cannot be reached from code, service catalogues, or architecture views will not be found when needed.

The smallest useful discipline

Start with one repository, a seven-field template, a review state, and supersession links. Apply it only to meaningful choices. Revisit records during incidents, migrations, and major design changes.

My design rule is not comprehensive documentation. I want to preserve the reasoning that code cannot express: what was known, which compromise was chosen, who accepted the consequences, and which evidence should make the team decide again.

Related reading

• Platform Engineering Golden Paths Without Golden Cages
• Modernizing Core Banking Without a Big-Bang Rewrite

References

• Michael Nygard — Documenting Architecture Decisions
• AWS — Architectural decision record process
• AWS — Using ADRs to streamline decision-making

I design AI observability to connect a user request to the exact model, context, tools, controls, cost, and outcome—without turning sensitive content into a second uncontrolled dataset.

Healthy infrastructure can hide a failing product

An AI endpoint can return HTTP 200 in two seconds while giving an unsupported answer, invoking the wrong tool, or costing three times more than the previous release. CPU, memory, error rate, and latency remain necessary, but they do not describe model behaviour.

The unit of observation should be a task attempt. Give it a trace ID and capture the configuration that shaped the result: application version, prompt template, model/provider, sampling parameters, retrieval/index version, tool definitions, policy version, and experiment cohort. Then connect technical execution to evaluation and outcome signals.

OpenTelemetry semantic conventions provide a common model for traces, metrics, logs, events, and resources. When this article was researched on 19 July 2026, the page referenced semantic conventions 1.43.0 and GenAI-related attributes were still evolving. Pin the convention version; do not assume an experimental attribute will remain stable.

Design a correlation spine

I start with one root span for the application task. Child spans represent retrieval, model calls, tool execution, and policy checks. I use stable, low-cardinality names such as chat.answer or payment_policy.lookup, not the user’s question. The OpenTelemetry guidance on writing semantic conventions explicitly emphasizes low-cardinality operation naming, stability, signal design, and careful treatment of expensive or sensitive attributes.

Record references to versions rather than copying entire artifacts into every span. A prompt registry ID can resolve to the approved template; a retrieval snapshot ID can identify the indexed corpus; a tool-call span can contain the validated operation name and status. This keeps traces queryable while preserving a chain to deeper evidence under stricter access controls.

Use metrics for aggregates and service objectives: request rate, first-token and total latency, tokens, cost estimate, tool error rate, empty retrieval rate, policy blocks, and task completion. Logs should capture exceptional diagnostic events. Traces explain causality across a sampled request. Evaluation records attach quality judgments that may arrive later.

Observe quality at three timescales

Inline controls run before or during the response. Examples include schema validation, tool authorization, citation presence, secret detection, and policy filters. Their outputs should appear as named decisions with policy versions, not anonymous “guardrail failed” strings.

Nearline evaluation samples recent traces after completion. Deterministic checks and calibrated graders can assess groundedness, instruction following, or tool correctness. Sample by risk and cohort; a uniform 1% sample may miss rare high-impact workflows.

Offline evaluation uses controlled, versioned datasets before release and after incidents. It measures regressions reproducibly. Production traces can suggest new cases, but copying live personal data into a test set without consent and governance creates another privacy problem.

The NIST Generative AI Profile, published in July 2024 and updated on its page in April 2026, supports lifecycle monitoring of performance, incidents, misuse, privacy, and information-integrity risks. It does not prescribe one telemetry stack; the organization must map signals to its actual harms and controls.

Treat privacy as architecture, not redaction later

Prompts, retrieved documents, model outputs, and tool arguments may contain personal data, financial information, credentials, or proprietary material. Do not log them by default. Decide per field whether to drop, transform, hash, tokenize, sample, encrypt, or store in a restricted evidence vault with a shorter retention period.

Separate a low-sensitivity operational stream from controlled content evidence. Most dashboards need counts, durations, versions, and status—not raw conversations. If content capture is justified for incident investigation, enforce purpose limitation, role-based access, audit logs, retention/deletion, and tenant isolation. Redaction should happen before data leaves the application trust boundary; downstream cleanup is too late.

Be cautious with labels. User thumbs-up is affected by interface and expectation; escalation may be the correct outcome; a grader score may drift when its model changes. Store label source, definition, rubric, and version. Do not merge unlike signals into a single “quality” metric.

Build alerts around decisions

An alert should name a failure, affected slice, threshold, and response. Useful examples include:

• unauthorized tool attempts above zero for a privileged workflow;
• groundedness regression for one language or corpus version;
• p95 cost per completed task exceeding its release budget;
• retrieval empty-result rate rising after an index migration;
• model fallback or policy block rate changing sharply by cohort;
• user corrections increasing while infrastructure metrics remain stable.

Avoid cardinality explosions by putting user IDs, full URLs, prompts, or document IDs into metric labels. Keep those identifiers in controlled trace/event records if justified. Sample strategically: retain all severe incidents, a representative baseline, and enough cases from rare high-risk slices.

A synthetic incident walkthrough

Here is how I would investigate a synthetic incident. Consider an assistant that answers policy questions from a versioned knowledge base. After a deployment, latency and HTTP errors remain unchanged, but user corrections rise for Arabic questions.

The correlation spine shows that affected traces used prompt version 12 and index version 31. Retrieval returned relevant passages, but a nearline evaluator flags incorrect date interpretation. The release comparison isolates the regression to the new prompt, not the model or index. Traffic is routed to prompt version 11, the failing traces become de-identified regression cases, and a language-specific release threshold is added.

Without versioned prompts, retrieval evidence, outcome labels, and slice-aware evaluation, the team would see only healthy servers and anecdotal complaints.

The observability contract

For every material AI workflow, document what constitutes success, which configuration identifiers are required, which signals are collected, who may access them, how long they persist, and which threshold triggers rollback or investigation. Test trace completeness just as you test application behaviour.

My design rule is not “log everything.” I want a deliberately limited evidence system that explains how a particular configuration produced a particular outcome and whether that outcome was acceptable. The privacy boundary is part of its correctness.

Related reading

• AI Agents in Regulated FinTech: Control Before Autonomy
• RAG for Financial Operations: Evidence Over Eloquence

References

• OpenTelemetry — Semantic Conventions
• OpenTelemetry — How to write semantic conventions
• NIST — Generative AI Profile

I consider responsible AI governance real only when every deployed use case has an owner, a traceable system record, evidence-based controls, and a tested way to stop or change it.

Policy is not yet a control

“Use AI responsibly” gives an engineering team no executable instruction. A control must identify the risk it addresses, the systems and actors in scope, the evidence required, who decides, when it runs, and what happens on failure.

The NIST AI Risk Management Framework 1.0 organizes work into Govern, Map, Measure, and Manage. That structure is useful because it connects institutional accountability with system context, evaluation, and treatment. NIST was working on a revision in 2026, so teams should record the framework version they map to rather than treating a living reference as timeless.

Governance is broader than model approval. A system may combine a foundation model, retrieval corpus, prompt, rules, tools, user interface, human review, and downstream workflow. A safe model can still participate in an unsafe process; a risky model can be constrained to a low-impact assistive role. Inventory the use case as an operational system.

Create one system record that joins the evidence

I start with a stable use-case ID before production access. Here is the system record I would require:

• accountable business owner and technical operator;
• intended purpose, users, affected people, and prohibited uses;
• jurisdictions, deployment role, and preliminary legal classification;
• model/provider, data sources, prompts, tools, dependencies, and versions;
• risk tier and rationale;
• evaluation, security, privacy, accessibility, and human-oversight evidence;
• approved limitations, monitoring thresholds, incident owner, and rollback plan.

Third-party models do not remove accountability. Track contractual restrictions, data handling, geographic processing, update policy, incident notification, and the provider version actually used. A silent provider update should trigger change assessment, not flow directly into a high-impact process.

Link artifacts rather than copying them into a governance spreadsheet. The inventory should point to source-controlled prompts, test reports, data lineage, threat models, approvals, deployment records, and incidents. The goal is a traversable evidence graph: from a production call to the approved configuration and from an identified risk to the controls that mitigate it.

Tier risk from context, not technology labels

“Generative AI” is not a risk tier. Map plausible harms using purpose, autonomy, data sensitivity, scale, reversibility, and effect on people. A summarizer for public documentation differs from an agent that can change a person’s payment limit.

Higher impact should demand stronger evidence and tighter runtime boundaries: independent validation, representative slice testing, least-privilege tools, approval for material actions, appeal routes, enhanced logging, rollout limits, and rehearsed rollback. The NIST Generative AI Profile, published in July 2024 and updated on its page in April 2026, identifies risks such as confabulation, information integrity, privacy, and misuse and suggests lifecycle actions. It is voluntary guidance, not a legal classification.

Risk tiers should also state residual risk. Passing controls does not make a system harmless; it makes the accepted exposure visible to the accountable decision-maker.

Separate legal applicability from internal assurance

The EU Artificial Intelligence Act, Regulation (EU) 2024/1689, establishes a role- and risk-based regime covering prohibited practices, high-risk systems, transparency, and general-purpose AI. Its applicability is phased and depends on geography, system classification, role, and use case. As of 19 July 2026, teams must verify the current application dates, Commission guidance, and professional legal advice before stating that a specific obligation applies.

Do not encode “EU AI Act compliant: yes/no” as a casual field. Record the legal analysis, role (for example provider or deployer), system category, applicable provisions, assessor, date, and assumptions. Keep internal responsible-AI controls distinct: an organization may choose stricter requirements than law, and voluntary NIST practices do not become statutory duties merely because both appear in the same control library.

Make approval a reproducible release gate

I would generate the approval package from the system record and the exact deployment candidate. It should include versioned evaluations, open findings, risk acceptance, use limitations, monitoring, and rollback. The approver must see what changed since the last approved release.

Classify changes before deployment. A copy edit may require basic checks; a new model, tool permission, jurisdiction, sensitive dataset, or autonomous action may require reassessment and independent review. CI/CD should verify that a valid approval covers the exact artifact versions. It should not allow a generic ticket to authorize later configurations.

Avoid approval theatre. A committee that receives a hundred-page document after implementation cannot provide effective challenge. Bring assurance specialists into mapping and test design, and define evidence templates that teams can produce continuously.

Close the loop with incidents and appeals

Monitor technical behaviour and human impact: prohibited tool attempts, unsupported claims, harmful content, override frequency, complaint/appeal outcomes, subgroup performance where lawful and appropriate, provider changes, drift, cost, and latency. Preserve privacy by collecting only the evidence required and controlling access and retention.

Thresholds need actions. A severe policy breach may disable a tool immediately; rising error on one slice may restrict rollout; a provider version change may route traffic to the last approved version. Incident handling must link affected calls to configuration, notify accountable roles, preserve evidence, correct harm where possible, and create new tests or controls.

Appeals and human overrides are not simply “noise.” They can expose a faulty policy, inaccessible interface, missing data, or unjustified automation. Feed those findings back into mapping, measurement, and risk acceptance.

The operating model

My design rule is that governance is a loop, not a launch checkpoint: inventory the whole system, map context and law, measure risks, approve a specific configuration, monitor use, manage incidents, and update the evidence.

The strongest test is operational: can the organization identify every affected deployment, explain why it was approved, show which controls ran, contact its owner, and roll it back quickly? If not, the policy has not yet become an engineering system.

Related reading

• AI Agents in Regulated FinTech: Control Before Autonomy
• RAG for Financial Operations: Evidence Over Eloquence

References

• NIST — AI Risk Management Framework
• NIST — Generative AI Profile
• EUR-Lex — Regulation (EU) 2024/1689

I model production vector search as two systems joined by a contract: an indexing plane that maintains searchable evidence and a query plane that retrieves authorized, relevant results.

The demo hides the hard contracts

A demo embeds ten documents, performs nearest-neighbour search, and prints five passages. Production must answer harder questions: which version of a document is searchable, who may see it, how deletion propagates, whether a filtered query preserves recall, and how a new embedding model is cut over without mixing vector spaces.

Dense retrieval is valuable because it can match semantic similarity beyond exact terms. The original retrieval-augmented generation paper combined generation with retrieved non-parametric memory, but retrieval remains only one subsystem. A fluent answer can still misuse a relevant passage; retrieval quality and generation quality must be evaluated separately.

I start with the retrieval contract: input query and identity context; eligible corpus and freshness target; returned identifiers, passages, scores, and versions; latency percentile; and the metric by which relevance will be judged.

Indexing is a governed data product

The indexing plane starts with authoritative sources, not arbitrary files. Every item needs a stable source ID, tenant or entitlement metadata, document version, effective time, deletion state, and provenance. Normalize formats, but retain a link to the original evidence. Chunking should follow semantic boundaries where possible and record parent-child relationships so a retrieved fragment can be interpreted in context.

Embedding records require at least the embedding model/version, dimensions, normalization rule, chunker version, and creation time. Vectors produced by different models generally do not share a meaningful distance space. Never silently write a new model’s vectors into an old index.

Treat reindexing as a migration. Build a shadow index, replay changes into both old and new versions, evaluate them against the same judgments, then switch an alias or routing rule. Keep a rollback window and verify that deletions and entitlement changes reached both copies. Freshness is an observable service level: measure source-to-search lag and deletion lag, not merely job success.

Query-time retrieval is more than ANN

At query time, normalize the query using the same contract as the index and carry caller identity into retrieval. Authorization should constrain eligibility before results leave the search boundary. Post-filtering an unrestricted top-k can leak metadata and destroy recall when most candidates are forbidden.

Semantic search should usually work alongside lexical retrieval. Exact identifiers, error codes, account products, and regulatory phrases often favour term-based methods; paraphrased questions favour embeddings. Hybrid fusion—such as reciprocal-rank fusion—combines ranks without pretending incomparable raw scores share a scale. A reranker can then use richer query-document interactions on a small candidate set.

Approximate nearest-neighbour indexes exchange perfect recall for speed and memory. The HNSW paper describes a multilayer graph approach with tunable search complexity. The operational consequence is not “HNSW is fast”; it is that build parameters, query exploration, filters, data distribution, and hardware jointly determine recall and latency.

The pgvector documentation distinguishes exact search from HNSW and IVFFlat and documents filtering, iterative scans, and recall measurement. Its repository showed version 0.8.2 when this article was researched on 19 July 2026; re-check the current release and behaviour before implementing. Product-specific defaults are not universal benchmarks.

Measure recall against a trustworthy reference

For a labelled query set, run exact search or a high-quality exhaustive reference and compare the approximate result set. Report recall@k or another named metric alongside p50/p95/p99 latency. Record:

• corpus size and vector dimensions;
• distance metric and vector normalization;
• index type and parameters;
• candidate and final k;
• filter selectivity and tenant distribution;
• hardware, concurrency, cache state, and software version.

Without those conditions, “10 ms search” is marketing, not engineering evidence. Evaluate by slice: short versus long queries, rare vocabulary, fresh documents, languages, tenants, and restrictive filters. Also measure end-to-end relevance such as nDCG or judged precision, because high ANN recall can faithfully retrieve the wrong content if the embedding or chunks are poor.

A failure scenario: the invisible tenant regression

Here is how I would test the filter boundary with a synthetic case. Imagine an index with one million chunks. An entitlement filter leaves only 0.2% eligible for a particular team. The ANN engine finds globally close neighbours, then a post-filter removes almost all of them. Latency appears healthy, but the team receives empty results.

Possible remedies include pre-filter-aware indexing, partitioning by a stable tenancy boundary, iterative scanning with a bounded budget, or retrieving a larger candidate pool. Each changes cost and recall. The correct choice depends on tenant size distribution and isolation requirements; it cannot be inferred from unfiltered benchmarks.

The same scenario shows why access control belongs in the architecture, not in a prompt. Cache keys, logging, reranking, and answer generation must preserve the authorization context. Deletion and revocation need tested end-to-end propagation.

Operate retrieval as a versioned service

Expose the chosen index, embedding, chunker, lexical configuration, filter policy, fusion method, and reranker version in traces. Monitor empty-result rate, source freshness, deletion lag, filter selectivity, score distribution, exact-versus-ANN recall samples, and human relevance feedback. Alert on slice regressions rather than one global average.

My design rule is simple: keep indexing and query flows separate, join them through a versioned retrieval contract, and test every optimization against exact search and user relevance. That is how vector search graduates from an impressive demo to an evidence service that can be upgraded, audited, and rolled back.

Related reading

• AI Agents in Regulated FinTech: Control Before Autonomy
• RAG for Financial Operations: Evidence Over Eloquence

References

• Malkov and Yashunin — Efficient and Robust Approximate Nearest Neighbor Search Using HNSW
• pgvector — official documentation
• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

I do not treat an LLM evaluation as a leaderboard. I treat it as a versioned argument—with evidence—about which failures a particular system can tolerate before release.

Begin with the decision, not the metric

“Is model B better?” is too vague to evaluate. A useful question names the task, users, consequence of error, operating configuration, and release decision. For a support copilot, the decision might be: can version 7 replace version 6 for English and Arabic refund questions without increasing unsupported policy claims or p95 cost beyond an agreed budget?

That framing matters because language-model quality is multidimensional. The HELM research framework evaluates models across scenarios and metrics including accuracy, calibration, robustness, fairness, toxicity, and efficiency rather than collapsing quality into one universal number. A production system adds more variables: system prompt, retrieval corpus, tools, sampling parameters, safety policy, and post-processing. Change any of them and the evaluated object has changed.

Create an evaluation dossier with immutable identifiers for the model, prompt, tools, retrieval snapshot, code commit, dataset, rubric, and judge. A score without that configuration is not reproducible evidence.

Build a test portfolio around risk

I start with a task taxonomy, then sample cases by consequence and frequency. The portfolio I would take to a release review contains:

• representative cases from real, consented, and de-identified workflows;
• boundary cases where policy, dates, currencies, languages, or permissions interact;
• adversarial cases such as prompt injection and unsupported instructions;
• regression cases distilled from incidents and user corrections;
• counterfactual pairs that vary one sensitive attribute or one critical fact;
• explicit abstention cases where the correct action is to refuse or escalate.

Protect a holdout set from prompt tuning and routine developer inspection. Otherwise the team optimizes to a familiar test rather than the task. Track provenance and licenses; remove secrets and personal data; document synthetic cases as synthetic. Repeated trials are necessary when sampling or tool execution is nondeterministic. Report the run count and dispersion, not only the mean.

Slice results by risk, language, workflow, document freshness, and tool path. A 94% aggregate pass rate can conceal a 60% pass rate on high-value transfers or Arabic policy questions. Confidence intervals are especially important for small slices: “zero failures in ten cases” is not evidence of zero risk.

Use an evaluation stack, not one judge

Different checks answer different questions.

Deterministic assertions should verify what code can know exactly: JSON schema, required citations, tool arguments, arithmetic, forbidden actions, latency, and cost. Reference-based metrics can compare structured outputs or retrieval results. Model graders can assess qualities such as relevance or groundedness at scale, but they introduce their own bias, instability, and blind spots. Human experts remain necessary for ambiguous, novel, or high-impact cases.

Calibrate a model grader against independently labeled examples. Publish its agreement, false-pass rate, false-fail rate, and ambiguous region by slice. Do not let the same model family silently grade its own preferred style. NIST’s work on evaluating machine-generated reports emphasizes dimensions such as completeness, accuracy, verifiability, and claim-to-source support—useful reminders that fluent prose is not a factuality metric.

Route disagreement and high-risk failures to human review. Measure inter-rater agreement and improve a rubric when qualified reviewers interpret it differently. “Human reviewed” is not a method unless the reviewer role, sampling rule, rubric, and resolution process are recorded.

Turn scores into release policy

I set thresholds before examining the candidate result; otherwise the goal tends to move around the preferred release. Here is how I would model a practical gate:

1. zero violations of non-negotiable controls, such as unauthorized tool calls;
2. no statistically credible regression on high-risk slices;
3. a bounded false-pass rate for the calibrated grader;
4. latency and cost budgets at named percentiles;
5. documented acceptance of residual risk by the accountable owner.

Suppose a collections assistant improves answer completeness from 86% to 92% but unsupported fee claims rise from 0.5% to 1.4%. The average may improve while the product becomes less safe. The correct response could be a narrower rollout, a deterministic policy lookup, or mandatory review—not simply another prompt tweak.

In regulated financial institutions, evaluation evidence may also feed model-risk governance. The US Federal Reserve’s revised model risk management guidance published on 17 April 2026 stresses risk-proportionate governance, independent validation, ongoing monitoring, use limitations, and effective challenge. Applicability depends on the institution and use case; an evaluation suite is evidence for governance, not proof of compliance.

Monitor the gap between tests and production

Pre-release cases are a model of reality, not reality itself. In production, record privacy-safe outcome signals: escalation, correction, task completion, policy violation, user appeal, latency, and cost. Join them to the evaluated configuration. Define drift alerts for input mix and failure slices, not only token volume.

Every confirmed failure should produce three artifacts: an incident record, a minimized reproducible test, and a decision about whether the release threshold or control must change. Re-run the suite when the model, prompt, tool, retrieval data, policy, or evaluator changes. Benchmark results decay as these components evolve.

What a defensible evaluation can claim

My design rule is to claim only what the evidence supports. A strong evaluation cannot guarantee that an LLM “must be right.” It can state a narrower and more useful conclusion: under a named configuration, on a versioned portfolio, with stated uncertainty, the candidate met defined thresholds and known residual risks are controlled or accepted.

That claim is less dramatic than a leaderboard win. It is also the kind of claim an engineering owner can reproduce, challenge, monitor, and reverse.

Related reading

• AI Agents in Regulated FinTech: Control Before Autonomy
• RAG for Financial Operations: Evidence Over Eloquence

References

• Liang et al. — Holistic Evaluation of Language Models (HELM)
• NIST — On the Evaluation of Machine-Generated Reports
• Federal Reserve — Supervisory Guidance on Model Risk Management

I treat retrieval-augmented generation (RAG) as governed evidence retrieval with a generator attached. It is useful in financial operations only when every material claim traces to current, authorized evidence; retrieval does not guarantee truth, completeness, or permission. The operations examples below are synthetic.

Define the decision the answer may influence

“Search our procedures” is too broad. Specify the user, task, evidence set, and allowed consequence. Answering where to find a policy is lower risk than recommending whether to release a payment exception. A system that drafts an investigation note differs from one that triggers a workflow.

The original RAG paper by Lewis and colleagues combines parametric generation with retrieved non-parametric memory and provides provenance for generated text. It establishes a technique, not a control framework for financial decisions.

Write an answer contract: supported sources, maximum source age, jurisdiction or entity, required citations, prohibited data, confidence/abstention behavior, and actions the answer cannot authorize. For a material decision, keep the human or deterministic rule responsible and present the retrieved evidence separately from the model’s interpretation.

Govern ingestion as a data product

Each document needs a stable identifier, version, effective and expiry dates, owner, jurisdiction, confidentiality label, retention rule, and source URL. Preserve the original artifact and a checksum. Chunked text is a derived representation; link every chunk to its page, section, and document version.

Process updates and deletions. A superseded procedure must stop appearing in new answers while remaining available to authorized audit workflows where retention requires it. Build deletion propagation through the object store, search index, vector index, cache, and evaluation fixtures.

Do not mix incompatible embeddings during a silent model change. Version the chunking algorithm, embedding model, and index. Build a shadow index, compare retrieval quality and access behavior, then cut over atomically. Retain enough lineage to reconstruct which index produced an answer.

Enforce authorization throughout retrieval

An entitlement filter drawn as one box is insufficient. Access control begins before ingestion, is attached to each derived chunk, is enforced during candidate generation and ranking, and is checked again before context reaches the model and before an answer reaches the user.

Use allow-based policies tied to authenticated identity, role, entity, jurisdiction, case, and purpose. Avoid retrieving broadly and filtering only after ranking: unauthorized titles or snippets may leak through logs, scores, caches, or model context. Partition indexes where the security boundary warrants it, and test negative access cases continuously.

Retrieved documents are untrusted inputs. They can contain malicious instructions or obsolete advice. Treat their content as evidence, never as system instructions. Strip active content, delimit quotations, allowlist tools, and prevent a document from changing the answer contract.

Separate retrieval quality from answer quality

Evaluate retrieval first. For versioned test questions, measure whether the required evidence appears in the top results, whether forbidden evidence is absent, and whether filters, freshness, and citations resolve to the expected version. Track recall at a fixed candidate count, rank quality, no-result rate, and latency by security slice.

Then evaluate generation. NIST’s work On the Evaluation of Machine-Generated Reports emphasizes dimensions such as completeness, accuracy, verifiability, and claim-to-source support rather than fluency alone. Break answers into atomic claims and label each as entailed, contradicted, unsupported, or not requiring evidence.

A citation is not proof. The linked passage may be real but irrelevant. Add a verifier that checks quotation location, document version, user entitlement, and whether the cited passage entails the claim. High-risk or low-confidence results should abstain with a useful explanation: what could not be established, which sources were checked, and what the operator should do next.

Keep privacy and auditability in tension

The NIST Generative AI Profile addresses information integrity, privacy, confabulation, monitoring, and provenance risks. Apply data minimization: do not log full prompts, documents, or answers by default. Record identifiers, versions, policy decisions, model configuration, latency, token and cost measures, citation results, and a protected reference to content where investigation requires it.

Use synthetic examples for development and evaluation. Redact personal and account data before model calls. Define model-provider retention and training settings, data residency, encryption, incident duties, and subcontractors. Do not put secrets or access tokens in the retrieval context.

Design the operating loop

Monitor unsupported-claim rate, citation entailment, stale-source use, access denials, abstention, retrieval miss, feedback by task, and operator overrides. Segment by document family, language, entity, and risk tier. User thumbs-up is not a correctness label; collect structured reasons and review sampled evidence.

Re-evaluate after changing the model, prompt, chunker, embedding, index parameters, reranker, policy engine, document set, or provider. Keep a release record that binds all those versions to results and known limitations. A kill switch should disable generation while leaving ordinary source search available.

Test the dangerous cases

Test a user asking about another entity, a document whose permissions changed, a superseded policy, conflicting sources, a malicious instruction inside a PDF, a missing page, an unsupported numerical claim, an answer in Arabic citing an English source, and deletion of a sensitive artifact. Verify both the response and the absence of leakage in logs and caches.

The architecture decision

Build RAG as governed evidence retrieval with a generator attached—not as a chatbot with a vector database. Version sources and derived data, enforce entitlement at every boundary, evaluate retrieval and generation separately, verify claim-to-source support, and make abstention successful. The goal is not the most fluent answer. It is an answer whose scope, evidence, and uncertainty an operator can inspect before acting.

Related reading

• AI Agents in Regulated FinTech: Control Before Autonomy
• LLM Evaluation for Systems That Must Be Right

References

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• NIST — On the Evaluation of Machine-Generated Reports
• NIST — Generative AI Profile

An AI agent should not receive financial authority merely because it can plan and call tools. I treat autonomy as a graduated control decision: constrain tools, bind every action to policy and identity, require approval for material effects, and preserve replayable evidence. The use cases below are synthetic.

Start with an authority map

“Agent” describes an interaction pattern, not a risk class. A read-only assistant that summarizes a public policy differs fundamentally from a system that changes a customer limit or initiates a refund. Inventory each use case by data accessed, tools callable, maximum financial effect, reversibility, customer impact, and legal or compliance owner.

The US banking agencies’ Revised Guidance on Model Risk Management, SR 26-2, issued 17 April 2026, describes risk-tiered inventories, validation, monitoring, governance, effective challenge, and third-party considerations for models used by banking organizations. Whether a particular agent meets a jurisdiction’s formal definition of a model requires legal and supervisory analysis; the control disciplines remain useful for the model, tools, and decision process.

The Bank of England and FCA’s 2024 survey of AI in UK financial services reports industry practices and concerns around accountable ownership, data governance, and third-party dependencies. It is survey evidence, not a rulebook.

Separate proposal from execution

Let the model produce a structured action proposal: tool name, typed parameters, purpose, supporting evidence, confidence or uncertainty, and requested authority. A deterministic policy gateway validates it against the authenticated user, customer/account scope, amount limits, time window, segregation of duties, and current case state.

The model never chooses its own credential. The gateway issues a short-lived, single-purpose capability to a narrowly scoped tool adapter. The adapter validates parameters again and maps the request to a domain command with an idempotency key. Financial state transitions remain owned by the domain service and ledger, not by the conversation history.

Material actions require an approval bound to the exact proposal. Show the reviewer the amount, destination, affected account, source evidence, model and prompt version, and expiry. If any material field changes after approval, invalidate it. “A human clicked approve” is not a control if the screen hides what will execute or the agent can substitute parameters later.

Treat all retrieved content as untrusted

Prompts, emails, documents, web pages, and tool results can contain instructions aimed at the model. Never let retrieved text redefine policy or grant tools. Label content and instructions separately, allowlist tool schemas, validate every output, and prevent the model from selecting arbitrary URLs, SQL, shell commands, or recipients.

The NIST Generative AI Profile, AI 600-1 identifies risks including confabulation, information integrity, privacy, misuse, and third-party dependencies, with lifecycle actions for governance, measurement, and management. Use it as voluntary risk guidance, not as law.

Redact or tokenize sensitive context before it reaches a third-party model where possible. Define data residency, retention, training-use, incident notification, and subprocessors contractually. Keep secrets outside prompts and traces. A tool call should reference a protected object identifier rather than copying account data into free text.

Build an evidence record, not a surveillance dump

For each run, record the authenticated actor, use case, policy version, model and configuration, prompt-template version, retrieval document identifiers and versions, proposed actions, approvals, tool requests, domain-operation identifiers, outcomes, and exceptions. Hash or reference sensitive content instead of logging it indiscriminately.

Evidence must answer: who requested this, what authority applied, which facts were available, what did the model propose, what deterministic controls allowed, what financial operation executed, and how was the final outcome reconciled? Preserve stable identifiers across traces, cases, and ledger records.

Replay does not mean re-executing tools. An evaluation harness should reconstruct the input and proposed decision in an isolated environment with mocked tools. Production replay must never move money.

Increase autonomy only after measured evidence

Use stages: offline evaluation; shadow mode; read-only production; recommendations requiring approval; low-value reversible actions within limits; and only then narrowly defined automation. Promotion criteria should cover task success, unsupported-action rate, policy-block rate, approval disagreement, tool error, security tests, customer harm, operational load, and rollback performance.

Monitor by risk slice, not only averages. A low overall error rate can hide failures for a rare product or language. Revalidate when the model, prompt, retrieval source, tool schema, policy, or provider changes. Maintain a kill switch at the policy/tool layer that does not depend on the model cooperating.

Test adversarial and operational failure

Test prompt injection in a retrieved document, a model attempting an unlisted tool, parameter substitution after approval, duplicate execution after timeout, expired credentials, a compromised third-party tool, conflicting policies, missing evidence, and a model/provider change during an active case. Assert that no unapproved financial effect occurs and that the evidence still explains the block.

The architecture decision

Put the model inside the control plane, not in charge of it. Models may interpret and propose; deterministic services authenticate, authorize, enforce limits, acquire approval, execute idempotently, and record outcomes. Autonomy should be narrow, measurable, reversible, and earned one use case at a time. That architecture supports useful agents without turning probabilistic text generation into unbounded financial authority.

Related reading

• RAG for Financial Operations: Evidence Over Eloquence
• LLM Evaluation for Systems That Must Be Right

References

• Federal Reserve — Revised Guidance on Model Risk Management, SR 26-2
• Bank of England and FCA — Artificial intelligence in UK financial services 2024
• NIST — Generative AI Profile, AI 600-1

Tokenization can reduce where primary account numbers travel, but it does not automatically remove PCI DSS scope. I start by drawing the complete card-data flow, including security impact, administrative access, the vault, keys, integrations, and provider responsibilities. The checkout examples below are synthetic.

Define what a token is—and is not

Tokenization replaces a primary account number (PAN) with a surrogate value. A well-designed token reveals no PAN, has little value outside its permitted domain, and can be mapped back only through a protected token service or vault. It is not encryption: an encrypted PAN remains cardholder data, and possession of a decryption key restores it directly.

EMVCo’s Payment Tokenisation framework describes payment tokens with domain controls and lifecycle functions. Its current specification page listed version 2.4, published 9 July 2026, at research time. Network payment tokens, a processor vault token, and an internal alias solve different problems; document which one is used and where.

The PCI Security Standards Council’s Tokenization Guidelines explicitly state that tokenization does not replace PCI DSS requirements and that systems with access to PAN, tokenization components, and systems that can affect their security may remain in scope. The supplement dates from August 2011, so it should be read with the current standard and assessor guidance rather than treated as a complete modern scoping rule.

Draw the data flow before drawing the boundary

Inventory every path on which PAN can appear: browser or mobile capture, API request, memory, log, trace, queue, retry store, support tool, analytics export, backup, error report, and disaster-recovery environment. Mark token flows separately. Include administrators, deployment systems, secrets managers, vulnerability tooling, and network controls that can change the card-data environment.

A common lower-exposure design uses processor-hosted fields or a hosted payment page so PAN is submitted directly from the customer’s browser to a validated payment provider. The application receives a token. That can reduce exposure, but the page, scripts, redirects, and systems that can affect the payment form may still carry obligations. The exact self-assessment questionnaire and scope depend on the integration and acquiring relationship; obtain an assessor or acquirer determination rather than promising an outcome in architecture documentation.

PCI SSC identified PCI DSS v4.0.1 as a limited revision and retained 31 March 2025 as the effective date for the future-dated requirements. Pin the version used by the assessment and verify current PCI SSC publications before release.

Isolate the token service

Place PAN capture, token generation, vault storage, and detokenization in a restricted card-data zone. Ordinary product services should accept only tokens and last-four/brand metadata needed for user experience. Enforce at multiple layers:

• separate network and identity boundaries;
• least-privilege service identities, with no shared vault credential;
• authenticated encryption and managed keys separated from stored ciphertext;
• explicit allowlists for detokenization purpose and destination;
• immutable audit records for vault administration and reveal operations;
• PAN discovery scans on logs, object stores, queues, tickets, and data warehouses;
• retention and deletion rules for PAN, tokens, mappings, and backups.

Do not rely on developers remembering not to log a field. Reject PAN-shaped data at API boundaries that should be token-only, redact observability pipelines, and test error paths. Use synthetic test numbers outside production; never copy live card data into lower environments.

Make tokens domain-constrained

A token valid everywhere becomes a bearer secret with a different shape. Bind tokens to a merchant, channel, device, transaction type, or processor where the token scheme supports it. Separate display aliases from payment credentials. Rotate and suspend tokens through an explicit lifecycle, and propagate lifecycle events without exposing PAN.

Token format also matters. A format-preserving token can pass legacy validation and accidentally enter fields assumed to contain PAN. Tag values with a token type and provider; never infer semantics solely from length or prefix. Keep the mapping between internal payment method identifiers and external tokens versioned and auditable.

Control detokenization as an exceptional operation

Most application workflows should never reveal PAN. Where a business process genuinely requires it, require a narrowly scoped service identity, purpose code, destination restriction, rate and volume limit, approval for high-risk use, and complete audit trail. Return the minimum data for the shortest time and prevent it from entering logs or traces.

Monitor reveal attempts, denied purposes, unusual volume, new callers, vault administration, key access, PAN discovery findings, and token use outside its allowed domain. Include the vault and token provider in incident-response and business-continuity exercises.

Validate the claim about scope

Test the architecture with data-flow review, code and configuration inspection, PAN discovery, segmentation testing, access review, and provider evidence. Reassess after changes to checkout scripts, observability, support tooling, processor routing, backups, or deployment privileges. A new “harmless” analytics integration can reconnect an otherwise separated environment.

Scope reduction is a conclusion reached from evidence by the organization, its acquirer, and where appropriate a Qualified Security Assessor—not a feature flag. Architecture should say “designed to minimize PAN exposure,” then record the formal determination separately.

The architecture decision

Capture PAN in the smallest isolated zone, replace it with a clearly typed and domain-constrained token, block PAN at ordinary service boundaries, and make detokenization rare and observable. Tokenization is valuable because it reduces exposure and simplifies controls. It is dangerous when it becomes a slogan that hides remaining security dependencies. Design the complete flow, validate it against PCI DSS v4.0.1 and current guidance, and let evidence—not the presence of a token—determine scope.

Related reading

• Real-Time Fraud Decisions Under 100 Milliseconds
• Open Banking Consent as a First-Class State Machine

References

• PCI SSC — Tokenization Guidelines
• PCI SSC — PCI DSS v4.0.1 publication notice
• EMVCo — EMV Payment Tokenisation

In a payment API, a timeout means “the caller does not know,” not “the payment failed.” My design rule is to separate transport recovery from financial truth and give every operation a durable identity that can be queried before repetition. The checkout cases below are synthetic.

Classify the operation before choosing a retry policy

HTTP method names are only a starting point. RFC 9110 section 9.2.2 defines idempotent methods by their intended effect and says clients should not automatically retry a non-idempotent request unless they know its semantics are idempotent or can detect that the original was not applied.

A balance query is normally safe to repeat. Creating a payment is not safe unless the API binds repeated attempts to the same durable operation. A capture or refund may be conditionally repeatable with an idempotency key and state transition guard. A payout routed to an external rail may remain unknown until a later status message arrives.

Publish an operation matrix for every endpoint: side effect, idempotency scope, retryable failures, non-retryable failures, unknown outcome, status-query path, callback, and maximum decision age. This makes resilience a domain contract rather than a generic HTTP middleware setting.

Use one deadline and bounded attempts

Set an end-to-end deadline at the caller and pass the remaining budget downstream. Connection, TLS, request, and response timeouts must fit inside it. A timeout should be based on measured latency distributions and business tolerance, not a round number copied between services.

The Amazon Builders’ Library article on timeouts, retries, backoff, and jitter explains why retries can amplify overload and why exponential backoff, jitter, and retry budgets are needed. Apply those techniques at one controlled layer. If a client, gateway, service, and SDK each retry three times, one request can multiply into dozens of provider calls.

Limit attempts by both count and deadline. Respect provider rate limits. Use jitter to avoid synchronized retry storms. Do not retry validation failures, authentication failures, or explicit business declines. For a timeout after request transmission, prefer querying the operation by its internal and provider references.

Make unknown a first-class state

Return a durable operation_id as early as possible. Persist the request fingerprint, caller, state, and provider attempt before initiating an external effect. If the response is lost after the provider accepts the payment, a repeated request with the same idempotency key should return the existing operation, not create another.

Model at least received, processing, succeeded, failed_definitive, and unknown_external. The last state is not an error bucket. It has an owner, a next query time, a maximum age, and a recovery route through provider status, webhook, reconciliation file, or manual investigation.

Callbacks are evidence, not magic. Verify signatures, store the raw event, acknowledge quickly, deduplicate delivery, and apply an allowed state transition. A late “succeeded” event must be able to resolve unknown_external; it must not overwrite a terminal refund or another payment’s state.

Place the circuit breaker at the provider boundary

A circuit breaker protects callers and a struggling dependency; it does not decide whether money moved. The AWS circuit-breaker pattern describes closed, open, and half-open states with controlled recovery probes.

Implement the breaker in the provider adapter, scoped so one failing provider or operation does not disable unrelated traffic. When open, reject or queue new work according to product semantics. Do not reroute an in-flight unknown payment to another provider. A fallback is safe only before any provider may have accepted the effect, or after the first outcome is definitively negative.

Expose breaker state, rejected calls, probe results, and the age of unknown operations. Operational dashboards must separate transport health from financial completion.

Design the API response for recovery

For synchronous success, return the operation identifier, stable state, and provider reference when disclosure is appropriate. For accepted asynchronous work, use an explicit processing response with a status URL and retry guidance. For unknown outcomes, state that finality is pending; do not return a generic failure that invites the user to pay again.

Status queries should be strongly tied to the original caller and protected from enumeration. They should expose monotonic financial facts while allowing operational metadata to evolve. If a previous response is replayed for an idempotency key, make that behavior observable in headers or metadata.

Test failure at each boundary

Inject loss before connect, after connect, after request bytes are sent, after provider commit, and after your database commit. Test duplicate callbacks, out-of-order events, a breaker opening during a retry, provider recovery in half-open state, exhausted deadlines, and an operation that remains unknown beyond its service target.

Assert the number of external effects, not only HTTP responses. Monitor attempts per operation, retry amplification, deadline exhaustion, breaker state, duplicate-effect prevention, unknown-outcome age, status-query success, and reconciliation breaks.

The architecture decision

Build resilience around operation semantics. Bound time with one propagated deadline, retry at one layer with backoff and jitter, guard effects with durable idempotency, place circuit breakers at dependency boundaries, and represent uncertainty explicitly. The safest payment API is not the one that retries most aggressively; it is the one that can tell a caller whether to wait, query, retry, or stop without risking a second financial effect.

Related reading

• Real-Time Fraud Decisions Under 100 Milliseconds
• ISO 20022 as a Domain Model, Not an XML Problem

References

• IETF — RFC 9110 §9.2.2, Idempotent Methods
• Amazon Builders’ Library — Timeouts, retries, and backoff with jitter
• AWS — Circuit breaker pattern

Core-banking modernization succeeds when it moves one owned capability at a time, preserves financial truth, and makes every migration checkpoint reversible. I start with the critical operation, not a fashionable target architecture; the migration examples below are synthetic.

Define the critical operation before the target architecture

Begin with the customer or business operation that must remain available: posting a transfer, calculating interest, servicing a loan, or producing a statement. Map its data, batch dependencies, downstream reports, manual controls, and recovery objectives. The Basel Committee’s Principles for Operational Resilience call for identifying critical operations, mapping dependencies, setting tolerances for disruption, and testing the ability to stay within them. Those principles apply directly to banks; they are also a disciplined frame for any regulated modernization.

A service decomposition is not the objective. The objective is to change a capability without losing balances, contractual behavior, audit evidence, or recoverability. A modular monolith may be a better destination than microservices if it gives clear ownership with lower operational cost.

Choose a bounded migration slice

The AWS strangler-fig pattern describes routing requests through a façade while new components incrementally replace old functionality. Its value is controlled scope, not a guarantee of safety.

Select a slice with a coherent business boundary and measurable exit criteria. “Customer profile” can still be unsafe if dozens of legacy jobs update it. “Read-only statement history for accounts opened after a cutoff date” is often easier to own. Document:

• commands and queries in scope;
• source of record for every field and business date;
• allowed writers during each phase;
• reconciliation and acceptance thresholds;
• rollback procedure and maximum rollback age;
• legacy dependencies that must be removed before completion.

Never leave “temporary” dual write as an architectural assumption. Two systems accepting independent updates create conflicts that timestamps cannot reliably solve. Prefer one authoritative writer, with changes propagated through a transactional outbox or controlled replication. If dual write is unavoidable for a short phase, define conflict ownership, failure handling, and a hard removal date.

Isolate legacy semantics

Legacy interfaces often encode product rules in field names, batch timing, or error codes. Do not spread those semantics into every new service. An anti-corruption layer translates between the legacy model and the modern domain boundary.

The layer should be intentionally narrow. It validates requests, maps identifiers and values, preserves the original reference, and exposes uncertainty rather than inventing success. Version mappings and test them against recorded, anonymized or synthetic cases. If it grows into a second core containing business policy, the migration has merely moved the monolith.

Separate data movement from authority transfer

Copying data does not make the destination authoritative. Treat migration as distinct stages:

1. baseline: inventory and checksum the source population;
2. backfill: copy historical data with transformation lineage;
3. catch-up: apply ordered changes after the baseline point;
4. shadow read: compare outputs without serving customers;
5. controlled cutover: route a defined cohort or capability to the new owner;
6. stabilize: monitor, reconcile, and retain a tested rollback path;
7. decommission: remove legacy writers, jobs, credentials, and data copies.

Change-data capture can support catch-up, but it does not explain business meaning. Deletes, out-of-order records, schema changes, and updates performed outside the captured log all need handling. Record a migration watermark and prove that every source change before it is present once in the destination.

For financial records, compare more than row counts. Reconcile opening and closing balances, transaction counts and amounts by currency, posting dates, account states, and independently computed control totals. Differences become owned cases; they are not automatically “fixed” by copying the latest value.

Make routing and rollback observable

The façade should emit the capability, cohort, chosen backend, rule version, operation identifier, latency, and outcome. A feature flag alone is not a migration control. Routing changes need approval, staged rollout, and an auditable history.

Define rollback before cutover. If the new system has accepted writes, switching traffic back may require replaying those facts into the legacy system or choosing forward recovery instead. A rollback plan that says “turn off the flag” is valid only while the legacy side remains complete and compatible.

Test failure during backfill, lag in replication, a poison record, schema drift, legacy batch overlap, a partial cohort cutover, and loss of the façade. Rehearse with production-shaped synthetic data and a clock that crosses end-of-day and month-end boundaries.

Know when the slice is finished

A capability is not modernized while the legacy system still computes its results, repairs its data, or remains the only recovery path. Exit criteria should include no legacy writers, reconciled history, operational runbooks, demonstrated recovery, performance within objectives, trained support teams, retired credentials, and a retention decision for old data.

Track business outcomes: change lead time, incidents, reconciliation breaks, recovery time, manual work, and cost per operation. Counting new services rewards movement rather than improvement.

The modernization decision

Use a façade to control traffic, an anti-corruption layer to contain old semantics, one writer to preserve authority, and explicit checkpoints to move data and responsibility separately. Modernization is complete only when a bounded capability has one clear owner and the obsolete path can be removed. The safest program is not the one with the most ambitious target diagram; it is the one that can explain and reverse each step without compromising financial truth.

Related reading

• ISO 20022 as a Domain Model, Not an XML Problem
• Real-Time Fraud Decisions Under 100 Milliseconds

References

• AWS — Strangler fig pattern
• AWS — Anti-corruption layer pattern
• Basel Committee — Principles for operational resilience

Reconciliation is not a month-end spreadsheet exercise. I design it as a control system that compares independently produced financial facts, proves completeness, classifies differences, and routes exceptions to accountable owners without erasing evidence. The settlement cases below are synthetic.

Independence is the point

Comparing a balance projection with another projection built from the same tables can reproduce the same defect twice. A meaningful control compares records created through independent paths: for example, an internal ledger against a processor settlement file, a bank statement, or a scheme report.

The principle is consistent with BCBS 239, which emphasizes accuracy, completeness, timeliness, adaptability, and traceability in risk-data aggregation. BCBS 239 directly applies to banks’ risk-data capabilities, not to every payment company; here it is a rigorous analogue for designing financial-data controls. The Basel Committee’s January 2026 implementation newsletter also identifies fragmented data estates, weak lineage, and manual processes as continuing obstacles.

Prove completeness before comparing values

A matching engine cannot detect a missing file it never expected. Begin each run with a source manifest:

• source, business date, timezone, and cutoff;
• file or API page sequence and expected count;
• record count, control total, and cryptographic checksum where available;
• schema and producer version;
• ingestion start, completion, and retry history.

Quarantine malformed or incomplete deliveries. Do not mark a business date reconciled because the records received happen to match. A green result requires evidence that the population itself is complete.

Normalize without destroying provenance. Convert external identifiers, signs, currencies, decimals, and timestamps into a comparison model, but retain the raw source reference and transformation version. ISO 20022’s standards repository provides official payment and cash-management message definitions; a local implementation must still pin the message version and market-practice profile it accepts.

Matching is a sequence of explainable rules

Start with deterministic exact matches using stable references, amount, currency, and expected date. Then apply controlled rules for known differences:

1. one-to-one: one ledger movement equals one external movement;
2. one-to-many: one settlement deposit corresponds to many captured payments;
3. many-to-one: split internal postings correspond to one provider record;
4. timing: both facts are valid but fall on different business dates;
5. value: amounts differ because of fees, foreign exchange, rounding, or an error;
6. missing/duplicate: a fact exists on one side only or appears more than once.

Every rule needs an identifier, version, effective date, owner, priority, and test set. Record which rule produced a match and the evidence it consumed. Tolerances must be justified by currency, rail, or contract; “less than 1” is not a safe global rule. Automatic write-offs need separately approved thresholds and aggregate monitoring so many small differences cannot hide a material loss.

Exceptions are products, not leftovers

An unmatched record should become a durable case with a reason category, amount at risk, age, owner, service-level target, and links to both source facts. Distinguish “awaiting expected late file” from “unknown value difference” and “suspected duplicate.” That lets operations prioritize risk instead of working a flat queue.

Investigation notes and evidence belong to the case. If an adjustment is approved, post a new balanced ledger transaction with the case identifier, reason code, approver, and effective date. Never edit the original entry to force a match. Re-run the comparison after the correction and retain both the pre- and post-adjustment results.

Ageing matters as much as counts. Monitor unreconciled value and item count by reason and age bucket, source completeness failures, late deliveries, tolerance usage, manual-adjustment volume, reopened cases, and the percentage matched by each rule version. A falling exception count can be bad news if a broad new rule is swallowing genuine breaks.

Design the operating boundary

The ledger owns internal postings. The ingestion layer proves and stores external evidence. The reconciliation service compares normalized facts and records decisions; it does not silently create money movements. The case workflow owns investigation and approvals. A posting service executes approved adjustments through the same ledger controls as any other financial transaction.

This separation makes replay safe. If a parser or matching rule changes, rebuild normalized facts and match decisions from immutable source snapshots. Compare old and new results before promoting the rule. Do not re-download a mutable provider endpoint and assume it represents what was originally received.

Test the control, not only the code

Use synthetic scenarios for missing files, truncated pages, duplicate records, daylight-saving cutoffs, one-to-many settlements, partial refunds, fee netting, currency precision, late reversals, and provider corrections. Inject a defect into a complete dataset and verify that the expected exception is created, aged, approved, adjusted, and closed with evidence.

Also test segregation of duties: the person who changes a matching rule should not approve the breaks it automatically resolves. Access to raw account data, case evidence, and adjustment actions should be independently authorized and logged.

The design decision

Reconciliation is a continuous, evidence-preserving control loop. Prove the population, normalize with lineage, match through versioned rules, classify every difference, and correct through controlled new entries. A spreadsheet can support investigation, but it cannot be the authoritative workflow. When reconciliation is designed into the platform, every green result explains what was compared, which rules were used, and who accepted the remaining risk.

Related reading

• ISO 20022 as a Domain Model, Not an XML Problem
• Double-Entry Ledger Architecture for Product Engineers

References

• Basel Committee — Principles for effective risk data aggregation and risk reporting
• Basel Committee — BCBS 239 implementation newsletter, January 2026
• ISO 20022 — Standards repository

A card payment is not one transaction moving neatly from “pending” to “complete.” It is a set of related financial events—authorization, presentment, clearing, settlement, reversal, refund, and sometimes dispute—created by different actors on different clocks. My design rule is to correlate those events without collapsing them into one mutable status; the purchase scenarios below are synthetic.

Start with two timelines, not one status

At authorization time, the merchant sends an amount and card context through its acquirer and the card network to the issuer. The issuer approves or declines and returns identifiers that help later messages refer to the decision. Approval normally causes the issuer to reserve available funds; it does not prove that money has settled to the merchant.

Presentment happens later, commonly in a batch. Its amount can differ legitimately from the authorization because of tips, estimated amounts, partial fulfilment, currency conversion, or multiple shipments. Clearing calculates obligations among participants; settlement moves funds according to the network arrangement. A refund is a new credit transaction after capture, while a reversal tells the issuer that an authorization is no longer needed. Those are not interchangeable operations.

That distinction matters to customers. A stale authorization can leave funds unavailable even though the merchant will never capture them. Visa’s merchant guidance explains that an authorization reversal should carry matching transaction information so the issuer can identify and release the hold. Exact fields and deadlines vary by network, region, merchant category, and processor contract; the guidance is an example, not a universal clock.

Available balance and ledger balance should therefore remain separate concepts. The issuer may expose a pending hold immediately while the settled ledger changes later. On the merchant side, an approved authorization is not yet revenue or cash. Product screens may simplify these views, but their labels and support tools must preserve the distinction.

Model correlation without collapsing events

Use a stable internal payment_id, but give every network-facing event its own identifier and immutable record. A practical model separates:

• authorization: requested amount, approved amount, decision, network references, expiry policy;
• presentment: presentment identifier, related authorization, amount, currency, sequence, received time;
• reversal: full or partial amount, reason, related authorization, acknowledgement state;
• refund: a new movement linked to the captured payment, with its own lifecycle;
• dispute: case identifier, disputed amount, condition, evidence deadline, and outcome.

Store both event time and ingestion time. The former says when the network says something happened; the latter explains when your platform learned it. Never overwrite an authorization amount with a clearing amount merely to make a single row “current.” Derive a customer-facing projection from the event history and retain the facts required to rebuild it.

The same rule applies to state. authorized, partially_presented, fully_presented, reversed, expired, and disputed may coexist across related records. A payment aggregate can summarize them, but its state machine must reject impossible transitions and expose uncertainty when a provider outcome is unknown.

Design for asynchronous and imperfect delivery

Processors frequently report later changes through webhooks. Stripe’s official webhook guidance documents duplicate delivery, asynchronous retries, and the absence of ordering guarantees. Treat that as a representative operational constraint, not as a rule for every processor.

Verify signatures, persist the raw notification, acknowledge quickly, and process from a queue. Deduplicate by the provider’s event identifier, while also making the domain handler idempotent. An event saying “payment updated” is a notification to query or apply a specific change; it is not automatically an accounting posting.

Unknown outcomes deserve an explicit state. If an authorization request times out, do not immediately route it to another acquirer: the first issuer may already have approved it. Query using the provider reference or wait for a definitive callback. Escalate aged unknowns to reconciliation instead of guessing.

Treat disputes as a separate case workflow

A dispute can begin weeks or months after authorization and settlement. It has reason-specific evidence, response deadlines, provisional financial effects, and an outcome. Visa’s Dispute Management Guidelines for Merchants, June 2024 illustrate why a dispute is not simply a terminal payment status.

Keep case management separate from the payment state machine. Link the case to the original authorization and presentment, then post fees, provisional credits, reversals, or final losses as distinct ledger transactions. This preserves both the operational history and the accounting explanation.

Controls engineers can test

Test more than the happy path:

1. an approved authorization followed by no presentment and an explicit reversal;
2. one authorization followed by two partial presentments;
3. presentment received before a delayed authorization notification;
4. duplicate reversal and dispute webhooks;
5. a timeout followed by a late approval;
6. refund requested during an open dispute;
7. currencies or amounts that do not match within the permitted scheme rules.

For every test, assert the hold projection, customer-visible status, ledger entries, reconciliation result, and audit trail. Monitor authorization-to-presentment age, unmatched presentments, unreleased holds, unknown outcomes, duplicate notifications, and dispute deadlines.

The architecture decision

Do not build a card platform around one mutable payment_status. Build it around correlated, immutable events with explicit ownership and separate clocks. The aggregate is useful for product experiences, but authorization, presentment, reversal, refund, settlement, and dispute must remain independently explainable. That is what lets support answer a customer, finance reconcile a processor, and engineering recover safely when messages arrive late or twice.

Related reading

• Double-Entry Ledger Architecture for Product Engineers
• ISO 20022 as a Domain Model, Not an XML Problem

References

• Visa — Authorization Reversals
• Visa — Dispute Management Guidelines for Visa Merchants, June 2024
• Stripe — Webhooks

I start with the authorization the user actually granted: scope, purpose, actors, time, and terminal states. I do not reduce open-banking consent to customer_consented = true.

State the jurisdiction before the schema

Consent semantics differ across UK Open Banking, EU payment-services rules, UAE Open Finance, and other ecosystems. This article uses the UK Open Banking Implementation Entity (OBIE) Read/Write API v3.1.2 as a concrete engineering example; it is not a universal legal model.

The OBIE Account Access Consents specification defines a consent resource with permissions, creation and status-update timestamps, optional expiration, and states including AwaitingAuthorisation, Authorised, Rejected, and Revoked. It also defines POST, GET, and DELETE behavior. The related Read/Write API Profile explains that an intent carries fine-grained permissions beyond coarse OAuth scopes.

Those versioned specifications support the architecture below. They do not settle what counts as legally valid consent in another jurisdiction; product, compliance, and legal owners must map the applicable rules.

Consent, authorization, and tokens are different objects

A third-party provider can create an account-access consent describing requested permissions and expiry. The bank authenticates the payment-service user and asks them to authorize that intent. Only then does the consent become Authorised. An OAuth authorization code and resulting access token let software exercise the authorization.

Here is how I would keep the boundaries visible:

• Consent resource: who requested what, for which purpose and accounts, under which policy/version, and in which lifecycle state.
• Authorization session: evidence of user authentication, disclosure presented, selections, and the authorization result.
• Access/refresh tokens: technical credentials with audience, scope, sender constraint, and expiry.
• Resource server decision: evaluates the current consent, token, account entitlement, and policy for each request.

A token can expire while a long-lived consent remains authorized; OBIE v3.1.2 explicitly describes this separation. Conversely, revocation should make further resource access fail even if a token’s cryptographic expiry is later.

Use a real state machine

Create the consent in AwaitingAuthorisation. From there, a successful user journey moves it to Authorised; rejection moves it to Rejected; cancellation or timeout follows the adopted profile. Rejected and Revoked are terminal in the cited OBIE profile. Never move a revoked record back to authorized; create a new consent with new evidence.

Make transitions conditional and atomic. Store status, status_updated_at, version, and an append-only transition log. A concurrent revocation and data-access request must resolve against a declared consistency rule. For high-value access, the authorization decision should read current consent state strongly enough that a completed revocation cannot be ignored by a stale cache.

Consent expiry is a scheduled transition or an evaluated condition, not a nightly cleanup detail. Record the exact instant and time zone, and test access at the boundary.

Model scope as data, not prose

Permissions need machine-enforceable identifiers. Bind the consent to the authorized accounts or selection rule, data clusters, transaction-history window, expiry, requesting party, bank, and user. Store the disclosure/policy version and a hash or immutable reference to what was shown.

Purpose can be legally significant but difficult to enforce technically. Keep it explicit and use it in downstream policy, monitoring, and audit. Do not assume an OAuth scope such as accounts captures account selection, date limits, or purpose.

For security-sensitive financial APIs, the OpenID Foundation’s FAPI 2.0 Security Profile defines protections such as sender-constrained access tokens. It reached Final status on 20 February 2025. Adoption still depends on the ecosystem profile; cite the version actually implemented.

Revocation is an end-to-end operation

Revocation must update the consent authority, invalidate or block tokens as required, clear caches, stop scheduled collection, and propagate to downstream data products. The OBIE consent specification says that when a user revokes through the third party, that party calls DELETE on the consent resource before confirming revocation.

Keep evidence of the revoked resource and transition even if the API presents deletion semantics. Data already collected has a separate retention and deletion policy; revoking future access does not automatically define how prior records must be handled.

Test the uncomfortable cases

Exercise authorization abandonment, user rejection, partial account selection, consent expiry during token refresh, revocation concurrent with data retrieval, duplicate callbacks, clock skew, third-party certificate rotation, and a downstream job running from cached permission data.

An auditor should be able to answer: what was requested, what was shown, who authorized it, which accounts and data were included, which policy applied, when access occurred, and why it stopped. Do not log raw credentials or unnecessary personal data to achieve that evidence.

Operational measures include authorization completion, rejection, revocation propagation time, access denied by reason, expired-consent use attempts, stale-cache detections, and downstream jobs stopped after revocation.

The engineering rule

My design rule is to make consent a first-class domain aggregate and let tokens reference it. I evaluate both on protected access, preserve transition evidence, and qualify the model by jurisdiction and specification version. A boolean can say “yes”; it cannot explain what was authorized or which system must stop when the answer becomes “no.”

Before launch, walk one consent through authorization, data access, token expiry, re-authentication, user revocation at each channel, and downstream deletion or retention handling. Confirm that policy and customer wording match the implemented state machine. Repeat the review when the jurisdiction, profile version, requested data cluster, or third-party role changes; those changes can alter both the contract and the evidence required.

Related reading

• Double-Entry Ledger Architecture for Product Engineers
• Idempotency at Scale: More Than a Request Header

References

• OBIE: Account Access Consents v3.1.2
• OBIE: Read/Write Data API Profile v3.1.2
• OpenID Foundation: FAPI 2.0 Security Profile—Final

I start a real-time fraud design by defining the stopwatch and the cost of a wrong decision. A sub-100-millisecond target is useful only when its boundary, percentile, degraded mode, and loss trade-off are explicit.

Define what the stopwatch measures

“The model responds in 12 ms” says little about an authorization decision. A useful service-level objective might be: p99 under 100 ms from the risk API receiving a complete request to returning approve, decline, or step-up, measured in one region at a stated throughput. Network travel to the merchant, 3-D Secure interaction, issuer processing, and processor latency are outside that boundary and must be reported separately.

There is no universal 100 ms fraud standard. It is an architectural budget. Publish p50, p95, and p99, workload, geography, and measurement point. Otherwise a precise title creates false confidence.

EMVCo’s official 3-D Secure material shows the broader risk-based context: lower-risk transactions may follow a frictionless flow, while higher-risk transactions can require additional authentication. The current protocol details are versioned in the 3-D Secure specification hub. A platform must state which 3DS version and market rules it implements.

Build the budget backwards

Here is how I would sketch a synthetic 100 ms internal p99 budget:

These numbers are synthetic, not a benchmark. The reserve matters because tail latency compounds across dependencies. The service should avoid synchronous calls to a data warehouse, case-management tool, or third-party enrichment that cannot meet the same budget.

Split features by freshness and consequence

Compute stable attributes—merchant history aggregates, known device relationships, account tenure—offline or through streaming pipelines into a low-latency online store. Compute request-local signals such as amount deviation and device velocity at decision time. Every feature needs an owner, definition, freshness timestamp, default behavior, and training/serving consistency test.

Missing data is not zero. If device history is unavailable, expose missing and the cause. Models trained on complete data can fail silently when the online store degrades. Monitor feature age and missingness by segment, not only overall.

Rules and model scores often run in parallel. A deterministic sanctions or blocked-credential rule may override the model. A model score may select approve, decline, or step-up thresholds by product and risk appetite. Record rule version, model version, feature-set version, threshold policy, and reason codes with each decision.

Design degradation before the outage

When the feature store times out, “fail open” and “fail closed” are both too crude. Define a tiered policy:

• low-value, established-customer traffic may use a conservative reduced feature set;
• high-risk operations may step up authentication;
• materially exposed operations may decline or queue when the product permits;
• the platform may reduce traffic or disable an unstable model version.

The policy should be approved, tested, and observable. NIST AI RMF 1.0 frames AI risk work across Govern, Map, Measure, and Manage; it is voluntary guidance, but its lifecycle approach is useful for documenting owners, testing, monitoring, and response.

Optimize expected loss, not a single metric

Accuracy is usually misleading for rare fraud. Evaluate precision, recall, false-positive rate, approval impact, calibration, and monetary outcomes by relevant slices. A simple decision cost can include fraud loss, authentication cost, interchange or processing cost, customer abandonment, and support burden.

Thresholds should reflect asymmetric consequences. A false decline can lose a good customer; a false approval can create fraud loss and operational work. Report both, and check whether performance differs across regions, devices, customer tenure, or other legally and ethically reviewed segments.

Close the delayed-label loop

Fraud labels arrive late and imperfectly through disputes, confirmed account takeover, investigation, and recovery. Store the label source and observation window. Do not treat every chargeback as fraud or every unreported transaction as legitimate.

Case management is asynchronous. Investigators need the decision evidence available at the time—not a reconstruction using today’s features and model. Their outcomes feed a governed training dataset with leakage checks, lineage, and approval. Compare production decisions with later outcomes to detect drift and threshold decay.

Operate the whole decision system

Monitor latency, errors, feature freshness, reduced-mode use, score distribution, rule hit rate, step-up rate, approval, confirmed loss, false positives, and label maturity. Shadow a candidate model before promotion and retain a rapid rollback path. Repeated trials and segment-level evaluation matter because aggregate averages hide tail failures.

My design rule is to keep the synchronous path small and deterministic, then move enrichment, investigation, training, and analytics off it. Under 100 ms can then be an honest SLO for a defined boundary—not a promise detached from financial outcomes.

Before promotion, run the candidate against a time-separated, versioned dataset and a production shadow stream. Review performance by material segments, repeat trials when inference is nondeterministic, and define rollback thresholds before seeing the result. Validate the degraded feature set as a separate decision system. Finally, confirm that investigators can reconstruct a historical decision without retrieving mutable features from today’s store.

Related reading

• Idempotency at Scale: More Than a Request Header
• Double-Entry Ledger Architecture for Product Engineers

References

• EMVCo: EMV 3-D Secure
• EMVCo: 3-D Secure Specification v2.3.1 hub
• NIST: AI Risk Management Framework 1.0

I start ISO 20022 work with the financial concept, not the XML tag. The standard is most useful as a governed vocabulary; an ingestion-only project can preserve syntax while losing meaning.

The standard is larger than a schema

The official ISO 20022 Business Model describes industry-agreed financial concepts and relationships used to derive message elements. The ISO 20022 FAQ clarifies that the standard defines a development methodology and repository, while XML and ASN.1 are generated syntaxes. The repository includes a Data Dictionary and Business Process Catalogue.

That distinction changes implementation. Dbtr, Cdtr, and EndToEndId are not valuable because their tag names are standardized; they are valuable because communities agree on their roles, cardinality, and business use in a particular message definition and usage guideline.

ISO 20022 does not provide one universal runtime payload. A message family, version, market practice, scheme rulebook, and bilateral constraints determine what is actually allowed. Saying “we support pacs.008” without the version and profile is incomplete.

Use a payment example

Imagine receiving a customer credit transfer. The platform may need the instructed amount, settlement amount, debtor, creditor, accounts, agents, remittance information, purpose, and identifiers. A naive mapper copies fields into a generic Payment object and drops unfamiliar structures into metadata. That makes the happy path easy and investigations impossible.

Here is how I would build the mapping catalog for every target field:

• source message and element path;
• semantic definition and multiplicity;
• profile/version in which it is valid;
• transformation or code-list rule;
• whether absence means optional, unavailable, or invalid;
• lossiness and preservation strategy;
• owner, tests, and effective date.

Preserve the original validated message under an appropriate retention and privacy policy. The internal model can optimize product workflows, but evidence must remain available when a mapping loses detail or a scheme interpretation changes.

Choose the canonical boundary carefully

A proprietary canonical model can reduce coupling across several rails, yet it can also flatten ISO 20022’s distinctions. “Party” is not enough when the source distinguishes ultimate debtor, initiating party, debtor, creditor, and ultimate creditor. A single reference field cannot safely absorb instruction ID, end-to-end ID, transaction ID, and UETR.

Adopt ISO concepts where they fit the product. Where the internal model intentionally differs, document the anti-corruption layer and loss. The ISO page on APIs and ISO 20022 explicitly notes that ISO business semantics can provide a consistent view for API exchanges; XML is not required at every internal boundary.

Version mappings as production code

Run old and new message versions in parallel during migration. Dispatch validation and mapping by declared namespace/profile, not by guessing from present fields. Store the mapper version with the resulting payment. Golden fixtures should cover optional parties, structured remittance, charges, return information, supplementary data, and external code sets—not only a minimal sample.

Schema validation catches structure. It does not prove compliance with a market-practice rule, a valid external code on that date, or a coherent business relationship. Add profile rules and semantic assertions. For example, validate that identifiers remain unique in their defined scope and that a return references the original transaction as required by the adopted scheme.

External code sets and market profiles evolve independently of application releases. Treat them as governed reference data with provenance, activation date, rollback, and monitoring. Do not fetch mutable code lists at transaction time without a controlled snapshot.

Preserve extensions without surrendering governance

Supplementary data exists for controlled extensions, not as a dumping ground. Define the extension owner, schema, namespace, consumers, and fallback behavior. A receiving party that does not understand an extension must still process or reject the message according to the applicable profile; internal teams should not assume an extension travels across every network.

When exposing JSON APIs, retain semantic names and constraints rather than mechanically converting XML tags. Publish an API contract that traces back to the ISO concept and profile. This makes it possible to adopt another serialization without reinterpreting the domain.

Operational controls

Measure validation failures by message version and rule, unmapped-field frequency, truncation, unknown code values, mapping latency, and reconciliation breaks traced to semantic loss. Provide an operator view showing original value, normalized value, mapping rule, and profile version.

For a version upgrade, run historical messages through both mappers and classify differences. Some changes are representation-only; others change a business interpretation and require product, operations, compliance, and counterparty sign-off.

The durable architecture

My design rule is to keep four layers visible: ISO business concepts, versioned message/profile definitions, serialization adapters, and the internal domain model. I test the mappings and preserve evidence at the edges. Starting with the business model keeps the durable question in view: what financial fact does this document communicate?

Before claiming support, publish a conformance statement naming the message definitions, versions, market-practice guideline, validation layers, unsupported elements, and extension policy. Give operations a searchable mapping view and a controlled way to quarantine—not silently repair—invalid messages. Revisit the statement whenever a code set, scheme guideline, or counterparty contract changes. “ISO 20022 compatible” without that scope is too vague to test.

Related reading

• Idempotency at Scale: More Than a Request Header
• Designing an Event-Driven Ledger That Auditors Can Trust

References

• ISO 20022: Business Model
• ISO 20022: Frequently asked questions
• ISO 20022: APIs and ISO 20022

I start a ledger design by asking which financial states must be impossible to represent. Double-entry provides a powerful constraint, but only when accounts, units, time, and corrections are modelled deliberately.

Work one posting before discussing architecture

A customer tops up a wallet with AED 100. In a simplified chart of accounts, the platform gains AED 100 in a safeguarded bank asset and owes AED 100 to the customer:

The journal balances in AED. “Debit” and “credit” are directions within an account classification, not synonyms for money entering or leaving a customer UI. A payout would reduce both the cash asset and wallet liability with the directions appropriate to those account types.

Square’s public description of Books explains why it chose balanced, immutable journal entries as a production consistency primitive. David Ellerman’s mathematical treatment formalizes double-entry using pairs of unsigned quantities. Neither source replaces an organization’s accounting policy; they support the invariant, not a universal chart of accounts.

Model transactions and entries separately

Here is how I would model it. A journal transaction is the atomic business posting, carrying a transaction ID, business reference, posting-rule version, recorded time, effective time, description, and lifecycle links. Its entries carry account, direction, amount, currency or asset unit, and analytical dimensions.

At commit, enforce:

• at least two entries per journal;
• sum of debits equals sum of credits for each currency or asset unit;
• amounts use an explicit fixed precision and cannot be negative;
• accounts accept the stated unit and were open at the effective time;
• a business operation cannot post twice;
• entries cannot be updated or deleted through ordinary application paths.

PostgreSQL constraints provide building blocks such as checks, unique constraints, and foreign keys. Cross-row balance normally also needs a controlled posting transaction or database routine; a row-level check alone cannot prove that the journal balances.

Never balance currencies by accident

USD 100 debit and EUR 100 credit do not form a balanced journal. Balance per unit. A foreign-exchange trade typically needs separate balanced legs and explicit FX position or clearing accounts, plus the agreed rate and valuation evidence. Reporting can translate values into a base currency, but that valuation is a derived view—not permission to mix denominations inside the posting invariant.

Represent money as integer minor units only when the currency and product precision make that safe. Otherwise use a fixed decimal with validated scale. Floating-point arithmetic has no place in the authoritative amount.

Pending is a product decision, not a universal entry type

An authorization hold can be represented with memorandum accounts, encumbrance entries, separate available-balance reservations, or another subledger design. Posting pending and settled positions as full journal entries is one valid choice, not a law of double-entry.

Choose based on which questions the ledger must answer. If available balance is financially material, the reservation needs durable identity, expiry, partial capture, release, and reconciliation. Do not mutate a pending entry into a settled one. Link the settlement transaction to the reservation and post the release or reclassification required by the chosen accounting model.

Correct with reversals, not edits

If AED 40 was credited to the wrong customer, create a reversal transaction linked to the original and then a corrected transaction. Preserve reason, initiator, approval, and timestamps. A reversal should copy the exact original units and accounts in opposite directions; a compensating adjustment that uses different accounts is a different accounting action and should be named accordingly.

This append-only approach keeps statements, period close, and downstream events explainable. Administrative database access still exists, so “immutable” also requires access control, audit logging, backups, and integrity monitoring.

Balances are projections with proofs

The authoritative balance is the sum of entries through a defined cutoff. For performance, maintain balance snapshots or account aggregates in the same transaction as posting, or rebuild them from the journal. Store a sequence/checkpoint and test that projected balances equal recomputed balances.

Concurrency matters when the product forbids a negative available balance. Lock the relevant account, use serializable logic, or maintain a conditional balance row so two withdrawals cannot both pass the same balance check. Double-entry proves the journal balances; it does not by itself prevent overspending.

The controls that make it production-grade

For every posting, an investigator should retrieve the initiating operation, posting rule, journal, entries, reversal chain, external reference, and resulting balance. Monitor rejected unbalanced journals, duplicate business keys, stale reservations, reversal rates, projection drift, and reconciliation breaks.

Property-based tests are especially useful: generate posting inputs and assert per-unit balance, immutability, idempotency, and rebuild equivalence. Then test concrete product scenarios—partial capture, fee, refund, chargeback, and currency conversion—with accounting review.

My design rule is not merely “use debits and credits.” I want a chart of accounts and posting boundary where every movement is balanced, denominated, traceable, corrected by new evidence, and independently recomputable.

Before releasing a product flow, have accounting and engineering review its full posting matrix: initiation, pending state, settlement, fee, partial completion, expiry, refund, dispute, and correction. Include the external evidence and customer statement description for every leg. A technically balanced journal can still be economically wrong when it uses the wrong account, date, party, or valuation rule; balance is a necessary control, not the whole accounting judgment.

Related reading

• Designing an Event-Driven Ledger That Auditors Can Trust
• Idempotency at Scale: More Than a Request Header

References

• Square: Books, an immutable double-entry accounting database service
• Ellerman: On Double-Entry Bookkeeping—The Mathematical Treatment
• PostgreSQL: Constraints

I start with the business effect, not the header. Idempotency is a concurrency contract that gives repeated requests one durable operation; it is not duplicate detection after the fact.

HTTP semantics are only the starting point

RFC 9110 defines an idempotent method as one whose intended server effect is the same for multiple identical requests as for one. It classifies PUT, DELETE, and safe methods as idempotent, while warning clients not to retry non-idempotent methods automatically unless they know the application semantics are idempotent.

Payment creation is usually POST because it creates a new operation. The application therefore needs a stronger contract: a client supplies an idempotency key; the server binds that key to one canonical operation, one payload fingerprint, and one outcome. Stripe’s idempotent request documentation is a useful public example: it replays the first result, checks that reused keys have matching parameters, and documents a retention window. Those details are Stripe-specific, not properties of HTTP.

Define the namespace before the table

abc-123 is not globally unique simply because a client generated it. Build a namespace such as (tenant_id, api_operation, idempotency_key). A key used for CreatePayment should not collide with the same text used for CreateRefund, and one tenant must not be able to discover another tenant’s operation.

Canonicalize the request fields that determine the financial effect, then hash them. Exclude transport-only metadata, but include amount, currency, destination, capture mode, and any account whose change would create a different operation. Store both the fingerprint and enough protected request metadata to investigate mismatches. Reuse with a different fingerprint should return a conflict, never silently replay the old result.

Claim atomically under concurrency

The hardest race occurs when two identical requests arrive before either has written a result. A read-then-insert implementation allows both workers to execute. Use a unique database constraint and an atomic insert/claim. The winner creates an operation record; the loser reads it.

Here is how I would model the durable record:

• namespace and idempotency key;
• payload fingerprint;
• canonical operation ID;
• state such as in_progress, succeeded, failed_final, or outcome_unknown;
• response status and stable response body or resource reference;
• lease owner and expiry for recoverable work;
• creation and expiry timestamps.

When a duplicate sees in_progress, the API can return 202 Accepted with the operation URL, briefly wait for completion, or return a documented conflict/retry response. It should not start another provider call. A lease helps recover an abandoned worker, but lease expiry alone does not prove that an external side effect did not happen.

Close the crash window

Imagine the worker successfully sends AED 75 to a provider, then crashes before saving the response. Reassigning the lease and sending again can duplicate the payment. The correct recovery depends on the downstream contract:

1. pass a stable downstream idempotency key derived from the internal operation;
2. persist the provider request reference before or with submission where possible;
3. after an ambiguous timeout, query provider status or await a signed callback;
4. keep the internal state outcome_unknown until evidence resolves it.

This is why a response cache placed after the provider is insufficient. The durable operation must exist before the effect, and recovery must understand the effect’s financial semantics.

Decide what gets replayed

Validation failures that occur before execution need not reserve a key forever, but the API contract should say so. Once execution starts, replaying the same stable response—including some failures—can be safer than running again. Stripe documents that its API v1 saves the first status and body, including 500 responses; its newer API behavior differs, so never generalize a vendor rule without checking the API version.

Some results should be represented by a resource reference instead of storing a large response blob. Reconstruct the response from an immutable operation and version the representation. Otherwise a later serialization change can make the “same response” impossible to explain.

Retention, regions, and scale

Retention must cover realistic client retry, offline queue, and webhook-redelivery windows. Expiring too early permits a delayed replay to create a new effect; retaining forever increases storage and privacy obligations. After expiry, a tombstone or business uniqueness rule may still be required for operations that cannot safely repeat.

Multi-region active-active writes complicate atomic claiming. Eventual replication can allow both regions to win. Options include routing a namespace to one home region, using a globally linearizable store, or accepting provisional requests while one authority assigns the operation. Measure the consistency cost instead of claiming “global idempotency” from a local unique index.

Operational metrics should include claim contention, in-progress age, fingerprint conflicts, unknown outcomes, expired replays, storage growth, and hot keys. Load tests must send concurrent duplicates, not only unique traffic.

The contract to publish

My design rule is to publish the contract—supported operations, namespace, retention, mismatch behavior, in-progress response, and error replay—and enforce the same identity downstream. At scale, I earn idempotency with one atomic claim and one recoverable operation, not with the presence of a request header.

Review the contract with failure injection, not only API examples. Pause a worker after its provider call, race one hundred identical requests, reuse a key with one changed field, fail over between regions, and replay after expiry. Inspect both the external side effect and the internal operation. The pass condition is one explainable effect; a collection of identical HTTP responses can still hide duplicated money movement.

Related reading

• Designing an Event-Driven Ledger That Auditors Can Trust
• Payment Orchestration Without the Spaghetti

References

• IETF RFC 9110 §9.2.2: Idempotent Methods
• Stripe: Idempotent requests
• Stripe: Handling duplicate webhook events

I use events to distribute financial work, but I do not let the message stream declare itself the ledger. My starting point is an explicit posting boundary, correction model, and replay contract.

Three records that teams often confuse

Consider a wallet transfer of AED 250. The request to move funds is a command: it can be rejected for insufficient available balance. The balanced debit and credit are accounting facts: once booked, they are corrected with new entries rather than edited. TransferBooked is an integration event: a versioned notification that helps other services react.

These records can share an identifier, but they do not have the same authority. A Kafka topic is not automatically a journal, and an event-sourced workflow is not automatically double-entry accounting. If downstream consumers independently infer financial postings from an evolving business event, the same transfer can acquire different meanings in settlement, statements, and reporting.

Square’s engineering account of Books describes an immutable, balanced accounting service as the consistency primitive. AWS’s transactional outbox guidance explains the complementary delivery pattern: write domain state and an outbox record in one database transaction, then publish asynchronously. It also states that duplicates remain possible and consumers should be idempotent.

Put the transaction boundary around the posting

Here is how I would define the posting boundary. The posting service validates the journal before commit:

• every entry belongs to one transaction identifier;
• debits and credits balance for each currency or asset unit;
• accounts, dimensions, and posting rule version are valid;
• monetary values use explicit minor-unit or decimal precision;
• the command’s business key has not already produced a posting;
• recorded time, effective time, and source time are preserved separately.

Inside one database transaction, insert the immutable journal entries and an outbox row describing the committed transaction. Do not publish first and hope the database commit succeeds. Do not write the ledger and broker independently. The outbox closes that dual-write gap, but it does not provide exactly-once delivery; the published event must carry a stable event ID and transaction ID.

This model also clarifies what “event-driven ledger” should mean: events drive workflows around an authoritative ledger, and committed postings emit events. It does not mean any event stream becomes financially authoritative by declaration.

Design corrections as accounting events

Suppose an operator booked AED 250 to the wrong merchant account. Editing the row destroys the evidence seen by previous statements and consumers. Instead, post a reversal linked to the original transaction, then post the corrected transaction. The current balance changes, but the historical explanation remains intact.

Corrections need a reason code, initiator, approval evidence where required, original transaction reference, and a new idempotency key. “Reversal” should not be used as a generic delete. Some rails reverse a pending authorization; the ledger may instead record release of an encumbrance. The accounting representation must follow the product’s chart of accounts and posting policy.

Make time and versions first-class

Financial systems have several clocks. recorded_at tells auditors when the platform accepted the posting. effective_at tells statements and interest logic when its economic effect applies. source_occurred_at preserves the rail or provider timestamp. Late settlement data can legitimately be recorded today with an earlier effective date; collapsing the clocks makes period close and investigation ambiguous.

Event schemas evolve too. Include a schema version and stable semantic identifiers. Additive changes are easier to consume, but a changed posting meaning deserves a new event type or versioned contract. Keep the original payload when regulatory or operational evidence requires it, while ensuring secrets and personal data are protected.

Replay without paying twice

Projection consumers should store the last processed event or a deduplication record in the same transaction as their local update. A replay can rebuild balances, statements, and analytics from authoritative entries, but it must never call an external payment provider merely because an old event was read again.

Separate pure projections from effectful process managers. A statement projection can be rebuilt. A process manager that sends a payout needs its own durable command state and idempotency boundary. This distinction turns replay from a production hazard into a recovery tool.

Controls auditors can test

An audit-ready platform can demonstrate:

1. the posting rule and code version that produced every journal;
2. balance invariants checked at commit time;
3. an unbroken link from command to posting to outbox event;
4. corrections represented as linked entries;
5. delivery lag, duplicate rate, dead-letter age, and projection checkpoints;
6. periodic comparison of ledger totals with external settlement evidence.

Test the awkward moments: database commits but publisher crashes; broker redelivers after acknowledgement loss; a consumer deploy changes schema interpretation; a late correction crosses an accounting period; and a projection is rebuilt while live events continue. Recovery is complete only when balances and evidence agree.

The durable conclusion

My design rule is that events distribute a committed financial fact; they never manufacture its authority. I trust the ledger when balanced entries commit atomically, corrections preserve history, time semantics are explicit, and every derived event traces back to a posting.

Before production, document which database transaction contains the posting and outbox row, how the publisher checkpoints, and how each consumer deduplicates. Rebuild a projection in a clean environment and compare it with production totals. Then simulate a publisher crash, schema rollback, and late correction across a period boundary. If the team cannot explain the resulting journal and consumer checkpoints, replay is not yet a safe recovery mechanism.

Related reading

• Payment Orchestration Without the Spaghetti
• Idempotency at Scale: More Than a Request Header

References

• AWS: Transactional outbox pattern
• Square: Books, an immutable double-entry accounting database service
• Stripe: Webhook delivery behavior

I start with one question: who can explain the payment when two processors disagree? Payment orchestration becomes valuable when it creates one truthful lifecycle across several processors—not when it merely wraps several APIs behind another endpoint.

Start with the ambiguity, not the routing rule

Assume a merchant submits a USD 180 order. The orchestrator sends an authorization to Processor A, but the connection closes before the response arrives. Retrying immediately with Processor B may improve availability—or create two valid authorizations. The transport failed; the financial outcome is unknown.

That distinction determines the architecture. A payment intent represents the merchant’s objective. Provider attempts represent executions of that intent. An attempt may be submitted, authorised, declined, unknown, reversed, or expired without forcing the intent into the same state. The orchestrator should never compress “we did not receive a response” into “the provider declined.”

Stripe’s official documentation illustrates both sides of this problem: its orchestration retries are constrained by processor capabilities and authentication outcomes, while its webhook documentation warns that events can be duplicated and delivered out of order. Stripe Orchestration was still described as a private preview when this article was reviewed in July 2026, so it is an example rather than a universal contract.

Separate policy, execution, and financial truth

Here is how I would model it, with four distinct responsibilities:

1. Intent service: owns the merchant-facing identifier, amount, currency, capture mode, and permitted terminal states.
2. Decision service: selects a route using declared inputs such as geography, scheme, cost, processor health, risk decision, and contractual constraints. It records the policy version and explanation.
3. Provider adapters: translate canonical commands into provider-specific authorization, capture, void, and refund operations. They map provider references and error classes without owning business state.
4. State and evidence store: atomically records attempts, transitions, idempotency keys, raw provider references, signed callback metadata, and the resulting ledger instruction.

This is a control plane with branches and feedback, not a five-stage pipeline. Risk may run before route selection, after a soft decline, or during step-up authentication. Capture may occur hours after authorization. A refund is a new financial operation, not a mutation of the original attempt.

Make cross-processor retry a guarded decision

Define an operation-specific retry matrix. A deterministic validation error is not retryable. A network failure before a request leaves the adapter may be safe to retry with the same provider and idempotency key. A timeout after submission is unknown and should move to status inquiry, callback waiting, or reconciliation—not automatic cross-provider execution.

A safe failover policy might say:

• retry Processor A once only when its API guarantees idempotent replay for the same merchant operation;
• do not route to Processor B while A is unknown unless the product explicitly accepts a temporary double hold and has an automated reversal plan;
• bind any approval or 3-D Secure result to the processor and attempt for which it is valid;
• cap attempts by payment intent, not independently inside every adapter.

The routing engine should return a decision record, not only a provider name: policy_version, evaluated facts, excluded routes, chosen route, and retry budget. This makes later disputes explainable and prevents a dashboard configuration change from silently rewriting history.

Treat callbacks as evidence, not commands

Verify callback signatures, persist the raw envelope, deduplicate on provider event identifiers, and then apply a legal state transition. If an authorised callback arrives after an attempt was marked unknown, it resolves uncertainty. If it arrives after a successful failover, it triggers a duplicate-authorization workflow; it must not overwrite the winning attempt.

The callback handler should acknowledge quickly and process asynchronously. The same principle appears in Stripe’s guidance to queue complex webhook work and handle duplicate delivery. Consumers should retrieve provider state when required because event ordering is not guaranteed.

Daily settlement reports form an independent control channel. Compare processor transactions, internal attempts, and ledger postings. This catches missing callbacks, operator changes, partial captures, and provider-side adjustments. Orchestration without reconciliation only centralizes uncertainty.

What to measure

Approval rate alone rewards aggressive retrying and can conceal customer harm. Operate the platform with a balanced scorecard:

• authorization rate by route, issuer region, and policy version;
• unknown attempt rate and age;
• duplicate authorization and reversal rate;
• callback delay and reconciliation breaks;
• incremental approval uplift net of fees, fraud loss, and false declines;
• manual interventions and time to explain a payment.

Chaos tests should inject a commit-before-timeout, duplicate callbacks, late success after failover, and a provider outage. The assertion is not merely that services recover. It is that one intent produces explainable attempts, balanced ledger effects, and a deterministic customer outcome.

The design rule

My design rule is simple: let the orchestrator own the payment intent and its attempt state machine, let adapters own protocol translation, and let the ledger own financial postings. I cross a provider boundary only through a recorded decision, and I resolve unknown outcomes before creating new money movement. That boundary removes the real “spaghetti”: not API variety, but contradictory ownership of the same financial fact.

Before launch, review each operation separately—authorization, capture, void, refund, and payout. Confirm its authoritative identifier, permitted transitions, retry evidence, callback behavior, reconciliation source, and customer-visible wording. Record who may change routing policy and how an emergency change is rolled back. A failover feature should not ship until operations can identify and reverse a duplicate outcome without database surgery.

Related reading

• Designing an Event-Driven Ledger That Auditors Can Trust
• Idempotency at Scale: More Than a Request Header

References

• Stripe: Cross-processor retries with Orchestration
• Stripe: Webhook delivery and duplicate-event handling
• AWS: Saga orchestration pattern