Design case study
How I think about one decision: what to put in the grey zone between a fraud model and a blunt rule.
The shift. In card fraud the expensive error is no longer the fraud you miss, it is the good customer a blunt rule wrongly blocks.
The bet. Put a read-only LLM only in the grey band (the model scored the payment low risk, a blunt rule blocked it anyway) and make it safe with independence: two reviewers with opposite jobs, reconciled in code, not one model grading itself.
The result (45 blind cases). Handed back 67.5% of good-customer dollars, with the loss rate on what it approved at 13% versus a 26% base, and 2 of 30 good customers wrongly declined.
The edge. The two-prompt setup is easy to copy. What is not is the safety envelope that lets a bank trust an LLM on the money path, and in production the durable moat is proprietary signals (device graph, chargeback history) plugged into this structure. The design is the frame; the data is the moat.
The honest limit. It cannot catch seasoned-card account takeover, which has no tell in these features and belongs to a downstream device model.
The full engineering log, with every script and number, is in DESIGN_DECISIONS.md. Every internal number below was computed from a run I can reproduce, and the raw counts are shown next to every percentage so nothing looks more solid than it is. The demo runs are small (13 and 45 cases) and I say so wherever it matters. External figures are attributed with links.
A bank runs two gates before a card payment clears. A trained fraud model scores it, and a layer of blunt rules (velocity caps, blocklists) can hard-block it. The industry has gotten good at the first job, holding fraud losses down. The cost has moved to the other side of the line.
The unsolved money problem now is the false decline: a real, paying customer whose payment is wrongly blocked. It is a large, recognized cost: in a 2026 PYMNTS survey, 47% of merchants said false declines cost them sales (PYMNTS, 2026). And the damage is rarely one sale. A wrongly blocked customer often walks, moving their spend to a card or merchant that let the payment through, so the real loss is the future relationship, not a single transaction.
So the frontier has moved. The pain is no longer only undetected fraud. It is the good customer blocked in the grey zone, and the slow human review that zone needs.
The reflex is "build a better model, or move the threshold." That shifts the frontier, it does not remove the tradeoff. Fraud is adversarial and non-stationary: attackers adapt, models drift, so teams keep the decline rules conservative to hold losses flat. Conservative rules block more good customers by definition. The grey band is structural, not a gap one more feature closes.
I can show the band in my own data. I trained a fraud model on a public card dataset (IEEE-CIS, sklearn HistGradientBoostingClassifier, time-ordered split so the test slice behaves like a live deployment: test ROC-AUC 0.9075, PR-AUC 0.5354, base fraud rate 3.5%). Then I isolated the exact conflict zone: payments the model scored under 5% risk that a blunt rule blocked anyway.
That pool is 4,641 payments. 4,496 of them (96.9%) are good customers. Only 145 (3.1%) are fraud (curate_cases.py, under5_pool.csv). So when a blunt rule overrides a low model score, about 97 times in 100 it is blocking someone who was going to pay. That is the leak. A rule can only block. It cannot look before it blocks.
The model and rules are the setup. The real design question is what sits in the conflict band and decides whether to hand a blocked customer back. Four options I actually weighed:
| Option | What it is | Why it fails / fits |
|---|---|---|
| A. Better model / lower threshold | Retune the existing gates | Shifts the frontier, does not remove the grey band (Section 2). Rejected as a complete answer. |
| B. Static rule on the model score | "If model under X and rule fired, auto-approve" | Cannot read per-case context (tenure, velocity), cannot produce an auditable reason, and blunt thresholds are what created the false declines. Rejected. |
| C. One LLM that decides and self-checks | Single agent grades its own call (LLM-as-judge) | I built this first. It failed: one small model asked to both manufacture doubt and judge it becomes a nervous junior that finds a con everywhere and escalates almost everything, and it hallucinated signals not in the data. Rejected after measuring it fail. |
| D. Two independent reviewers, reconciled in code | A "risk" reviewer and a "revenue" reviewer judge the same evidence blind to each other; plain code combines them | Chosen. Independence is the thing a self-check structurally lacks. This is the banking four-eyes / maker-checker control. |
The rejected options carry the judgment. Option C is not a strawman, it is the obvious agent design and it is what most demos ship. I chose D because I watched C fail for a root-cause reason: doubt generation and doubt judgment conflict inside one model head, and one head has no second opinion.
FraudLens is a read-only investigator that works one blocked-but-low-risk payment at a time, only from that conflict pool. Its job is not to catch more fraud. The model and rules already catch the easy fraud. Its job is to cut the false-decline cost while holding fraud flat: hand back the genuinely good customers a rule wrongly blocked, and keep declining or escalating the rest.
One framing point a fraud team will press on: the agent is not second-guessing the fraud model. It only touches cases where a blunt rule overrode the model, blocking a customer the model itself scored as low risk. It reconciles a rule that overrode the model. It never touches the model's own confident declines.
Three stages per payment, all in agent.py.
Stage 1, Investigator. An LLM with six read-only tools (tools.py) that translate raw dataset columns into plain signals. It gathers facts and writes a neutral summary. It is explicitly barred from making the approve/decline call. The tools: get_account_history (card tenure, the anchor signal), get_velocity (burst/clustering), get_identity_match (name/address match), get_device_linkage (device consistency), get_account_details (network, product, email, geography), get_trend_brief (an ambient threat brief, a static snapshot that in production a separate batch job would refresh every 6 hours, so there is no live web call and no prompt-injection surface on the money path).
Stage 2, Review board. The same fact sheet goes to two reviewers, the same model given opposite jobs: a Risk reviewer (protect the bank) and a Revenue reviewer (protect loyal customers). Each makes its own call (approve / decline / escalate) with a written note, blind to the other. The fact sheet is built only from real tool outputs, so a reviewer cannot invent a signal.
Stage 3, Reconcile (plain code, not a third LLM). Permissiveness rank decline < escalate < approve. Rules: if they agree, act; if one says approve and the other decline (a true directional clash), escalate to a human; on any partial disagreement, take the more cautious action. Approve therefore requires both reviewers to approve. Disagreement never loosens a decision.
Permission boundaries and failure behavior. Exactly three terminal actions: approve, decline, escalate. The agent cannot move money or change an account. A hard value cap ($2,000) always escalates regardless of how clean the case looks. A spend cap aborts the run before it exceeds $2. Any model error, tool failure, or timeout fails safe to escalate. It never auto-approves on error. Cost in practice was about 1.2 cents per case (measured across both runs).
Where it deliberately does not act: the model's confident declines, anything over the value cap, and any case where the two reviewers point opposite directions. Those go to a human.
"You put a non-deterministic, injectable, hallucination-prone LLM in the middle of a money decision. A rules engine and a better model are safer."
Fair, and here is the honest answer. The LLM never sets the score and never touches the easy cases. It works only in the band the model already admitted it cannot call, where today's alternative is a slow human or a blunt rule that just blocks a paying customer. Four things bound the downside: the reviewers see only a fact sheet built from real tool outputs (not free text), so there is no live injection surface on the money path; approve needs two independent yes votes; a value cap and a fail-safe-to-escalate default cap the worst case; and the actions are read-only, so the agent cannot move money.
What it gives up, stated plainly: it is slower and costlier per case than a threshold, and it will sometimes be wrong in ways a rule would not be. That is exactly why it escalates instead of auto-declining when evidence is thin.
And the honest ceiling: there is one class it cannot catch, seasoned-card account takeover, an aged, calm, identity-matched card used by a fraudster. There is no tell in these features, which is why the trained model also scored those under 5%. Chasing them means declining good customers who look identical, which is the exact error this project exists to prevent. The right home for that case is a downstream device/behavior model, not this agent.
Raw accuracy is the wrong metric here. The question is dollars: how much good revenue did it hand back, and how much fraud leaked doing it. I ran a blind holdout of 45 cases I never curated or tuned on (30 good, 15 fraud, drawn by fixed seed from the never-seen pool with blind_sample.py; graded offline against labels the reviewers never see). The sample is deliberately fraud-heavy, 15 of 45, to stress the design, so the percentages below are within this constructed sample, not a population rate.
Let X be all dollars the rule froze, split into G (good-customer dollars, the real leak) and F (fraud dollars, correctly blocked). Y is good dollars recovered, W is fraud dollars leaked.
| Symbol | Meaning | Blind holdout (45 cases) |
|---|---|---|
| X | all frozen dollars | $1,975 |
| G | good dollars (the leak) | $1,466 (74% of X) |
| F | fraud dollars (correctly blocked) | $509 (26% of X) |
| Y | good dollars recovered | $990 (67.5% of G) |
| W | fraud dollars leaked | $148 |
| Loss rate on approved dollars | W / enabled | 13.0%, vs 26% in the sample |
By case count: of 30 good customers, 18 recovered, 10 escalated to a human, 2 wrongly declined; of 15 frauds, 2 declined, 8 escalated, 5 approved. So the agent auto-decided 27 of the 45 and sent 18 to a human. That escalation rate (40%) is high, and the segmentation below explains why: it routes thin-file cards to review rather than guessing. Of those human hand-offs, only 2 came from a true clash where one reviewer wanted to approve and the other to decline.
It recovered two-thirds of the good money while the fraud rate on everything it approved (13%) was half the fraud density of the sample it was picking from (26%). That halving is the proof it is selective, not an approve-everything machine.
I do not net Y against W into one number, because a dollar of fraud loss and a dollar of recovered revenue are not equal in cost (chargebacks, ops, lost lifetime value). I report them separately and let the reader weight them.
Cutting the blind results by card tenure: on seasoned cards (25 cases) it recovered 18 of the 19 good customers (the last one escalated) with zero wrong declines, and every recovered dollar lives there. On new/young cards (20 cases) it never auto-approved and never leaked a single fraud dollar; it escalated or declined. Both wrong declines were new-card good customers whose first purchase happened to wear the full card-testing costume, an irreducible overlap. So the honest reframe: FraudLens is a good-customer recovery layer for established cards, and a safe triage lane for thin ones. I chose not to hard-code a tenure gate, because the agent already behaves this way on its own and hard-coding it would look rigged.
For contrast, a curated 13-case showcase (clean, legible patterns, agent_run.json) recovered 7 of 8 good customers with 0 wrong declines and $2,606.63 handed back. That set is a showcase, not evidence. The blind 45 is the honest generalization number, and it is lower (60% recover, 2 wrong declines) exactly because it was not hand-picked.
Scope: third-party fraud (stolen card, account takeover, card testing) at the transaction decision point. First-party / friendly fraud is out of scope and is the larger emerging problem that needs a different system.
Source: PYMNTS, 47% of Merchants Say False Declines Cost Them Sales (2026).