Most writing about trustworthy AI is either policy prose with no system behind it, or benchmark papers with no deployment behind them. This post is neither. It's a description of how a fraud detection system running inside a real insurance claims pipeline decides how much to trust its own components — and how human adjusters, just by doing their jobs, continuously renegotiate that trust.

The system is the Phrontlyne claims agent, which I built and maintain. It processes motor insurance claims: damage photos come in, computer vision models detect damage and parts, and a fraud detector scores the claim before any money moves. The interesting part is not the detection. It's the scoring — specifically, what happens when the system's components disagree, and who gets believed.

Four signals, one score

The fraud detector is a composite of four independent layers, each answering a different question about the claim:

Cause-to-loss validation (prior weight 0.35) — does the damage match the claimed cause?
Metadata analysis (prior weight 0.25) — is the EXIF data consistent with the claim's story?
AI image detection (prior weight 0.25) — is this photo AI-generated?
Sequence validation (prior weight 0.15) — do the images form a coherent sequence?

The composite score maps to four risk bands, each with its own human-response level: below 0.25 the claim is eligible for auto-approval; 0.25–0.5 goes to manual review; 0.5–0.75 is flagged for investigation; above 0.75 is rejected. Every layer's individual contribution is logged with the decision, so an adjuster who disagrees can see which signal drove the score — the decision decomposes into contestable evidence rather than arriving as an opaque number.

For a long time, those four weights were constants. Constants are honest in one way — you can read them — but dishonest in another: they encode a guess about the reliability of each layer, made once, by me, and never revisited. If the metadata layer starts false-flagging because a popular phone model writes odd EXIF fields, the composite keeps trusting it at 0.25 forever.

Weights as beliefs

In v2.5 the weights became beliefs. Each layer carries a Beta posterior over one question: "when this layer fires, is it right?"

The priors are seeded to match the old constants — cause-to-loss starts at Beta(35, 65), metadata at Beta(25, 75), and so on — so on day one the system behaves byte-for-byte identically to the fixed-weight version. That's deliberate: a trust system that changes behavior the moment you deploy it has not earned the change.

Then the adjusters take over, without knowing it. Every time an adjuster confirms or overrides a fraud flag, the system attributes the verdict back to each layer that fired: record_fraud_layer_outcome(layer, was_correct), where correctness means the layer's firing agreed with the human's final call. Each verdict is one Bernoulli observation folded into that layer's posterior. At scoring time, the detector reads the normalized posterior means as the current weights.

After roughly a hundred adjuster labels, the evidence dominates the priors. A layer that regularly false-flags decays out of the blend. A layer that reliably catches fraud grows. Nobody tunes anything; the weights are the running answer to "who has been right lately?", computed from the people whose job is to know.

Why Bayesian, concretely

Three properties made Beta-Bernoulli the right tool rather than a fashionable one:

Cold start is graceful. A brand-new pattern with zero observations reports a success rate of 0.5 — the mean of Beta(1,1) — not a divide-by-zero or an overconfident 0.0. New signals enter the system agnostic, not trusted or distrusted.

Uncertainty is first-class. Every learned pattern exposes its 95% credible interval and effective sample size. "This pattern has 90% success" means something completely different at n=200 with CI [0.85, 0.94] than at n=10 with CI [0.55, 0.99], and the admin surface shows both. A point estimate hides exactly the thing an auditor needs.

Exploration is automatic. When several learned patterns match a claim, the system picks between them by Thompson sampling — draw once from each posterior, take the argmax. Under-tested patterns occasionally win, proportional to the probability they might actually be best. There is no epsilon hyperparameter, no exploration schedule to tune, and the exploration is itself auditable: the decision log records every matched pattern alongside the Thompson winner.

The same machinery learns thresholds. Instead of hand-tuned cutoffs for fraud similarity and auto-approval, a bucketed Beta learner watches labelled adjuster outcomes across the score range and proposes the cutoff where the posterior probability of "the agent was right" crosses 0.5 — with a confidence derived from the credible-interval widths at the boundary. The old heuristic rules ("if false positives exceed N, raise the threshold by X") are still there, but demoted to a burn-in safety net until the learners accumulate a few hundred labelled events.

The bug that taught me the most

The most instructive failure in this whole system was quiet: for a while, the write-back path from adjuster verdicts to pattern posteriors didn't exist. Patterns were selected, decisions were made, adjusters corrected them — and none of it flowed back. The system looked like it was learning. It logged like it was learning. It was not learning.

Nothing crashed, no metric alarmed, because a learning loop that silently doesn't close is indistinguishable from a working one on any single decision. You only see it in the aggregate: posteriors that never move. I now think of "is the loop actually closed?" as a first-class trustworthiness property, on par with calibration — and one that's embarrassingly easy to lose in a refactor. If you run any system that claims to learn from human feedback, go check that the feedback physically reaches the update rule. Today, not in the retro.

Who does the system learn to trust?

Here's the part that turns an engineering post into a research question. The adjusters are now inside the model. Their verdicts are the ground truth the posteriors consume — which means the system doesn't just learn about fraud, it learns the adjusters' judgment, including whatever is systematic in it: their blind spots, their regional priors, their sensitivity to certain claim types.

That cuts both ways. It's the mechanism that makes the system correctable — a human override is not just an exception handled, it's evidence incorporated. But it also means the question "is the fraud detector biased?" is inseparable from "whose feedback shaped these posteriors, on which claims, and were those claims representative?" A layer's weight is a compressed history of human decisions. Auditing the weight means auditing the history.

That's the follow-up work: slicing the posterior updates by claim characteristics to see whether the learned trust is uniform, or whether the system has quietly learned to trust its layers differently for different kinds of claimants. The machinery to do this already exists — every update is attributable to a logged verdict on a logged claim. The trail is there. Someone just has to walk it.

What this adds up to

Strip away the domain and the design pattern is general:

Decompose autonomous decisions into independently auditable signals.
Treat component trust as a posterior, seeded to current behavior, updated from human verdicts.
Expose uncertainty, not just point estimates, wherever a human reviews the system.
Log enough to attribute every belief update to the human decision that caused it.
Verify the loop is closed — mechanically, not aspirationally.

None of this required a safety team or a policy document. It required treating "how much should I trust this signal?" as a modelling problem with the same rigor as the detection itself. Trustworthiness, in my experience, is mostly built in exactly these unglamorous places: a prior that matches the constant it replaced, a credible interval on an admin page, a write-back path that actually writes back.

Kwabena Obeng

Comments (0)

Leave a comment