Building an Autonomous Insurance Claims Agent: Composite Vision, Fraud Detection, and MCP Tool Orchestration

Notes on the design of Phrontlyne Claims Agent v2, a production system that processes vehicle insurance claims end to end with a Claude driven MCP tool layer, dual YOLO detectors joined by IoU geometry, and a four signal composite fraud detector.

Context

Motor insurance claims processing is a workflow with a well defined shape and a long tail of edge cases. A policyholder submits photographs of a damaged vehicle and a narrative describing what happened. An adjuster verifies the story against the images, estimates repair cost, checks for fraud indicators, and either approves the claim, rejects it, or escalates it for investigation. The workflow is repetitive enough to automate. It is also adversarial enough that naive automation fails in expensive ways.

Phrontlyne Claims Agent v2 is a production system that automates the workflow while keeping human adjusters in the loop where it matters. It runs as a FastAPI orchestration server that exposes 45 tools to a Claude 3.5 Sonnet chat engine over the Model Context Protocol. The system integrates two independent object detectors, a retrieval augmented document layer for policies and endorsements, a four component fraud pipeline, real time marketplace price scraping, and a learning engine that consumes adjuster feedback.

This piece walks through the parts of the design that were interesting to build and that generalise beyond insurance.

The problem the system is actually solving

Any credible claims automation system has to reconcile four sources of truth that do not always agree.

What the images show.
What the policyholder said happened.
What the policy actually covers.
What the repair should cost.

A traditional pipeline handles each in isolation and glues the results together with rules. That works when the four sources agree. It fails silently when they diverge, because the failure lives in the seam between components rather than in any single one.

The design principle behind Claims Agent v2 is to model those seams as first class concerns. Every disagreement between the four sources is either explicitly reconciled, or it is surfaced as a risk signal that feeds the composite fraud score.

System architecture

The system is layered. From the top down:

Presentation layer. A Laravel Blade frontend hosts a chat widget (Ama) and a claims dashboard for adjusters. The widget communicates with the FastAPI backend over HTTP for one shot requests and Server Sent Events for streamed responses.

Controller layer. Two Laravel controllers broker traffic. ChatAgentController manages sessions, file attachments, and streaming. ClaimsAgentController handles evaluation workflows and configuration.

MCP tool layer. The FastAPI server exposes 45 tools organised into six categories: Claims Management, Damage Detection, Fraud Detection, Pricing, Approval, and Learning. The chat engine selects and invokes tools autonomously.

Service layer. A set of async HTTP clients and local inference wrappers that isolate each capability behind a stable interface.

Engine layer. The chat engine, the approval engine, and the learning engine. Engines make decisions. Services fetch facts.

The service topology in production looks like this.

Claims Agent Server (FastAPI)            port 9091
YOLO11m Damage Detection microservice    port 8080
Cause-to-Loss Validation microservice    port 8001
MySQL (phrontlynedb + phrontlyneai)      port 3306
YOLOv8 Parts Detection (ONNX)            local inference
PaddleOCR API                            remote HTTPS
Gemini Embeddings                        remote HTTPS
Jiji Price Scraping                      outbound HTTP
Laravel Frontend                         port 80 / 443

Splitting damage detection and cause validation into separate microservices was deliberate. Both are GPU sensitive workloads with their own dependency graphs. Isolating them lets each scale, restart, and version independently of the main orchestrator, and lets the orchestrator degrade gracefully when one is unavailable.

Damage detection with two models joined by IoU

The most interesting piece of engineering in the vision stack is not either detector on its own. It is the pipeline that composes them.

Two models run in parallel on every submitted image.

YOLO11m, hosted as a remote microservice, detects six damage classes: dent, scratch, crack, broken lamp, shattered glass, and flat tire. It answers the question "what is broken."

YOLOv8, running locally as an ONNX runtime, detects sixteen vehicle parts: wheel, door, fender, bonnet, bumper, grill, light, rocker panel, door knob, wind shield, roof, mirror, window panel, top fender, boot, and number plate. It answers the question "what part of the car is this."

Neither detector answers the question the claims workflow actually cares about, which is "what is broken and what does it cost to fix." That question is answered by a third component: the damage to parts association pipeline.

For every pair of a detected damage box and a detected part box, the pipeline computes Intersection over Union.

$$ \mathrm{IoU}(A, B) = \frac{|A \cap B|}{|A \cup B|} $$

The part with the highest IoU above a configurable threshold, defaulted to 0.05, is assigned as the affected part for that damage.

Three things about that formulation matter.

The threshold is low. A 0.05 IoU threshold looks permissive. It reflects a reality of the data. Damage boxes are often tight around the damage itself, while part boxes cover the entire component. A dent in the middle of a door produces a small damage box and a large door box that share only a fraction of their union. Requiring high overlap would systematically miss the association.

IoU alone is insufficient for some damage classes. Broken lamps live inside light boxes. Flat tires live inside wheel boxes. Shattered glass usually lives inside wind shield or window boxes. For these classes the pipeline applies implicit inference rules that override IoU with high confidence semantic priors.

Damage class	Inferred part	Confidence
broken lamp	light	1.00
flat tire	wheel	1.00
shattered glass	wind shield	0.85

Severity is derived from geometry. Once a damage is associated with a part, severity is computed from the ratio of the damage bounding box area to the total image area. A damage occupying under 2% of the image is minor, between 2% and 8% is moderate, above 8% is severe. This is coarse but stable. It survives changes in camera model, lighting, and framing better than any pixel level metric I tested.

Severity maps to a repair recommendation through a three factor decision matrix over damage type, part category, and severity. Minor dents on body panels go to paintless dent repair. Moderate dents go to dent repair and respray. Severe dents go to panel replacement. Crack damage on glass parts always goes to glass replacement regardless of severity, because the physics do not permit repair. Encoding these rules explicitly, rather than learning them, gives adjusters a decision path they can audit.

Composite fraud detection

Fraud in motor insurance is not a single problem. It is a family of adjacent problems that share the property that no single signal detects any of them reliably. The system's fraud pipeline reflects that.

Four independent analyses are run for every claim, each producing a score between zero and one.

Cause to loss validation. The stated cause of loss, for example "reversed into a pole," is compared against the set of damage types detected in the images. A dedicated microservice on port 8001 handles this. It normalises and tokenises the cause description, then computes cosine similarity between the cause embedding and canonical damage descriptions. A dent from a rear collision is semantically close to "reversed into a pole." A shattered windshield is not. If any significant detected damage falls below the similarity threshold, the claim is flagged.

Metadata analysis. EXIF data from submitted images is inspected for timestamp anomalies, GPS inconsistencies, device model coherence, and post processing software fingerprints. An image with an EXIF timestamp preceding the claim incident date is not automatically fraud, but it is signal.

AI image detection. Each image is passed to Claude Vision with a prompt tuned to identify artifacts of AI generation and synthetic edits. The output is a probability of the image being generated or heavily modified.

Sequence validation. When multiple images are submitted, the system checks whether they are visually consistent. Different lighting, different weather, different backgrounds across images of the same claimed incident is signal.

The four scores combine into a composite risk with fixed weights.

$$ \mathrm{risk} = 0.35 \cdot \mathrm{cause_to_loss} + 0.25 \cdot \mathrm{metadata} + 0.25 \cdot \mathrm{ai_detection} + 0.15 \cdot \mathrm{sequence} $$

The weights are not learned. They reflect a considered position about which signals are hardest to fake. Cause to loss is weighted highest because it requires the fraudster to align a narrative with visible damage, which is the hardest constraint to satisfy for opportunistic fraud. Metadata and AI detection are weighted equally because they defend against different threat models. Sequence validation is weighted lowest because it only fires when multiple images are submitted, and its false positive rate on legitimate claims is nontrivial.

The composite score maps to one of four risk levels.

Level	Threshold	Automated action	Human action
LOW	below 0.25	auto approve eligible	none
MEDIUM	0.25 to 0.50	hold, pending review	adjuster reviews
HIGH	0.50 to 0.75	flag for investigation	senior adjuster
CRITICAL	above 0.75	automatic rejection	compliance notified

Two things about this design matter. First, the system never silently rejects a claim without a clear reason chain. Every risk score decomposes into its four components, each of which decomposes into the specific signals that raised it. Second, low risk does not imply approval. It only makes the claim eligible for the approval engine to consider on its own criteria.

The RAG layer for policies and endorsements

Insurance runs on documents. Every claim is evaluated against a specific policy, which references a specific product schedule, which may have been modified by one or more endorsements. Hardcoding policy logic is not viable. The system uses retrieval augmented generation to make policy documents queryable at claim time.

The RAG pipeline has four stages.

Ingestion. Documents arrive as PDFs, images, or DOCX files. A dispatcher chooses the OCR path. Searchable PDFs go to native text extraction via PyMuPDF. Scanned PDFs and images go to PaddleOCR, hosted as a remote HTTPS API. DOCX files go to docx2txt. Pytesseract runs as a local fallback when PaddleOCR is unreachable.

Chunking. Extracted text is split into overlapping chunks sized to the embedding context window. Chunking respects paragraph and section boundaries where possible.

Embedding and indexing. Each chunk is embedded with Gemini text embedding 004 and indexed in FAISS on CPU. FAISS on CPU rather than GPU is a considered choice: the index sizes involved are small enough that GPU transfer overhead exceeds the retrieval savings.

Retrieval. At claim evaluation time, the chat engine issues queries against the FAISS index for the specific policy attached to the claim. Retrieved chunks are injected into the LLM context.

The important design decision here is that RAG is scoped per policy, not global. A general query over all documents in the corpus produces noise. A query over the exact policy, product schedule, and endorsements attached to the claim being processed produces signal.

The MCP tool layer

The chat engine is Claude 3.5 Sonnet. What makes it useful for claims processing is not the model. It is the tool interface.

Forty five tools are exposed over MCP, grouped into six functional categories.

Claims management, thirteen tools. Reading and mutating claim state. get_claim_details, list_pending_claims, create_loss_adjustment, update_claim_status, get_claim_images_by_type. This category is the surface area the model uses most heavily during a typical session.

Policy tools. get_policy_insurance_period, check_claim_in_insurance_period. Reading policy metadata attached to a claim.

Damage detection tools. detect_vehicle_damage, detect_car_parts, crop_damage_regions, analyze_damage_with_vision, detect_damage_with_parts, get_repair_cost_estimate. Wrappers around the vision services.

Fraud detection tools. validate_cause_to_loss, detect_ai_generated_image, analyze_image_metadata, analyze_image_sequence, comprehensive_fraud_analysis. Wrappers around the fraud pipeline.

Pricing tools. get_market_prices, compare_estimate_with_market, scrape_jiji_prices, get_price_statistics. Real time and cached price lookups against the Jiji marketplace.

Approval tools. check_approval_eligibility, auto_approve_claim, flag_for_manual_review, evaluate_claim_for_approval, get_approval_flow. Terminal actions.

The tools do not implement policy. They expose capabilities. Policy lives in the approval engine, which is called via evaluate_claim_for_approval and enforces business rules that must not depend on the model's judgment.

The token efficiency system

A full claims session naively expressed can consume enormous context. The image records for a single claim, the policy retrieval hits, the tool definitions for 45 tools, and the conversation history quickly saturate even a 200k window. Version 2.4.0 introduced a token efficiency system that achieves an 80 to 90 percent reduction in tokens per session without reducing capability.

Two mechanisms carry most of the weight.

Dynamic tool loading. Not all 45 tools are relevant at every turn. The system inspects the current claim state and the user's message and loads only the subset of tools applicable to the current stage. A claim in the intake stage does not need auto_approve_claim. A claim under review does not need crop_damage_regions. Filtering the tool manifest per turn saves a significant fraction of every prompt.

Three layer history compression. Recent messages are kept verbatim. Middle history is summarised into structured facts (claim ID, decisions made, tools called). Old history is compressed further into a single running state summary. Each layer preserves the information required for downstream decisions while discarding conversational surface.

The design principle: an LLM's context window is a shared resource. Every token spent on redundant history is a token unavailable for the current decision. Treating history compression as an active engineering problem, rather than as something the model handles implicitly, is what makes long sessions tractable.

The learning engine

The system does not treat its own decisions as final. When an adjuster overrides an automated decision, that override becomes training data for the learning engine.

Feedback ingestion captures the claim, the automated decision, the override, and the adjuster's reason if provided. Pattern recognition identifies clusters of overrides that share features, for example claims from a specific region where the fraud detector systematically over flags. Threshold optimisation proposes adjustments to the composite fraud weights and severity thresholds, presented to a human for approval rather than applied automatically.

Automatic self modification would be faster but not defensible. The learning engine keeps humans in the modification loop.

What was hard, honestly

Coordinating two independent detectors. Getting the damage detector and the parts detector to agree required more calibration than I expected. The 0.05 IoU threshold, the implicit inference rules, and the severity ratios were each tuned against a labelled internal set of a few hundred images. That set is small enough that overfitting is a real risk. A larger labelled dataset is the single highest leverage next investment.

The prompt engineering budget for tool selection. With 45 tools available, the model occasionally selects a plausible wrong tool. Reducing that failure rate required tool descriptions that explicitly state when the tool should not be used, and prerequisite chaining that prevents the model from calling terminal actions before it has gathered the required inputs.

Fraud pipeline calibration. The composite score weights are principled but not learned. On some claim distributions they under weight metadata evidence. A learned combiner would likely outperform the fixed weights, at the cost of interpretability. That trade off is not settled.

OCR fallback discipline. Building the pipeline to degrade gracefully when PaddleOCR is unavailable, without letting the fallback path silently produce lower quality extractions that flow through the rest of the pipeline unmarked, required annotating the extraction quality at the source and threading that annotation through every downstream stage.

What I would change

A learned severity model. The area ratio heuristic is stable but coarse. A small classifier trained on damage crops with severity labels would improve severity precision without breaking auditability.

Retrieval reranking on policy RAG. Dense retrieval on policy documents occasionally returns semantically similar but factually wrong sections. A cross encoder reranker over the top twenty dense hits would improve precision at the top three.

A calibrated fraud combiner. The fixed weights carry a lot of design intent, but the empirical performance would benefit from a learned combination trained against confirmed fraud outcomes. Presenting both the fixed weight interpretation and the learned score to the adjuster, rather than replacing one with the other, keeps interpretability.

A first class model card for every deployed model. YOLO11m, YOLOv8 ONNX, Claude 3.5 Sonnet, Claude Vision, and Gemini embeddings all sit in the critical path. Each should have a machine readable card recording its version, its calibration statistics on a fixed evaluation set, and its known failure modes. Without that, model updates propagate silently.

Closing

The interesting property of a production automation system is not that it works on the average case. It is that it degrades legibly on the cases it does not work on. Every component in Claims Agent v2 exposes a decomposable rationale: the composite fraud score decomposes into four signals, each of which decomposes further. The IoU association decomposes into per pair overlaps and explicit implicit rules. The approval decision decomposes into eligibility rules and threshold checks.

Adjusters trust the system because they can inspect it. That property is what makes the automation viable at scale, and it is the design constraint that shaped almost every technical decision above.

Kwabena Obeng

Comments (0)

Leave a comment