My grad-agent is a small Python tool (on PyPI, source on GitHub) that watches new arXiv papers, resolves the authors behind them, and drafts cold emails to professors whose work matches mine. It is exactly the kind of system that can go wrong quietly: email the wrong "Wei Zhang," cite a paper someone did not write, or contact the same person twice under two spellings of their name. This post is about the design decisions that make those failures either structurally impossible or, when they cannot be prevented, visible and contestable.

The theme is simple: the most informative output of an autonomous pipeline is often what it refused to do. So the agent keeps a permanent record of every lead it skipped, and why.

Refusal as a first-class outcome

Identity resolution is the highest-risk step. Two researchers can share a name; Semantic Scholar can merge or split author records; a name-string search is a guess dressed up as a lookup. The agent resolves identity in three tiers, in order of decreasing structural guarantee:

Paper-anchored. The trigger is always a specific arXiv paper, so the agent fetches that paper's canonical author roster from Semantic Scholar and matches the senior author against it. Identity is anchored to the paper's metadata, not a name search — a "Wei Zhang" collision is structurally impossible at this tier because we never search by name at all. The output is a stable authorId, not a string.
Name-validated. If paper anchoring fails, the agent falls back to name search, but accepts a candidate only if that candidate's own publication list provably contains the trigger paper.
Refusal. If both tiers fail, or two candidates remain comparable, the agent returns a structured refusal — for example "identity ambiguous: 2 comparable candidates, h=40 vs h=38" — and drafts nothing.

The important part is what happens to that refusal. It is not a log line that scrolls away. It is persisted.

The skip audit sheet

Every skipped lead is appended to a dedicated sheet in the outreach workbook with a fixed schema: date, source, research area, professor name, the stage at which it was dropped (identity, faculty-gate, region, dedup, no-papers), the exact reason text, the trigger paper's title and arXiv id, and which resolution tier was attempted. There is an MCP tool to page through it.

This is the contestability layer. If I ever wonder "why did the agent never surface Professor X?", I do not have to rerun anything or trust my memory of the filters — I can look up the row, see that she was skipped at the identity stage because two comparable candidates tied, and decide the refusal was wrong. The audit trail turns a silent omission into a reviewable decision. In fairness terms: false negatives are usually invisible, and invisible errors are the ones that never get fixed. The skip sheet makes the false-negative surface inspectable.

Freshness warnings: flag, don't fix

Bibliographic databases go stale. A professor moves labs; Semantic Scholar still lists the old affiliation. The agent cross-checks the stored affiliation against the professor's live homepage at draft time, and checks whether their latest paper is recent. Crucially, a mismatch never blocks the draft — it attaches a warning: "S2 says X but the homepage does not mention it; they may have moved labs; verify before sending."

This is the same calibration stance I used in my RAG work: when the system detects a possible error it cannot resolve, it should surface the uncertainty to the human rather than silently pick an answer. Auto-correcting would just replace one unverified claim with another. There is also a cache-invalidation command for when I confirm a move, and a client-side throttle with exponential backoff to prevent verification from hammering the upstream API.

No autonomous sends, verified claims

The pipeline's only outbound verb emails drafts to me. Nothing in the codebase can email a professor. Marking a draft as sent is a manual step I take after review, and that record then feeds deduplication — keyed on authorId, so two people who share a name can never be conflated in the contact log, and the same person under two name spellings can never be contacted twice.

Before a draft even reaches me, a second model pass fact-checks every non-personal claim in the email against the paper's abstract; unsupported claims trigger a rewrite. The human approval gate is the last line of defense, not the only one.

What this adds up to

None of these mechanisms is exotic. Together they implement a stance: prevent the errors you can make structurally impossible (paper-anchored identity, id-based dedup), refuse when you cannot (tier three), record every refusal so it can be contested (the skip sheet), and surface uncertainty instead of resolving it silently (freshness warnings). That is most of what "trustworthy" means for a small autonomous system — not a model property, but an accounting discipline.

The agent is open source and installable via pipx install grad-agentThe skip sheet ships by default.

Kwabena Obeng

Comments (0)

Leave a comment