Teamday
← Back

Fieldguide: 4 Controls So Audit Agents Can Show Their Work

AI Engineer
agentsenterprisesafetyknowledge-workbusiness

Why Fieldguide Builds Audit Agents That Can Defend Every Conclusion

Anirudh Veeraragavan is lead security engineer at Fieldguide, which builds AI agents for audit and advisory firms. His AI Engineer talk starts from a fact about his industry: "in an audit, the conclusion is the product." A conclusion is only worth the process behind it, so the question for an agent is not whether it got the right answer but whether anyone can reconstruct how it did. He works through one procedure, a bank reconciliation, where an agent compares a client's book balance to the bank balance and explains the difference through deposits in transit and unpaid checks.

The inspection problem: The obvious build works. Give the agent the books, the reconciling items, the evidence and the firm's methodology, add prior-year examples, benchmark it against reviewed audits and route the result to a human for sign-off. Users are happy, metrics look good. Then the audit gets selected for inspection and the inspector asks why that was enough. "I don't care what your ratings say. I don't care what your metrics say. I don't even care what your users say." Veeraragavan's reminder that the stakes are real is Peregrine Financial Group, the Iowa brokerage that claimed over $200 million in a bank account that held about $5 million, for 20 years.

Control 1, authorization against object state: In his example, a reviewer flags a check as suspicious and signs the line. Days later new evidence arrives, the agent is triggered and flips the status to "no exception." The agent had permission, had evidence and reached the right answer, yet the reviewer's signature now sits on content they never saw. His rule: authorization is not whether a tool call is allowed, but whether it is allowed on this object in its current state. An agent may draft unreviewed lines in its own name but cannot overwrite human-reviewed work. "It must exist as its own proposition, not as an ultimate truth."

Control 2, a recorded run you can replay step by step: Run the reconciliation in 2027, get no exceptions, rerun it in 2028 for an inspection and get a different result. Logs of inputs and outputs are not enough to explain why. Fieldguide records each run as a chain of steps, each with an ID, inputs and outputs, and a clear marker of whether the step was deterministic code (fetch an item, compare two numbers) or a model call (extract a field from a PDF, reach a conclusion). Individual steps can be rerun, so two runs can be compared and the exact step where they diverge identified. The point is to prove the original answer was not luck.

Control 3, evidence provenance: The agent cites "January bank statement, page 2." With hundreds of similarly named documents, that string is useless: which version, who uploaded it, where did it come from. "a link is an identifier, not just a line of text." A citation should carry a document ID, a version, a content hash and the source it was obtained from, which then allows rules about admissible evidence, for example only statements pulled directly from the bank, not files the client uploaded and could have altered.

Control 4, eval integrity: The benchmark set was built from verified prior-year audits, but verified does not mean correct. Reviewed items get superseded, decisions rest on outdated data, methodologies change. In his example an agent scores 97%, yet four of the nine misses were wrong labels in the eval set and twelve "passes" were actually wrong under current business rules. The headline number hides whether the agent is better or worse than it looks.

When the agent must stop: The escalation triggers follow from the controls: the agent wants to change reviewed work, a rerun contradicts a prior run, evidence breaks the admissibility rules. Because automation bias is real and people start clicking yes to everything, "these escalations should be easy to verify, but not necessarily easy to approve." He expects the agent to handle about 80% of items and wants the human effort concentrated, with full context, on the 20% that matter.

5 Takeaways From Anirudh Veeraragavan on Defensible Finance Agents

  • Borrow the auditor's standard - An experienced auditor new to the engagement must be able to trace the agent's test, evidence and decision.
  • Authorization is per object and per state - Agents draft, humans sign, and a signed line is immutable to the agent.
  • Record steps, not just outputs - Tag each step as code or model, give it an ID, and make it individually replayable.
  • Citations are identifiers - Document ID, version, hash and origin, so you can enforce which evidence is admissible.
  • Audit the eval set itself - A 97% score built on stale labels tells you nothing about the agent.

What Auditable Agents Mean Beyond Audit

Veeraragavan says the same four questions apply wherever an agent makes a consequential financial call: approving or rejecting a loan, closing or escalating an anti-money-laundering alert. His closing line is that a defense of "the model said so" no longer works. The talk pairs with Kepler's argument that code should write every number, Two Sigma's five controls for agents and Speakeasy's point that a valid key is not valid intent. For teams putting human-in-the-loop review on agents, the design target is clear: fewer, better-evidenced escalations and an eval set you trust as much as the agent.