Teamday
← Back

Brainbase Took a Domino's Agent From 41% to 97% Accurate

AI Engineer
agentsenterpriseautomationknowledge-workbusiness

How Brainbase Hill-Climbed a Back-Office Agent for Domino's

Milind Sunil Shah is a member of technical staff at Brainbase Labs, which helps enterprises deploy and manage internal AI agents (its platform is called Kafka Workforce). His AI Engineer talk is a case study with one customer: Domino's, starting with a single region and the CMO's office. The question was simple: what does a pizza cost at the competitor down the street, on its website and in delivery apps, with and without promos? An analyst spent two hours a day building that spreadsheet by hand. Version 1 of the agent was 41% accurate, took more than two hours and cost over $2 per source (one item, one competitor, one website). Version 11 reached 97% at $1 per source. The model didn't change: "All of this went just through instructions and skills and playbooks changing."

Kickoff and ontology: Brainbase starts by agreeing on what a win looks like: the job today, what a human signs off on, the cost, the accuracy target, who judges the output and what the agent must never do alone. Then it writes down every term the agent needs. "If you don't write something down, the agent's going to guess and probably get something wrong." For Domino's, the rules file says to record the price paid at checkout, not the struck-out list price; defines what counts as a large pizza (the item itself or a bundle); separates delivery from takeaway; applies promotion math before writing a number; and logs anything it can't verify as a gap instead of guessing. Each rule came from a real failure in a trace. The agent writes observations into Brainbase's Postgres-like memory, so every cell in the final spreadsheet points to its source page and the rule behind the price.

Benchmarks from old work: The offline benchmark reruns analysts' past spreadsheets. Archive.org snapshots recreated competitor websites on the right dates, but delivery apps didn't serve data that way, so Brainbase stored browser screenshots of every run from version 1 onward. Scoring uses three layers: code checks (every cell filled, discount math right, prices moving more than 20% from the last run flagged), an LLM judge that reads the trace for the right item and promo, and an "agent as a judge" that revisits each flagged price over multiple tool calls to confirm the flag.

Ship narrow, then iterate: "Version one's job is to be measured. It's not to be good." Scope grew from one app, one item and one city to burgers and takeaway, another app, then other websites. Shah's iteration rules: fix the most expensive and frequent failures first, change one thing at a time, turn each failure into a benchmark case, and fix instructions before switching models. One fix was a single rule: the agent stopped at the menu section it was linked to and missed a cheaper deal lower on the page. Another fixed browser behavior. On a promotions page where prices sat inside 26 images, the agent kept zooming in to double-check after finding the right card at minute 13, until an engineer stopped it at minute 15. Telling it to read the HTML DOM, image titles and alt text first, and to stop past a set zoom level, cut image reads to a third and cost by more than half.

5 Lessons from Brainbase's Domino's Agent

  • Agree on the win before building - Current time, cost, required accuracy, who judges and what the agent may never do alone, written down with the customer.
  • Write an ontology - Every term the analyst knows implicitly (checkout price, delivery versus takeaway, what counts as a large pizza) becomes a rule, usually after a failure.
  • Use three kinds of judges - Deterministic code where possible, an LLM judge on traces, and an agent judge that re-checks flagged outputs.
  • Expensive failures can be cheap fixes - A rule about reading the whole page and a rule about the DOM did more than a model change would have.
  • Hand off the loop, not the agent - The agent now runs on Domino's schedule or from an email trigger, analysts review only exceptions, and each correction becomes a benchmark case.

What the Domino's Case Means for Teams Deploying AI Agents

Shah's closing point is the one most deployments miss: "You're trying to hand off an entire loop and not just the agent." Accuracy came from writing down what analysts knew, scoring every version and letting the people who review exceptions feed corrections back as test cases. That is agent evaluation as an operating process, with a human in the loop only where the agent is unsure. For a similar loop in wealth management, see how Arca turns staff edits into harness fixes.