Milind Sunil Shah
Member of Technical Staff at Brainbase Labs
About Milind Sunil Shah
Milind Sunil Shah is a member of technical staff at Brainbase Labs, which helps enterprises deploy and manage internal AI agents through its Kafka Workforce platform.
In an AI Engineer talk, he walks through Brainbase's work with Domino's: an agent that collects competitor pizza prices from websites and delivery apps into a spreadsheet, a job an analyst used to spend two hours a day on. Version 1 was 41% accurate; version 11 reached 97% at about half the cost per source, without changing the model.
Notable Positions
On Ontology
Every term the agent needs should have a written definition: the checkout price rather than the struck-out price, delivery versus takeaway, what counts as a large pizza. Anything not written down, the agent guesses.
On Evals
Brainbase scores agent work with deterministic code checks, an LLM judge on traces, and an "agent as a judge" that re-checks flagged outputs over multiple tool calls. See Agent Evaluation.
On Iteration
Ship a narrow first agent that can be scored from day one, change one thing at a time, turn each failure into a benchmark case, and fix instructions before changing the model.
Key Quotes
- "Version one's job is to be measured. It's not to be good."
- "You're trying to hand off an entire loop and not just the agent."
Related Reading
- Agent Evaluation - Scoring agents before and after release
- Human-in-the-Loop - Analysts reviewing only exceptions