Jonathan Osborne
Lead AI Engineer at NatWest Group
About Jonathan Osborne
Jonathan Osborne is a lead AI engineer at NatWest Group, the UK's second-largest retail bank. He works on taking AI agents in Cora, NatWest's digital assistant, from prototype to production for about 19 million retail customers.
In an AI Engineer talk, Osborne explains what NatWest had to build to get approval to put customer-facing agents live in a regulated bank.
Career Highlights
- Lead AI engineer at NatWest Group
- Works on moving about a third of Cora's journeys from scripted answers to AI-generated answers and agents
- Speaker at AI Engineer (2026) on proving agent safety in banking
Notable Positions
On Shrinking the Attack Surface
Before any generative model runs, NatWest routes conversations through context filters, deterministic rules, a classical intent classifier and policy rules, so every decision is auditable and the system still answers when a model is down.
On LLM Judges
On 14,000 complaint cases scored by humans and an LLM judge, the judge's faithfulness scores barely correlated with the humans'. Osborne treats the judge model as configuration and uses a different model family from the agent. See Agent Evaluation.
On Shadow Mode
Running the new system in shadow mode, classifying real traffic while the scripted bot still answers, gives model risk teams production evidence without customer risk.
Key Quotes
- "With 19 million customers, if we get a few of these utterances wrong, that isn't a bug ticket." (on blast radius)
- "A router that's 94% accurate but 100% confident that 6% of the time it's wrong is way more dangerous than one that's 85% confident and knows it." (on confidence)
Related Reading
- Agent Evaluation - Testing agents for production work
- Agent Observability - Tracing and monitoring agents in production