Teamday
← Back

Noyan Tokgozoglu: How Optiver Ships New AI Models Same Day

AI Engineer
agentsenterpriseinferencebusinessproductivity

How Optiver Runs an AI Agent Platform with P&L on the Line

Noyan Tokgozoglu is head of AI engineering at Optiver, the global market maker, and previously worked at Apple. In this AI Engineer talk he explains how a trading firm that makes "10 million plus trades a day", prices more than a million instruments and processes petabytes of data daily built its own AI platform. The difference from a typical tech company is the feedback loop: Optiver zeroes out its balance sheet at the start of each day, so the P&L impact of a change shows up within hours, and a missed opportunity is expensive.

Same-day frontier models: Optiver's rule is that "if a new frontier model is made available publicly, we want to be able to leverage in the production flows the same day". That takes more than an API key. The team partners with key trading teams to collect golden-set benchmarks, runs every new model on them immediately, adds the model to its inference gateway with cost management, updates the agent harnesses, and publishes a recommendation on how to use it. Support rotations across regions keep it running around the clock, because downtime means the firm isn't making money.

One gateway, many models, on-prem by default: As a regulated firm that guards its IP, Optiver runs everything on its own GPUs and infrastructure. The only traffic that leaves is third-party model inference, under zero data retention agreements. An inference gateway sits in front of third-party, self-hosted and open-source models, handles routing and failover between providers, and tracks spend and budgets per model. Every interaction is logged and fed automatically into the evaluation system. On top sits a "meta harness" layer where teams define portable harness configurations wired to their own knowledge base, and an agent gateway that acts as a single ledger of every agent in the company, with identity management for humans and agents alike.

Context bloat was the biggest cost leak: AI spend jumped with the Claude Code moment in December and again with the summer's frontier releases. Then the team found that on a fresh session, "sometimes people have like 200k tokens worth of context, like by default minimum, sometimes even more." Self-service had let everyone dump MCPs and knowledge into context. The fixes: MCP tool search, CLIs that wrap MCPs more efficiently, tuning when prompt caches get invalidated, moving material from huge system prompts into knowledge bases, and per-team harness reviews. Teams also got visibility into their own spend and could set their own alerts and limits; Tokgozoglu found that people given that agency "actually do a pretty good job." Open-source models where they are good enough, and routing that lets a frontier model plan while smaller models implement, cut costs further.

Build what power users are already hacking together: Traders were tmux-ing into dev boxes and building hacky web UIs around different harnesses. Optiver built one web portal for sessions, chat, artifacts and collaborative documents, plus a Slack client, and is now adding multiplayer rooms where people's agents and deployed agents work together. Traders send PRs to the portal themselves. In his words: "our users are very high agency, they will go and build something if we don't build it for them. So we just build a better version of it."

Small agents over one bloated agent: When a team's successful agent gets adopted by a second team, the tempting move is to merge both teams' functionality into one agent. Optiver found that works poorly. Agents are versioned, and the current guidance is to scope each agent to a task it is good at and measurable on, then let agents call each other through the agent gateway.

5 Lessons from Optiver on Scaling AI Agents in a Trading Firm

  • Benchmark on your own data - External benchmarks don't match Optiver's data distribution, so golden sets from trading teams decide whether a new model is worth using.
  • Stay model- and harness-agnostic - An inference gateway and a meta harness let Optiver swap providers, survive outages and avoid moving at a vendor's roadmap speed.
  • Self-service needs sane defaults - Unchecked, sessions started with 200k tokens of context; tool search, CLIs and knowledge bases fixed it.
  • Show teams their spend - Visibility, alerts and self-set limits brought costs under control without central policing.
  • Forward deploy engineers to the users - Optiver's AI engineers sit with traders, learn how they work and bring what they learn back into the platform.

What Optiver's AI Platform Means for Enterprises Deploying Agents

Optiver's lesson is that the model is the easy part. Using a new model the same day depends on evals, a gateway, harness updates, cost controls and support already being in place. The open problem Tokgozoglu names is evaluating agents whose tools depend on the state of the world at the time, like Slack, which can't easily be replayed. Companies that want agents doing real work need the same plumbing, whether they build it themselves or rent it.