Teamday
← Back

imec's Steven Latré: When Self-Hosting AI Beats the API

AI Engineer
inferenceenterprisebusinessagentsresearch

When Running Your Own AI Models Beats Paying for APIs

Steven Latré leads the AI department at imec, the Belgian non-profit semiconductor research center that has worked on transistor technology for more than four decades. imec uses agentic AI itself, and it sees the same pattern inside the company and among its customers. Teams start with closed APIs, move on to cost optimization and model routing, and then, as token use keeps growing, consider hosting open-weight models on leased or purchased hardware. imec is now in that third phase. Latré's AI Engineer talk presents an imec benchmarking study of thousands of runs that measures what the move from API to infrastructure is worth for agentic work.

The setup: imec focused on long-horizon tasks, where owning infrastructure makes the most sense, and used SWE-Bench Pro coding tasks. It tested three coding harnesses, including Codex and Claude Code, against an API baseline and a range of open-weight models, each mapped to hardware that could serve several concurrent users. It measured resolve rate, latency and cost.

Accuracy is not linear in cost: Smaller models, up to a few hundred billion parameters such as Qwen and DeepSeek, resolved about 35% to 40% of tasks. GLM performed at about Opus level. Kimi K3 resolved 86%, though Latré noted some of the tasks may have been in its training data. The hardware view matters more than the accuracy figures. Qwen and DeepSeek performed about the same, but one needed GPUs costing roughly four times as much to rent. Moving from GLM to Kimi K3 added about 20% accuracy for only about 15% more GPU rent. Small changes in model choice can push you into a different hardware class.

Concurrency is the hidden constraint: "Concurrency is actually probably even more important than accuracy in this sense." Memory has to hold both the model and every user's KV cache. Even on large hardware, "running a Kimi K3 model only gives you about four concurrent users that can really run reliably on this." Latency was still about three times slower than the Claude Code API baseline.

Where self-hosting pays off: With small models on low-end hardware, a GPU has to be busy about 89% of the time to beat the API, which Latré calls infeasible. On large B200-class systems, about 15% utilization is enough. GLM-class models are roughly where infrastructure starts to win. For coding assistance, imec's total cost of ownership analysis found about 25% savings.

Harnesses change the GPU bill: All three harnesses reached similar accuracy, so the choice looks unimportant if you pay per API call. On your own GPUs, wall time is the cost. With GLM, Codex generated about half as many tokens as the other harnesses, which halved GPU time. In Latré's words, the right harness-to-model choice "might not have an impact on application performance, but it has really an impact on your total cost of ownership".

5 Takeaways from imec's API vs Self-Hosting Study

  • Self-hosting is a staged decision - Closed APIs, then cost optimization and routing, then hosting open weights on leased or purchased hardware, driven by token volume.
  • Long-horizon agent work is the first candidate - Batch-like, long-running tasks make the best use of owned GPUs. Interactive use often doesn't.
  • Pick models by hardware step, not leaderboard rank - Similar accuracy can cost four times as much in GPU rent, and a big accuracy gain can cost only a little more.
  • Plan for concurrency and KV cache - Even top open-weight models may serve only a handful of reliable concurrent users per system.
  • Benchmark the harness too - On owned infrastructure, a harness that uses half the tokens halves the bill at the same accuracy.

What imec's Numbers Mean for Enterprise AI Costs

Latré's study replaces the self-hosting debate with measurements: savings are real (about 25% for coding assistance at GLM class), but only at sufficient utilization, with the right hardware class and a token-efficient agent harness. The variables, including model, harness, hardware, quantization and techniques like speculative decoding, keep changing, and a real enterprise mix includes many workloads. imec plans to publish its benchmark data openly and is starting an alliance connecting software teams with chip and infrastructure providers. For how a regulated firm runs its own GPUs today, see Optiver's AI platform.