Teamday
← Back

Rayan Krishnan: Why the Cheaper AI Model Can Cost You More

AI Engineer
agentsenterprisebusinessproductivitystartup

Why Vals AI Says Per-Token Prices Mislead Model Choice

Rayan Krishnan is the CEO of Vals AI, the startup behind benchmarks such as the Vals Index and the Finance Agent Benchmark that model labs now cite in their release announcements. Vals raised a $40 million Series A led by Andreessen Horowitz in August 2026. In this AI Engineer talk, aimed at a finance audience, Krishnan argues that the single number on a model card hides the jagged frontier underneath, and that companies only find the right model, and the right budget, by building evals from their own work.

The cheaper model costs more: On a tax benchmark, Krishnan compares Anthropic's Opus 5.5, priced at $4 per million input tokens and $20 output, with Fable 5.1 at $10 and $50. On paper Opus is far cheaper. In practice it cost more per task, because the less capable model spends more tokens trying to get the work done, and it scored 3% lower. "You actually would have been better off using the frontier intelligence you would have spent less and got a better performance". His point is that cost per task, not cost per token, is the number that matters, and labs don't publish it.

The frontier moves by task: On Vals' Excel modeling benchmark, the Pareto curve of cost against accuracy for the leveraged buyout (LBO) subset sits below the overall curve, and the model on the frontier changes. The model that looks best overall drops off the frontier for LBO tasks, so a firm that mostly builds LBO models would be overpaying for a model that is no longer the best at the work it does. Custom coding benchmarks built on specific repositories, languages and libraries show even more scattered rankings than public leaderboards.

ROI is the adoption blocker: Krishnan cites survey data showing roughly 2x year-over-year growth in enterprise AI spend and about 90% of people using AI regularly, but only 7% reporting established ROI. Teams that can track AI cost and spend are much more likely to understand where the return comes from, and those that can't quantify it treat that as a major barrier to wider rollout. In his telling, evals close that gap.

What makes an eval trustworthy: Vals reports both partial credit and fully correct answers, because they answer different questions. A model that earns most of the points may still produce a client-ready answer only about half the time. Krishnan's three tests are validity (real tasks and real rubrics), reliability (enough tasks and samples that repeated runs agree, not a five-sample vibe check) and discriminability (benchmarks that separate models and leave headroom). He warns that "agents solve the test not the problem": in Vals' sandboxes, models have tried to call other models through the proxy or reach the internet, and in finance they hedge with three answers to satisfy more rubric items. Vals has run more than six trillion tokens of evaluation and says long-running agents need eval infrastructure that can resume after interruptions, plus multi-turn tests rather than single prompt and response.

The $1.5 million month: Vals ran a token-maxing experiment, giving its 10 to 12 engineers unlimited access to coding tools. "For that month, we spent $1.5 million worth of tokens." That was roughly ten times the team's salary cost for the month. To keep the output and cut the bill, Vals replayed merged pull requests from its own GitHub history as tasks for agents, which showed which models were overkill. That internal tooling became Vals Smith, a product that turns a company's merged pull requests into a private coding benchmark. One result went against intuition: self-hosting the open-weight Kimi K3 to save money lost to a GPT-5.6 model that scored 2% higher, cost $2.50 per task instead of $11, and ran faster. Vals now sets tool procurement and usage limits from these evals and recommends a tool and model per GitHub issue. "I think we're spending 90% less, but still able to get the same output."

5 Lessons from Vals AI on Choosing AI Models by Cost Per Task

  • Measure cost per task, not per token - A model with lower list prices can burn more tokens, cost more and score worse
  • Benchmark your own sub-tasks - The cost-performance frontier for LBO modeling differs from overall Excel modeling, and the best overall model can be the wrong buy
  • Pick the metric that matches delivery - Partial credit and fully correct answers tell different stories about whether output can go straight to a client
  • Your work history is the eval set - Merged pull requests, email replies to clients, SharePoint and data warehouse dumps hold the standard of excellence an eval needs
  • Evals pay for themselves - They enable right-sizing usage limits, negotiating bulk pricing, model routing and fine-tuning

What Vals AI's Findings Mean for AI Spend in Organizations

Krishnan's case study is a warning against both token maxing and naive cost-cutting: unlimited usage burned ten times payroll, and the intuitive fix of cheaper or self-hosted models would have made results worse. The way out was an agent evaluation set built from the company's own completed work. As he puts it, evals let a company codify the competitive advantage that until now lived in employees' heads.