A New Model Ships Every Few Weeks. How Do You Know If Switching Is Worth It?

How we test every model before it powers Billy, and what the latest round taught us about accuracy, speed, and cost per correct answer.

Every few weeks, a new model lands with a launch post, a benchmark chart, and an implied promise: switch and everything gets better.

Benchmarks measure someone else's task. The question that matters for any team running AI in production is narrower: is this model better for our workload, on our prompt, with our tools, at a cost that makes sense?

Here's how we answer that question at Finout, and what the latest round of testing showed.

The best answer wasn't the most expensive one. Across 122 identical questions, cost per answer ranged from under half a cent to more than 15 cents, and the most accurate model came in at under six cents. That's why we judge every model by a single number: cost per correct answer.

Why model choice is a production decision for Billy

Billy is Finout's FinOps agent. FinOps teams ask it about cost trends, anomalies, waste, budgets, and allocation, and then act on the answers.

Usage has grown fast. Billy prompts grew 75% from Q2 to Q3, and weekly active users grew 40% over the same period. FinOps is a discipline where people expect accuracy to the cent, and a model change touches every one of those answers.

So we treat the model behind Billy the way a FinOps team treats a commitment purchase: it has to be justified with data before it goes live.

How we do it

Testing Billy isn't a one-time event. It runs on three layers.

1. The upgrade exam. Before any model change, Billy answers a fixed set of 122 real-world FinOps questions in six categories: cost analysis, anomalies and waste, budgets and governance, discovery and context, historically hard questions, and edge cases.

• One variable. Same code, same prompt, same tools. Only the model changes.
• Three judges, scored 1–5. Correctness (is it right), task completion (did it do what was asked), and usefulness (would a FinOps lead act on it). The judges are deliberately stricter than the ones we use in production.
• Full traces. Latency split into model time and lookup time, lookups per answer, tokens, and cost per answer.
• A fixed order for the bar. Correctness first, then task completion and usefulness, then speed, then cost. A cheaper or faster model only wins if teams get the same right answer.

2. Continuous evaluation in production. Judges score Billy's real interactions on an ongoing basis, and a weekly capability report shows the team where answers can get better and what users are asking Billy to do next.

3. Monthly question analysis. Every month we map what FinOps teams ask Billy by topic. About 80% of questions fall into the patterns covered in Billy's Cheat Sheet. The rest show us where to build next.

The track record: when an upgrade earns its place

The last model change that passed the exam was the move to Sonnet 5. Alongside improvements to how Billy reasons and selects tools, it delivered:

• Answer quality up almost 2x
• Finout documentation used for 70% of product and how-to questions, up from 53%, so answers are grounded in the official source
• Charts in tag and allocation answers twice as often
• The longest investigations completing about 16% faster

That upgrade shipped. The exam exists to find upgrades like that one, and to make sure the next one is at least as good.

This round: three models, one bar

This month, Sonnet 5 went up against two newer models: Sonnet 5.5 and Haiku 5.5.

Decision: Billy stays on Sonnet 5, with the most accurate answers at $0.059 each. Haiku 5.5 moves to the next round as the cost-per-answer candidate.

The table shows what happened. The useful part for anyone choosing models is why.

What we learned

1. Every model has different defaults, and your prompt was tuned for the last one

Billy's prompt tells it to investigate thoroughly before answering. With Sonnet 5, that produces about 2.5 lookups per answer: enough to ground the answer while keeping it fast.

Sonnet 5.5 runs more lookups by default. With our "investigate thoroughly" instruction on top, it averaged 7.2 lookups per answer, wrote about three times as much text, and took nearly twice as long. It also produced the most charts of the three.

That's a prompt-model fit question, not a model question. A prompt is tuned to how one model behaves. When you switch models, you're running a prompt tuned for a different model's defaults.

Takeaway: treat a model switch as a prompt project, not a config change. Retune for the new model's defaults, then test again.

2. Test the whole system, not just the model

Sonnet 5.5 works in several passes. It writes part of an answer, calls a tool, writes more, and refines as it goes. Billy's harness was built to keep the model's final message as the answer, and with Sonnet 5 the final message is the answer.

With a model that works in passes, the final message can be a short wrap-up line. Most of the answers where Sonnet 5.5 scored lower traced back to exactly this.

So the exam improved Billy for every model: we're updating the harness to keep the complete answer, however many messages it takes. A test meant to evaluate one model ended up improving accuracy across the board.

Takeaway: your eval tests the whole system, including the code around the model. That's where some of the best improvements come from.

3. Small models are closing the gap

Haiku 5.5 scored 4.50 correctness against Sonnet 5's 4.62. That's about 97% of the quality, slightly faster, at roughly 1/14 of the cost per answer.

The remaining gap is specific, which makes it workable. Haiku 5.5 drew charts less often and added caveats less often than Sonnet 5. Both are prompt-level behaviors.

Next: a prompt pass for Haiku 5.5 focused on charts for trends and breakdowns, caveats, and Billy's feedback step. Then a rerun of all 122 questions. If correctness lands within 0.1 of Sonnet 5 and the charts show up, it moves to a 48-hour live test.

Takeaway: set the promotion bar before the run. Then a cheaper model has a clear path to earn its place.

The FinOps lens: price the answer, not the token

This is where being a FinOps company changes how we read the results.

Token price is a list price. Cost per correct answer is the number that matters.

At 10,000 answers, the three models in this round come out to roughly:

• Sonnet 5: $590
• Haiku 5.5: $40
• Sonnet 5.5: $1,550

Same questions, same agent, and a nearly 40x spread between the lowest and highest cost.

Most of that spread comes from behavior, not the price table: how many lookups the model makes, how much it writes, how many passes it takes. Every lookup adds tokens to the next call, so an agent that investigates twice as much costs more than twice as much.

Two more things we track:

• Pricing tiers. Haiku 5.5's list price is $0.10 input and $0.50 output per million tokens for prompts up to 100K tokens, and $0.50 and $2.50 above that. Billy's calls average around 46K input tokens, well inside the lower tier. An agent whose context keeps growing can cross that line without anyone changing a setting.
• Quality is part of the unit. A cheaper answer that needs a follow-up question isn't cheaper. That's why cost and the three quality scores always sit in the same table.

Run this on your own agents

You don't need a research team. You need five things:

1. A fixed question set from real usage. Cover your main use cases plus the questions your agent historically finds hardest. Version it so every run is comparable.
2. One variable per run. Model, prompt, or tools. Never two at once.
3. Judges stricter than production. Score correctness, completion, and usefulness separately. Each one measures something different.
4. Traces, not just scores. Latency split by model and tools, lookups per answer, tokens, and cost per answer. The traces explain the scores.
5. A promotion bar set in advance. For example: "within 0.1 correctness of current, at a lower cost per answer." Then let the data decide.

The teams that get the most out of new models won't be the ones who switch first. They'll be the ones who know within a day whether a switch is worth it.

See what Billy answers today

Billy runs on Sonnet 5 because that's where our tests say it gives FinOps teams the most accurate answers. Most of what teams ask falls into a handful of patterns, and we keep the best prompts for each in Billy's Cheat Sheet.