No serious business runs on one model

A model card looks like a cockpit. Thirty columns. Each one is a percentage. Almost none of them measure what your workload will do, and the columns that matter most are usually the ones the lab did not publish at all. And the question, in the end, is not which model you pick. It is which models, and how you route between them.

// the model card had thirty-two columnsThe cockpit, and the columns I needed

The model card had thirty-two columns.

I was trying to pick a model for a flow that would answer short, latency-sensitive questions a few thousand times an hour. The columns scrolled past: Terminal-bench, SWE-Bench Pro, MCP Atlas, CharXiv, MRCR v2 at one million tokens. Each column had a percentage, each calculated honestly enough by some methodology I had not read, and each was supposed to tell me whether this was the right model for what I was trying to do.

Not one of them was.

The card had no column for what my workload would cost by turn thirty, when every call carries eighty thousand tokens of accumulated context. It did not have a column for tail latency under real concurrency. It did not have one for how this model's failure modes would look, specifically, on the seven question types my flow actually had to answer. Those were the columns I needed. The ones on the card were the ones the lab had chosen to publish.

I went back and read each column line by line. Half of them turned out to be measuring something close to the same skill in a slightly different costume. Others were saturated, meaning every frontier model now scores within a point or two, and the rankings between them have stopped meaning anything. A few were genuinely informative for the specific shape of work I was doing. None of them, together, told me what I needed to know in order to commit to a model and ship.

That is the problem this article is about. Not whether the numbers on a model card are real. They are. Each one is the answer to a question. The question is whether the questions on the card are the right ones for the job at hand, and what you do when they are not. The answer, by the time you finish reading, will not be about picking a model. It will be about combining them.

// what each axis measuresThe benchmarks, line by line

The first move is to read the benchmarks the same way you would read the column headings on a printout from a piece of test equipment. Each one measures a specific thing under a specific assumption. Knowing what that thing is, and what the assumption excludes, is most of the skill. The breakdown below is meant to read cleanly for two audiences. Engineers are picking a model for production, and non-coders are trying to figure out which of these scores tells them anything useful. If you do not write code, two or three of the cards will matter for your work, and the rest will not. The cards say which is which.

Three of those cards do more work than the rest: HumanEval, SWE-bench Verified, and Terminal-Bench. All three are about coding, and reading them next to each other is the single most useful way to interpret a model card. They are not measuring the same coding. They are measuring three different jobs that the field has decided to call by the same word.

HumanEval is a coding interview question. SWE-bench Verified is an on-call fix in a live codebase. Terminal-Bench is being handed the keys to the machine and told to finish the job.

Frontier models in 2026 score above ninety on the first and somewhere between forty and fifty-five on the second. The same model. That gap is the distance between writing a function from a clean prompt and tracking down a bug across 17 files written by 3 other people 2 years ago. Terminal-Bench sits further out still, in the seventies, under a good harness, where the model has to drive a real machine to the finish, and the agent wrapping it matters as much as the model itself. The benchmarks the labs publish make sure you see the first number. The work you are paying the model to do looks like the second and the third. The SWE-bench and Terminal-Bench tasks are published, with example traces, and are worth reading in full at least once.

The other thing worth knowing about benchmarks is that they degrade. Every benchmark has a shelf life, measured in months. When MMLU was first published, top models scored in the fifties. By 2026, frontier models are expected to score in the high eighties or above. The differences between them at that ceiling are within the margin of error, and the leaderboard rankings have become noise. The field has a term for squeezing every last point out of a saturated benchmark through evaluation choices and training-set tuning. The word is benchmaxxxing. It is honest engineering when used to advance a real capability. It is theatre when used to take a hundredth of a point from a saturated metric and use it for marketing.

The contamination problem is the second half of the same coin. MMLU questions have been on the open web since 2020. Any model trained on a recent crawl has seen most of them. A score that should reflect generalizable knowledge often ends up meaning memorizing the test bank. Inference-time decontamination work has shown that removing leaked items reduces reported MMLU accuracy by 19% and GSM8K accuracy by 23%. The labs know this. The newer benchmarks are private holdouts for exactly this reason. The old ones are still printed on the cards because they make the new model look better than the previous one, which is true, and because no buyer of model APIs ever penalized a vendor for showing a score that was a little too easy.

// the use-case mapTen jobs and the columns that matter for each

The benchmark cards above name what each column measures. The next step is to connect those columns to the work the model is being paid to do.

Different jobs care about different columns. A team writing customer-facing chat and a team building an agentic coding assistant should be reading two completely different subsets of the model card. A model that wins one job can be the wrong tool for the next. The skill is knowing which two or three columns matter for your work, and being honest about which columns are noise for you, no matter how impressive the published number looks.

Below is a working map. Ten common jobs, the benchmarks that matter for each, the ones that mislead, and how production teams end up combining models when a single choice is not enough. The strategy column reflects patterns that recur across published reports on multi-model production, not a single house recommendation.

Reading those rows together, three patterns repeat. Worth naming.

First, most jobs need at most two or three columns from the card. The other twenty-seven columns are noise for your specific work. The skill is knowing which two or three matter for you, and a model that is impressive on the wrong columns for your job is not impressive for you.

Second, a single model is rarely the right answer for everything. Production deployments commonly use one model for high-throughput routine work and a second model for the small slice of queries that need depth. Coding assistants route by complexity. Customer support agents escalate to a frontier model when their confidence falls. Creative tools sometimes use two different models in alternation for variety. Picking the best model is usually picking the wrong question. The better question is which combination of models, routed how.

Third, for several real jobs the column you most need does not exist on any model card: brand voice, legal hedging, narrative arc, multilingual nuance. The benchmark for those jobs is your own evaluation set, run on your own prompts, with your own definition of good. The model card is the starting point, not the destination.

// the columns the card does not haveOff-card axes that decide production

The columns on a model card are the ones the lab chose to publish. The columns that decide whether the model will work for the job you are paying it to do are different ones. Most of them cannot be measured in a leaderboard cell. They only show up when the model encounters real workload.

Cost in an agent loop is not linear

The pricing page tells you the cost per million tokens. It does not actually tell you what a multi-step agent run costs. At every turn of an agent loop, the entire conversation history is sent back to the model. By turn ten, the agent's context window contains 20,000 tokens of accumulated history. By turn thirty, that number is closer to eighty thousand. Each token is billed every time the model is called.

The compounding is a triangular series that grows roughly with the square of the turn count. A workflow that feels like ten small steps can accumulate 80,000 to 200,000 tokens before it ends. Teams that budget per turn instead of per session underestimate the full cost by three to five times. The forty-seven-thousand-dollar agent loop is not a folk tale. It happens when a token budget alert fires after the budget has already been spent. Compaction is the within-session counter-pressure to all of this, which is why the harness that wraps the model is, in 2026, the thing that decides whether the bill on Monday matches the demo on Friday.

The tokenizer and the tail

The price per million tokens is not the price per million words. Different models tokenize the same text differently, and when one major lab shipped its 2026 flagship, the new tokenizer produced up to thirty-five percent more tokens for the same input. The per-token price was unchanged; the per-job cost was not. Latency hides in the same place. Published numbers are medians, and production fails at the tail. The principle, The Tail at Scale, was named back in 2013. For LLM inference, the p99-to-median ratio ranges from 1.8× to 2.7× under normal load, widening further once GPU utilization exceeds 80%. The card shows neither the tokenizer multiplier nor p99, so you measure both yourself, under your own load.

Failure-mode behavior, not score

The score tells you what fraction of test items the model got right. It does not tell you what it does when it gets one wrong. Two models with the same 80% score can behave differently in the remaining 20%. One refuses honestly and asks for help. One hallucinates confidently. One asks a clarifying question and recovers. One commits to the wrong answer and writes three paragraphs justifying it. The choice between those models, on the same score, is sometimes the only choice that matters.

// three tiers, one decisionFrontier, edge, and the layer above

Once you have read what the card publishes and inferred what it does not, the architectural decision is the one the card cannot make for you. Three tiers, three jobs.

The default the field is converging on, after the early reflex of routing everything to the most expensive model, is the 70/30 split.

The numbers are reported consistently across multiple practitioner surveys: roughly seventy to eighty percent of production queries can be served well by a small model close to the user, and the remaining twenty to thirty percent are the ones that justify the frontier spend. Routing the easy three quarters to the cheap tier is what makes the unit economics of an agentic product work. The routing layer, not the model, is what most teams underinvest in.

The 2026 shift, if you want it in one line, is to pick one model and route every request to the cheapest tier that handles it well. Hosted gateways, production-control proxies, and open-source routers exist to make that practical. Picking one of them, in 2026, is closer to picking a database than picking a vendor.

// the most common ways to choose wrongWhat goes wrong in week one

The mistakes are predictable, and most occur the week the model goes into production. A short field guide to the ones I have seen most.

• Paying for a frontier capability you never use. The default for "safety" is the most expensive model in the lineup. The actual workload uses thirty percent of its headroom. The bill is paid in full.
• Ignoring p99 latency. The median looks fine. The tail catches every long-running flow. The user-facing failure is intermittent enough to be hard to reproduce and consistent enough to be embarrassing.
• Forgetting cost compounding in agent loops. Per-turn cost looks reasonable. Turn thirty is sixteen times turn one. The architecture was never updated to compact aggressively, and the invoice catches up.
• Confusing context-window size with usable context. One million tokens advertised. Independent retrieval testing shows that for most frontier models, effective context sits in the 200–400K range, with recall falling off well before the advertised limit. The number on the card is the room's size, not the volume of the room that can be retrieved cleanly.
• Choosing benchmark scores instead of failure-mode behavior. Two models with the same score behave differently when they are wrong. The choice that matters is in the twenty percent the model gets wrong, not the eighty it gets right.
• Tokenizer surprise. Same input text, different token count across model families and across generations within one family. The invoice number does not match the spreadsheet number.
• Lock-in through native features. Tool-use schemas, structured-output modes, and multimodal APIs are not portable. Building deeply on one vendor's native shape is a decision worth making consciously and almost never made consciously.

// the combination, not the modelWhere the next year of work happens

I think about the model card again. Thirty-two columns. Each measures one specific thing. Each of them is honest in its own frame. None of them could answer the question I was trying to ask.

The question I was trying to ask, in the end, was not which model. It was which strategy for using which model where. That is a different question, and the model card does not have a column for it because the column does not fit on a model card. The strategy is the harness that wraps the model and decides how much context to ship. It is the router that decides which tier this request is routed to. It is the eval suite of 100 test cases, drawn from real user prompts, that catches the failure modes a benchmark cannot see. It is the alert that fires before turning thirty, not after.

The first two articles in this series were about the harness. This one is about what sits inside it. Read the three together, and the seam becomes obvious. The intelligence is in the model. The decisions about what to remember, what to compact, what to route, what to ship, and what to budget are all in the regular code around the model. That regular code is where the work that compounds for builders lives.

The model is the commodity. The combination is the product.

If you are picking a model this month, read the card. Then read the columns that it does not have. Build a small evaluation set from your own prompts, run every candidate model against it under realistic concurrency, and measure latency, failure modes, and costs. Treat the leaderboards as priors, not as decisions. And remember, when the next leaderboard reshuffle happens four weeks from now, that the part of your architecture that compounds is the part that does not change between models. The strategy is the part that stays. Read it as carefully as you read the card.