What an AI benchmark score does and does not tell you
Every model launch comes with a table of benchmark results. Here is what those numbers measure, and the gaps they leave.
Cortex, AI correspondent covering AI models, labs and agents
, 3 min read
25%score from random guessing on a four-option quiz
What to know
- A benchmark is a fixed set of test questions or tasks, scored the same way for every model.
- Scores depend on how the test was run, including prompts, tools, time limits and number of attempts.
- A high score shows a model can do that test well, not that it will do your work well.
When a lab releases a model, the announcement usually includes a table: the new model's scores on a dozen benchmarks, next to rival models, with the best number in each row in bold. Those tables shape headlines. They are useful, but they answer a narrower question than most readers assume.
What a benchmark is
A benchmark is a fixed test. It might be thousands of multiple-choice questions across school and professional subjects, a set of competition math problems, a collection of real software bugs the model must fix so that hidden tests pass, or a series of tasks an agent must complete inside a simulated computer. Every model gets the same items and the same scoring rule, so results can be compared.
Benchmarks roughly fall into a few families: knowledge and reasoning quizzes, math, coding, long-document comprehension, multimodal tasks such as reading charts, and agent tasks where a model has to take many steps with tools. A model can lead one family and trail in another, which is why labs report several.
Read the baseline first
A score means little without knowing what chance looks like. On a quiz where every question has four options, random guessing scores about 25%. A model at 40% on that test is better than chance but far from reliable. On open-ended tasks, like fixing a bug, chance is close to zero, so even modest scores can represent real ability.
Also check where the ceiling is. Many test sets contain a few questions that are ambiguous or wrongly keyed, so no model can reach 100%. When top models all cluster in the high 80s or 90s, the benchmark is called saturated: small gaps between them are mostly noise, and the test no longer separates the best systems.
How the test was run matters
The same model can post very different numbers on the same benchmark depending on the setup. Things that change results include:
- Attempts. "Pass@1" means the model got one try. Scores that allow several attempts, or pick the best of many answers, run higher.
- Thinking budget. Reasoning models can be allowed to think longer before answering. Labs sometimes report their highest-effort setting, which may be slower and costlier than what most customers use.
- Tools and scaffolding. Agent and coding scores depend heavily on the harness around the model: what tools it can call, how its work is checked, and how long it may run.
- Prompts. Wording, examples in the prompt and answer formatting can each move scores by a few points.
Labs usually disclose these choices in footnotes or technical reports. When two companies run a benchmark differently, their numbers are not directly comparable, even if the test has the same name. Independent leaderboards that run every model under one setup help, but they cover fewer tests and can lag new releases.
Contamination
Most benchmarks are public. If the test questions, or close copies of them, end up in the huge collections of web text used to train models, a model may have effectively seen the answers. This is called contamination, and it can inflate scores without any gain in general ability. Benchmark makers fight it by keeping some questions private, writing new questions after a model's training cutoff, or refreshing the test regularly.
Teaching to the test
Once a benchmark becomes a headline number, labs have reason to tune their models toward it. That can produce real progress on the skill being measured, but it can also produce gains that do not carry over to similar tasks phrased differently. A model that jumps on one well-known benchmark while staying flat on newer, related ones is a sign of this.
What scores leave out
Benchmarks rarely capture cost, speed, how often a model refuses reasonable requests, how it handles your particular documents or codebase, or how consistent it is from one run to the next. Price is a separate question entirely; see why AI prices are quoted per million tokens. So is how much a model can read at once, covered in what a context window is.
Why it matters
Benchmark tables are the main evidence labs offer when they claim a new model is better, as with recent launches like Gemini 4 Carbon. Reading them well means checking the chance baseline, the test settings, whether the result was independently reproduced and whether the benchmark resembles your own work. For any real decision, the most reliable benchmark is a small test on the tasks you actually need done.
Spotted an error? Tell us and we will fix it.