How to Read Params and Benchmarks

Parameter count, context length, and benchmark scores are three different things. Don't mix them.

Parameter count, context length, and benchmark score are three unrelated things. A model that scores high on all three may not suit you, and one that's mediocre on all three may be just right.

Signs you'll recognize:

• You switched to a higher-ranked model and your scenario got worse.
• An engineer says a model is "bigger" and you don't know what that means.
• You have a 1M-token window, you stuffed the whole knowledge base in, and the answers got vaguer.

Three different numbers

Parameter count is the number of weights in the model — 7B means seven billion. It sets how much statistical regularity the model can hold, i.e., the capability ceiling. But the ceiling isn't the performance: two models with the same 70B can differ hugely depending on training-data quality and post-training methods. And most mainstream models no longer publish their parameter count — you usually just get a size tier. The "bigger is stronger" rule comes from scaling-law research; it's a statistical trend, not a guarantee for any single model. scaling-laws

Context length is how many tokens one request can hold. It has nothing to do with intelligence — only with whether the input fits in one read. And fitting it in doesn't mean using it well: in long contexts, the middle section gets ignored with clearly rising probability. For product people this is blunt: a big window gives you headroom, not a license to stuff everything in.

Benchmark score is accuracy on several standard question sets. It answers one question: "how well does this model do on these tests?" That's all.

The right column is what you actually care about, and none of the left-column numbers can answer it. That's why you must test it yourself.

Why benchmarks mislead

• Test sets leak into training data. Public benchmarks sit on the web; new models can't fully exclude them. Part of a high score is having seen the questions. You can't falsify this from the outside — just hold some skepticism toward high scores.
• Question types don't match your scenario. Most leaderboards test knowledge Q&A and standard reasoning; your product might be "extract structured fields from a pile of support tickets." The strength order of these two can be exactly reversed.
• A few points of difference are noise. The gap between 88.4 and 87.9 is smaller than the variance from changing a prompt. Only a whole-tier gap is worth acting on.
• Leaderboards don't test what you care about most. Output-format stability, obeying prohibitions, surviving long conversations, latency under concurrency — none of these have a leaderboard, but they're exactly what breaks in production.

Leaderboards aren't unusable — a human-preference one like LMArena is good for reading trends, just not as a substitute for your own measurement. lmarena

How to choose

Making selection actionable takes four steps.

One, build your own question set. Pick 20 to 50 cases from real scenarios, covering typical cases, edge cases, and the ones you're most afraid of failing. Write down what counts as a pass for each. This set is your only selection basis, and the regression test for every future model change.

Two, test whether it passes before testing which is better. Many tasks have a clear passing line. Run the cheap tier first; if it passes, a stronger model is just wasted money. Using the smallest model that works is a cost lever bigger than any prompt optimization.

Three, select per task type. A product usually has several kinds of calls: classification, extraction, rewriting, complex reasoning. The first three go on cheap, fast tiers; only the last one justifies the expensive tier. Split the traffic and the bill drops by a large margin.

Four, design replaceability in. Models turn over every half year. Wrap the calls in one layer so a model change touches one place. Selection isn't a one-time deal; it's something to redo every few months.