How Model Providers Differ
Skip the vendor rankings. Make a real choice by capability, cost, ecosystem, deployment, and compliance.
This section won't hand you a vendor ranking. Leaderboards turn over every few months; memorizing them is pointless. What stays reusable is a method of measuring.
Signs you'll recognize:
- You read a pile of reviews and still don't know which one to pick.
- You chose the highest-scoring model and it performs worse than the cheaper option.
- You integrated a provider and only then discovered compliance can't pass.
Why no ranking
Three reasons.
First, leaderboards and your task are not the same thing. Public benchmarks test standardized tasks like competition problems and general Q&A; your task is "label these tickets into our five internal categories." A high benchmark score doesn't mean your use case works — the two are often reversed. Human-preference leaderboards like LMArena are good for spotting trends, but they can't replace your own measurement. lmarena
Second, the gap is narrowing and moving fast. Public benchmark differences between top models keep shrinking, while new versions and new prices arrive every few months. Any fixed ranking starts expiring the day it's written. ai-index
Third, selection was never just about capability. Price, stability, compliance, migration cost — any of them failing is a veto, even if the capability is the strongest.
The six selection axes
Put the candidates on this table and score them one by one. It still works next quarter.
| Axis | What to measure | How |
|---|---|---|
| 1. Task performance | Accuracy and stability on your kind of task | 20–50 real samples, same set for every model, human grading |
| 2. Context length | Whether it fits the material you need to load | Test with your longest real input; don't trust the sticker number |
| 3. Real cost | The cost of one complete call, not the per-million-token rate | Measure with real request volume — inputs, outputs, re-sends openrouter-models |
| 4. Stability | Rate limits, timeout rates, behavior under failure, SLA | Load-test and watch P95 latency and error rate, not just averages |
| 5. Compliance | Data residency, retention, training use, signable contract | Have legal review the data-processing terms, not the marketing page |
| 6. Migration cost | How much changes if you switch | Check interface compatibility and how much prompting needs rewriting |
The first two decide whether it works at all; the middle two decide whether you can afford it; the last two decide whether you dare to use it. Any failure is a veto.
Axis 1. is the most time-consuming and the most valuable. Building a sample set, running it, grading it by hand — do it once and you reuse it on every future model swap. It's the highest-return investment in the whole exercise.
The temperaments of each camp
No rankings, but the camps do have stable personality differences that won't shift soon.
Overseas closed-weight leaders (OpenAI, Anthropic, Google): usually the highest capability ceiling, mature engineering for tool calling and long tasks, complete docs and ecosystem. The cost: higher prices, and using them from China raises network and compliance issues.
Domestic platform vendors (Alibaba, ByteDance, Tencent, Baidu): strong in Chinese-language scenarios, compliance qualifications, enterprise contracts and invoices, and integration with their own clouds. They typically offer both first-party models and hosted third-party models.
Domestic value camp (DeepSeek, Moonshot, Zhipu, MiniMax): aggressive pricing, some open weights, cost-performance as the main selling point. This camp keeps pulling down the price floor of Chinese models.
Open-weight models: you can deploy them yourself — data stays in-house, no fear of being de-listed or repriced. The cost: GPUs, operations, and concurrency are all on you. At small-to-medium scale that usually costs more than calling an API; what you buy is control, not cheapness.
A scale reference: as of August 2026, overseas flagship input pricing runs roughly a few US dollars per million tokens, while mainstream domestic models run a few RMB or less, and top products often attach free quotas. These numbers move fast — always check the provider's current pricing page. claude-pricing
Running a selection
Four steps, doable in a day or two.
First, strike the non-compliant. Put data residency, retention, and contractability on the table; anything that fails is out. Doing this first saves all the wasted work after.
Second, build a real sample set. 20 to 50 cases covering common scenarios and a few nasty edge cases. This set is the ruler for every later decision.
Third, run every candidate on the same set. Same prompts, same parameters, human grading. One trap: different vendors respond very differently to the same prompt, so when a result disappoints, give it one chance with adjusted prompting before judging.
Fourth, pick two, not one. A primary and a backup — and the backup must have really run. Rate limits, outages, and de-listings are normal; don't bet product availability on a single vendor.