Tokens: The Billing Unit
LLMs don't bill by characters. The price gap between languages comes from how text is tokenized.
The model does not see characters; it sees tokens. Everything you write is first sliced into fragments by a fixed table, then converted into numbers.
Symptoms you'll recognize:
- The cost estimate doubles after launch because you priced by characters.
- You ask it to "count how many characters are in this sentence" and it cannot.
- The same document has a mysteriously different token count in Chinese than in English.
How the slicing works
The split table is not written by people; it is derived statistically from a corpus. The algorithm repeatedly merges the character pairs that appear together most often, round after round, until it reaches a vocabulary of tens of thousands to hundreds of thousands of entries. This is byte pair encoding, the standard statistical tokenizer. bpe-wiki The more common something is, the more likely it became one token; the rarer it is, the more it gets chopped up.
That produces counterintuitive results. A common English word usually costs one token. A long technical term splits into two or three. A Chinese character can be one token, or, if rare, split into two or three by bytes. An emoji typically costs two or more. Spaces and newlines count, so heavily indented code costs more than you would think.
One more fact: the vocabulary is fixed at training time. Your product's coined words, internal abbreviations, and code names have no entry, so they get sliced into fragments. That is both more expensive and harder for the model to treat as a single concept.
Where the bill comes from
Every provider bills by token, input at one price and output at another, usually several times higher. So the formula is not "character count times unit price"; it is convert characters to tokens first.
Measure, do not guess. Take one real request and count it with the target model's tokenizer. The Hugging Face tokenizers library is the easy open-source way to experiment. hf-tokenizers Mixed Chinese-English, tables, and code regularly make estimates off by two times or more.
Bill the whole context, not the current sentence. In multi-turn conversations, every round resends everything before it. Round ten's sentence is short, but that call's input may be tens of thousands of tokens. The last sentence is never the expensive part.
Output length is your most valuable dial. Output is priced higher and emitted one token at a time, so it directly drives latency too. "Answer in three sentences" saves money and time at once.
The Chinese-English price gap is not as large as legend says, but it is real. Newer vocabularies compress Chinese much better; still, rare characters, proper nouns, and vertical tables hit Chinese harder.
The side effects
Tokens are not just a billing unit; they are the model's unit of perception. A lot of inexplicable failures trace back here.
It cannot count words. It sees tokens, not characters. Ask for "exactly 100 characters" and it can only estimate, and estimates are always off. To control length precisely, your code truncates, or it calls a tool to count.
Its arithmetic is shaky. A long number splits into several tokens, and the relationship between digits is not a strong statistical pattern. For computation, the right move is to have it write the expression and hand it to a calculator, not do mental math.
Character-level tasks are hard. Counting occurrences of a letter, spelling a word backwards, checking palindromes — these require seeing characters, and it only sees tokens.
Context length is measured in tokens too. "200K context" means 200,000 tokens, not 200,000 characters. You can fit more Chinese text than the number suggests, and less code than you would estimate.