Multimodality: How Images Get Read
Multimodality is a retrofit. How it was added decides how closely the model can look.
Multimodality is a retrofit, not an inborn ability. The retrofit works by cutting an image into small tiles, encoding them into vectors, and splicing them onto the text sequence. Every strength and every flaw follows from that design.
Signs you'll recognize:
- You send a full-screen screenshot and ask for a button's location; it describes the UI fluently and gets the position wrong.
- You send a table as an image; it reads the title correctly and garbles the small print.
- Two models read the same image differently, and you can't tell why.
How the image gets in
First the image is scaled to a size the model accepts. Then it's cut into tiles (patches), and each tile goes through a vision encoder to become a vector. A mapping layer projects those vectors into the same space as text tokens, and they're appended to the text sequence before the whole thing enters the model. claude-vision
The last step is the key: once joined, the model no longer distinguishes text from image. It keeps predicting the next token, one at a time. When you ask what's in the image, it's doing the same next-token continuation — the only difference is that a few hundred image vectors now sit in the sequence.
This explains several things. First, image reading uses the same reasoning machinery as text reading, so every text weakness — inventing, mixing things up, going along with you — shows up in images too. Second, images consume context, and a lot of it. Third, the ceiling on visible detail is fixed at the scaling-and-tiling step. Asking follow-up questions can't recover what was already lost.
The boundaries that follow
Anything requiring pixel-level precision is unreliable, and the mechanism is clear:
| What fails | Why |
|---|---|
| Reading small print | Scaling blurs it before tiling |
| Exact coordinates | It knows roughly which tile, not which pixel |
| Counting | Counting is already weak; tiling makes alignment harder |
| Subtle color and spacing | Compression erases these differences first |
| Very long screenshots | Compressed harder, or split into too many tiles |
| Handwriting and dense tables | Rare in training data |
The first two rows decide a rule: don't ask it for pixel-based judgments. If you need an element located, hand it structured information (the page's DOM or an element list), not a screenshot to stare at.
On the other side, it's reliable at the big picture: what the image is about, whether a layout is off, how two designs differ, what trend a chart shows. Tasks that don't need pixel precision are where it shines.
How to hand it images
- Crop, don't scale. Send only the region you care about, at original resolution. A full-screen screenshot gets crushed; a cropped region keeps its small text readable. This is the single most effective trick.
- Duplicate key information in text. Numbers, field names, error messages — paste them if you can. The image is for structure and layout, not for reading.
- One image at a time. Multiple images interfere with each other and each one eats context. If you must compare two, say clearly which is A and which is B.
- Ask a specific question. "What's wrong with this image?" gets generalities. "Are the primary and secondary buttons' visual weights reversed?" gets something useful.
- Budget for image cost. An image usually costs hundreds to thousands of tokens, more at high resolution. Estimate before building multi-image features. openai-vision
Final judgment for product people: if your feature has a hard accuracy requirement, don't hand the recognition step to a general model. Reading invoices, license plates, and barcodes is what dedicated OCR and detection models are for — more accurate and cheaper. The general multimodal model belongs in understanding and connecting: after you've extracted the structured information, let it judge what that information means.