Multimodal Input

An image is resized, cut into patches, encoded into vectors, and spliced into the text sequence. It is not looking — it is continuing a sequence.

It does not see the image. It cuts the image up, encodes it into the same kind of vectors text becomes, splices it into the sequence, and writes on. That retrofit decides how closely it can look — and which mistakes it will make when it reads.

After reading, you should be able to answer:

• Why can it describe a report fluently and still misread one number?
• Why can two models read the same image differently?
• What three things should you do to a photo before handing it over?

It is not looking, it is continuing

An image enters the model through four operations. First it is shrunk to a size the model permits. Second it is sliced along a grid, each cell being a tile. Third every tile goes to a vision encoder and becomes a string of vectors. Fourth those vectors are projected once more, landing in the same space as text tokens, and are then mixed into the text sequence to go in together. claude-vision

That last operation is the one that matters: once mixed in, the model can no longer tell which content started out as text and which started out as an image. It goes on deciding, at the same position, what the next token should be. Asking what is in the picture is still completing a sequence — the only difference is that a few hundred vectors from the image now sit among the material available for that continuation.

Three consequences follow. First, looking at images and handling text draw on the same reasoning machinery, so the familiar text failures — inventing, mixing things up, going along with what you said — show up just the same. Second, images take up context, and a considerable amount of it: an ordinary screenshot usually works out to several hundred tokens, sometimes over a thousand. Third, how much detail it can resolve is capped at the scaling and slicing operations, and no amount of follow-up questioning recovers what was already lost.

Pixel-level work is what it is built to fail at

Turn those four operations around and each kind of reading error has its own origin. These are not occasional slips. They fall out of how the retrofit was built; a newer model may soften them, but it will not rub them out.

The first family of errors starts at the scaling step. Once an image is shrunk, what disappears first is the gap between fine strokes: photograph a printed manual and the heading comes back word-perfect while the small-print footnote comes back wrong throughout. Subtle colour and brightness differences go the same way — two posters differing only in how deep the background tone is will usually be reported as looking the same.

The second family comes from the tiling. It knows roughly which tile something falls in, but within a tile there is no finer coordinate to speak of, so ask where a bar sits or which pixel an element occupies and all you get is a coarse position. It also does not count: ask how many of one icon appear and the answer is an estimate, and quite how tiles line up along their edges is not clear to it either.

The third family comes from the frame. A long image that will not fit on one screen has no good route: shrink the whole thing and detail disappears faster, or cut it into many tiles and the correspondence between tiles becomes a new source of error. Show it ten screens of chat history and it can tell you what the conversation was about, but not reliably which screen a given message sat on.

The fourth family comes from the training data. Handwritten signatures, scanned stamped documents, tables whose gridlines need zooming before you can read them — samples of these were rare in the corpus, so it never had the chance to become dependable on those shapes.

Together the four families point to one rule: anything that needs a pixel-based judgment should not go to it at all. If you need precise positions, rely on structured information you supply — an element list, the DOM, instrumented events — rather than letting it guess from the picture.

How much of each family you hit varies by model, because the scaling, tiling, and encoding steps are each vendor's own choices: the largest edge length permitted, the most tiles one image may be cut into, the data the encoder was trained on — every one of those moves the detail ceiling. So when model A reads the amount on a blurry receipt and model B reads a different number, it is often nothing more than two different settings for scaling and tiling. You did not ask the wrong question, and one of them is not simply cleverer. Since the output alone cannot tell you whether the model or the image is at fault, try a sharper image once: if both models still get it wrong, the input is the problem; if only one does, the model is.

What it is good at is equally clear: the whole, the relationships, the meaning. What the image says overall, whether a layout is visibly awkward, where two designs differ, what trend a line chart presents — judgments that do not call for pixel precision are ones it handles dependably. Draw that line first, and only then decide how to hand it the image.

Before you hand it over, change three things

The first is crop, do not scale. If you care about one region, cut that region out and let it arrive at original resolution, instead of hoping detail can be coaxed out of the full image afterwards. Given a screenshot of a back-office ticket page, do not send the whole thing and ask about the status flow — cut out that column on its own. Of the three, this one does the most work.

The second is restate the key details as text. Numbers, field names, error messages — write them out in text wherever you can. The image is there so it can grasp structure and layout, not so it can read characters.

The third is one image at a time, each with a stated position. Several images interfere with each other, and each one eats a share of context. If you really must compare two, say that the first is before the redesign and the second after, then ask what differs — do not leave it to guess the order.

None of the three is a trick. They follow directly from those four operations: anything that keeps the input closer to original pixels, or gives the judgment firmer ground, improves the result.

The more specific the question, the more useful the answer

"What is wrong with this image?" mostly returns vague talk. Switch to "in this report, is the refund column higher than last month?" or "which of these two pages has the wider top margin?" and the answer has somewhere to land. The reason is unchanged: it is continuing your sentence, and your sentence decides which way it continues.

Bear in mind too that an image is not a free input. It is billed through the context it takes up, more at higher resolution, and in multi-image scenarios that quantity is easily undercounted. openai-vision So estimate before building: how many images per request, what each works out to, what one call costs, how many calls a day. Skip that and you discover the cost after launch, when the only move left is cutting features. How image input is priced, and why it costs more than the same amount of text, has its own section in Three Prices Per Call.

Do not hand accuracy-critical work to a general model

If your feature is strict about accuracy, do not use a general multimodal model for the reading step. Identity document numbers, the address block on a parcel label, defect photos on a production line — each has dedicated recognition and detection models: lower cost, higher accuracy, deterministic output.

Where a general multimodal model belongs is understanding and connecting: after the structured information has already been extracted, it judges what that information means, whether anything is anomalous, which bucket it belongs in. Doing both jobs in one place buys you higher fees and a higher error rate together.

Where structured information will do, do not use an image

The final judgment is more fundamental: where you can get structured information, do not make the model look at a picture. To find out where an element is or whether it is obscured, an element list is cheaper and more accurate than a screenshot; to count things, a query beats counting in an image. Keep images for the places that genuinely need a look at the whole — whether a screen feels right, where the visual weight of each option falls.

Stop Making the Model Carry Data is the other face of the same principle: exactness goes to the system, judgment goes to the model. The multimodal layer is where it is easiest to break, because throwing an image at the problem looks so effortless.