Controlling Output Format

From "return JSON" to enforced formats, and the trade-offs when streaming.

If a program has to consume the model's output, the format must be stable. And "just say it in the prompt" is roughly 90% reliable — which sounds fine until you realize that in production it's hundreds of failures a day.

Signs you'll recognize:

• You asked for JSON only, and it still prefixed "Sure, here's your result:"
• Your parser is full of compatibility hacks and still crashes now and then.
• You enabled streaming, and the frontend receives half a JSON object it can't parse.

Three layers

Layer one: ask in the prompt. The basic move — describe the format. Two tricks meaningfully raise the hit rate: give a format sample (far more effective than describing it — this is the few-shot trick), and say explicitly "no explanatory text; the first character must be an opening brace."

But it's a request, not a guarantee. The model is fundamentally doing probabilistic continuation, and "Sure, here's your result:" is extremely common in the corpus, so it will surface with some probability.

Layer two: enforce through API parameters. Most platforms offer something like response_format that constrains output at the generation layer — only tokens that fit the JSON grammar are allowed. This isn't post-checking; it takes away the option to stray. Taking it further is providing a schema: not just valid JSON, but which fields and types. Use it whenever you can — this is the single biggest reliability jump. Both OpenAI and Anthropic ship structured-output support, and their docs are the practical reference for what each actually enforces. openai-structured claude-structured

But know the boundary of what it guarantees: syntax is guaranteed, content isn't. Fields are present, but values can be invented; types are right, but numbers can be wrong. Format constraints can't stop hallucination.

Layer three: validate yourself. This can't be skipped, even with the first two layers in place. You must write the branches for parse failure, missing fields, and clearly unreasonable values.

Common fallbacks: retry once (attach the error message and let it fix itself), degrade (switch models or fall back to a simpler mode), or refuse (tell the user it failed, instead of showing garbage).

One practical detail: make the parse-failure rate a monitoring metric. When it creeps up, it usually means the upstream model version changed — especially if you're pinned to a "latest" model name.

Design the format itself

Constraints aren't enough. Whether the model fills the format correctly depends on how you designed it. Four pieces of experience:

• Self-explanatory field names. category beats c; confidence_0_to_1 beats score. The model fills values by following the field names' semantics; ambiguous names make it guess what you want.
• Don't nest too deep. Beyond about three levels, the error rate climbs visibly. Flatten when you can.
• Bound the enums. For classification, list every allowed option — "category can only be one of these five values." Otherwise it invents new categories, plausibly enough that you won't notice at first.
• Leave an "I don't know" exit. If every field is required, when the material lacks the information, it must fabricate. A legal null or "not mentioned" option gives it an honest path — one of the most practical anti-hallucination moves.

The conflict with streaming

This is a product trade-off you must decide early; many teams only hit it after building.

Streaming dramatically improves perceived speed: characters appear one by one, far nicer than a spinner. But it conflicts with structured formats — half a JSON object can't be parsed. You'd have to wait for the full output, and then streaming is pointless.

Three ways to handle it, by scenario:

For humans, don't use JSON. If the output is text shown directly to a user, enable streaming and ask for plain text or Markdown. Don't force structure for tidiness.

For programs, disable streaming. Extraction and classification output is for code; users don't see the intermediate process, so streaming adds nothing. Turn it off and use strict format constraints.

For both, split into two segments. The common real case: you want a natural answer for display plus structured metadata. Make it stream the human-facing text first, then output a short structured block at the end — the frontend renders the text, the program parses the tail. Or simply make two separate calls, each with its own job.

Decide this trade-off in the design phase. "Is this output for a human or for a program?" is a question to ask before writing the prompt.