Few-Shot Examples
An example is the highest-bandwidth way to convey a format or a judgement. It also teaches your typos, your inconsistencies, and your biases, so pick before you paste.
In a prompt, examples are the highest-bandwidth way to convey a format or a judgement — at the price of teaching your typos, your inconsistencies, and your biases along with it. So pick your examples rather than grabbing two at random. The official guides list the practice among the most effective techniques there are. openai-prompt Both vendors say the same thing here. anthropic-prompt
After reading, you should be able to answer:
- Why does one example outperform three paragraphs of adjectives, and how many should you give?
- Which ones should you pick, and what must they cover?
- What makes an example actively harmful, and how do you check yours?
What one example carries that adjectives cannot
Because an adjective is compressed information, and everyone unpacks it differently.
What does "a natural tone" actually mean? Jargon allowed or not? Long sentences or short? Address the reader by name or not? Lead with the conclusion or open with context? — When you say those words you have a very specific picture in mind, but that picture never leaves your head. It falls back on the most common writing found near "natural" in its corpus, which is the average.
One example changes that. An input paired with an output carries length, whether a name is used, whether it leads with the conclusion or handles the feeling first, what a refusal sounds like, whether a timeline is promised — every unstated requirement, delivered in one go.
There is a mechanical reason too. What it does is continue existing text: having seen two inputs with their outputs, then seeing a third input, the most natural continuation is a result in the same style. You are not teaching it anything; you are handing it a pattern to fill in. The paper that put few-shot learning on the map says this plainly: without touching the weights, a handful of examples in the context is enough for it to handle a task it has never seen. gpt3-paper
One common question worth settling: it does not "remember" those examples. They are part of this request only; once the run ends they are gone, and the next call without them behaves differently — that is the fundamental difference between examples and fine-tuning.
That being so, how many should you give? Two to five is the sweet spot for most tasks. Give only one and it cannot tell which features are mandatory and which are incidental to that particular sample; the common pattern only becomes visible at two or three. Beyond that, returns fall off fast — every example is resent on every turn at real cost.
Which ones you pick matters more than how many
The count sets a floor; the selection decides the outcome. Three principles:
One, cover different cases, not one kind. If every example you bring in is a simple case, a complex input leaves it with nothing to go on. A workable mix: one typical case, one slightly harder, plus one edge case.
Two, include how it handles "I don't know." This one pays off disproportionately. Put in an example reading "not in the material → output: no such record in the material," and it does more than repeating "do not invent" ten times in your constraints. It is the most concrete anti-fabrication move there is.
Three, the examples must be correct. It sounds like a truism, yet plenty of teams write their examples by hand: inconsistent formatting, sometimes outright wrong. It will learn the mistake faithfully.
Use structure to separate examples from instructions, with fixed markers for input and output, so it cannot confuse an example with a requirement.
When examples cost money and buy nothing
Conversely, there are three cases where you can skip them entirely — adding them there is money for nothing.
One, the task is completely standard. Pulling amounts out of a contract, normalizing dates to one format — it has seen far more of these than you can supply, and examples add nothing.
Two, variety is the point. Naming a new feature, generating a batch of taglines — examples box it in, and it drifts toward whatever you supplied. These cases call for few or no examples and more randomness.
Three, the format can be enforced. If all you want is output in a fixed set of fields, enforcing the format is more reliable and cheaper than stuffing in examples. Examples are for conveying style and judgement, not for propping up a schema.
Bias in your examples gets amplified, not averaged out
This is the trap that catches people, and it is hard to attribute afterwards — because the examples look fine when you read them.
Trap one: the bias in your examples gets amplified. Send four examples that all run to a single sentence and it will never write two, even when the input clearly needs room to breathe; send four in a refusal tone and a request you could have approved may come back sounding refused. Which features you designed on purpose and which are coincidence is a distinction it cannot make. So vary your examples deliberately, especially in length, tone, and the direction of the conclusion.
Trap two: ordering and ratio effects in classification. On intent detection, six of eight examples labelled "refund" will tilt it toward predicting refund; the order matters too, especially whichever comes last. The fix: balance the counts across classes, shuffle the order, and do not group examples of the same class together.
Trap three: inconsistent example formatting. The most insidious one. If two of three examples open with "Customer: / Our reply:" and the third just starts writing, it treats that inconsistency as a pattern worth imitating, and the output structure starts drifting at random. Align the format before you paste, one example at a time.
Before you ship it, look at the examples alone
The three traps share one property: you cannot see them from inside. So do not go by feel. Run one crude check.
Copy the examples out on their own, cover up the requirements, the constraints, everything else, and read only the examples — what would a newcomer produce by following just these?
If what they would produce matches what you want, the examples are clean. If they would also be unsure, the model will be too. If they would produce something else entirely, your written requirements and your examples are pulling in different directions, and you have to decide which side to change first. It takes minutes, and it surfaces all three traps before you ship instead of after.
Good examples and good test cases are the same set of data
The last piece of advice, and the most practical: examples should come from real data, not from your head.
Pull a few out of your actual user input and pair them with answers you would accept. Three reasons:
They are real. The examples you invent yourself tend to be smoother than you think — whereas real user input arrives truncated, misspelled, incomplete. A prompt tuned on invented examples folds the moment it meets real traffic.
They carry the edges. Real data naturally contains the awkward inputs. Deliberately keep one with missing information, one that is unusually long, one with odd formatting; those cover exactly where production breaks.
They double as your test set. That same batch of real inputs with accepted answers is a set of test cases — put a few into the prompt as examples and keep the rest for scoring. Just never let one item serve as both an example and a test case; that is handing out the answers in advance. The bigger job for that batch — judging whether a prompt change helped — is covered in its own piece.
One more bill comes due later: examples make edits sticky. Say the business raises the per-order refund cap from 200 to 500, and you find the old figure written into three of your examples — each needs changing, and you have to go back and confirm none was missed. That is a prompt asset-management problem, so annotate each example now: which problem it was added to prevent.
Once the format is settled, one more thing deserves separate treatment: when a program reads the output, propping the format up with examples is the weaker route. When a Machine Reads the Output covers how to pin the format down at the source.