How to Iterate on Prompts

Change one thing at a time and keep a set of test cases. Tuning by feel doesn't work.

The previous articles covered how to write a prompt. In real work, though, prompts are rarely right on the first try — they're tuned. And tuning has one precondition: you have to be able to tell whether a change actually made things better.

Symptoms you'll recognize:

• You try one input after an edit, it feels better, you ship it — and a different class of inputs regresses.
• The prompt has been patched so many times that no one can say why each line is there.
• You switched models and have no idea whether or how to change the prompt, or whether the result improved.

Get a ruler first

This section has exactly one core action: build a fixed set of test cases.

Take 20 to 50 real inputs, each paired with an answer you accept (or at least a criterion for right and wrong). This set becomes the ruler for every prompt decision you make.

How to pick: mostly common cases, plus five or six edge cases — very long inputs, incomplete information, weird formats, and the ones where you want it to answer "I don't know." Edge cases are the first things sacrificed in a rewrite, and they're exactly what fails in production.

The set has a bonus: it's the same data as the examples you'd put in the prompt. Good samples go in as examples; the rest become the test set. Just don't use the same item as both an example and a test — that's an open-book exam.

Building it takes half a day to a day. It's the single highest-return investment in prompt engineering, because every future change — new model, new technique, new constraint — gets judged with it. The official guides from both major vendors put "establish evaluation before changing the prompt" first in their iteration process. anthropic-prompt-eng

Change one thing at a time

With a ruler in place, the rest is discipline.

1. Change exactly one thing. Change three things and, when it improves, you won't know which one did it — worse, two might be good and one bad, the net effect looks like a small win, and you keep the bad one.

2. Write one line per change. What you changed, which problem it was meant to solve, and how the score moved. That line becomes very valuable later — six months on, when someone asks "can we delete this constraint?", you have an answer.

3. Run the full suite, not one case. This is the step people skip most. Testing one input and "feeling good" is the most common self-deception in prompt work. Run everything and look at the total.

4. Revert when it doesn't improve. Don't keep a line just because you spent time writing it. Every extra sentence adds cost, dilutes the focus, and burdens future maintenance.

As for order, work down this priority list: fill in missing parts first, then add examples, then adjust techniques, and only last touch parameters. The earlier steps usually pay off far more than the later ones.

How to read the results

When you get a set of scores, don't fool yourself in three ways.

1. Don't look only at the total. 38 to 40 looks like progress. But if it's actually "7 got better, 5 got worse," that change is dangerous — it may have just moved errors from one category to another. Compare case by case and see which ones moved.

2. Separate "unclear" from "unstable." These have completely different fixes.

Run the same input three times. If it fails all three, and fails identically, the prompt isn't clear enough — add parts. If the three runs wander, it's a randomness problem — adjust temperature. When in doubt, run the same case three times. That one small habit saves hours of guessing. The oldest advice in the official guides — change one variable, record the result — is exactly these two disciplines. openai-prompt-eng

3. Don't overfit your test set. Late in tuning, a familiar trap appears: you add contorted special cases just to make two samples pass. They pass, and everything else in production gets worse.

The signal: you start writing rules for individual samples. Stop and ask whether that sample is representative. If it's an outlier, accept that it occasionally fails.

One honest closing point: prompts don't need to be perfect, they need to be good enough and stable. Pushing 85 to 90 often costs more than moving 60 to 85. Spend the remaining gap on validation and human review instead — usually the cheaper path.