Test Zero-Shot and Few-Shot Prompts on the Same Task

You added three examples to your prompt. The output looked better. You shipped it.

That is not a result. It is a feeling. The examples may have fixed your format, sharpened your labels, or done nothing at all — and you have no way to tell which, because you never wrote down what "better" meant before you looked.

Here is the sharper model: adding examples is a change to the prompt, and every prompt change is a hypothesis. A hypothesis needs a test. By the end of this exercise you will have a fixed task, a fixed set of test cases, a scoring rule written before you run anything, and a result you can defend to someone who asks "how do you know?"

You already know how to pick a strategy by intuition. This is about checking that choice with output.

Why "Examples Help" Is Not a Result

The standard advice is: start zero-shot, add examples if it fails. That is a good default. It is also a heuristic, not evidence about your task.

Examples can help. They lock down output format, demonstrate a tricky label, and show the model the shape you want. They can also hurt. A poorly chosen example pulls the model toward the wrong pattern. A set of easy examples teaches the model to handle easy inputs. Research on reasoning tasks has found that adding worked examples sometimes introduces repetition and logical errors rather than improving reasoning, and that for capable models the main benefit of examples is often format alignment rather than better thinking.

That last point matters. If examples mostly fix format, then a comparison that only measures "did the answer look right" will credit examples for something a stricter output contract could have done for free.

Published comparisons will not settle this for you. Results depend on the model, the task, the wording, and how you score the output. So run your own. Three things make the comparison honest:

• A fixed task.
• Fixed test cases.
• Criteria written down before you look at any output.

Pick a Task You Can Score

Your first experiment needs a task where right and wrong are visible at a glance.

Good candidates:

• Classification with a small label set (sentiment, intent, category).
• Extraction into fixed fields (names, dates, amounts).
• Reformatting text into a strict shape (a single line, a fixed field order).

Bad candidates for a first run:

• Open-ended writing.
• Long reasoning chains.
• Anything where you would argue about whether the output is good.

Write the task instruction once, in plain language, and keep it identical across both prompts. Decide the output contract now: exact label names, field order, or a single line of output. Then state the model and settings you will hold constant. If you change the model or the temperature mid-experiment, you have invalidated the comparison and learned nothing.

Build a Small Fixed Test Set

Aim for 8 to 12 cases. Enough to see a pattern, small enough to inspect every output by hand.

Choose cases to expose failure, not to flatter the prompt:

• A few easy cases.
• A few ambiguous ones.
• At least one edge case where the correct answer is genuinely debatable.

Write the expected answer for each case before you run anything. If you decide what "correct" means after seeing the output, you are not scoring, you are rationalizing.

Common mistake: Putting a test case into the prompt as an example. Then you are measuring memorization, not generalization. Keep the test set out of the prompt entirely.

Say the limitation out loud now: this is a smoke test, not a benchmark. Ten cases will not prove anything. They will tell you where to look.

Write the Zero-Shot Prompt First

The baseline prompt contains three things and nothing else: the task instruction, the output contract, and the input.

Classify the sentiment of the text below.
Respond with exactly one word: positive, negative, or neutral.

Text: {input}

Do not sneak in a worked example as a "clarification." The moment you add one, your baseline is a few-shot prompt and the comparison is dead.

Run all test cases. Save the raw outputs verbatim, including the malformed ones. Watch for the classic zero-shot failure: correct reasoning wrapped in an output format you cannot parse. The model says "This is clearly negative" when you asked for one word. The answer is right; the output is unusable.

Add Examples Without Changing Anything Else

Now build the few-shot variant as a controlled change. The instruction, the output contract, and the test cases stay byte-for-byte identical. The only new material is the examples.

Classify the sentiment of the text below.
Respond with exactly one word: positive, negative, or neutral.

Text: The package arrived two days early.
Sentiment: positive

Text: It works, but the manual is useless.
Sentiment: negative

Text: {input}

Use two to four examples. Draw them from the same distribution as your test cases but never overlap them. Choose examples that demonstrate the output format and cover the tricky label — not examples that are simply easy.

Run the same test cases and save the outputs the same way.

Warning: If you also rewrote the instruction while adding examples, you changed two variables. Any difference in results is now unattributable. Change one thing.

Score Both Runs Against Written Criteria

Turn raw outputs into a number you can trust. Score each case on two separate axes:

Axis	Scale
Task correctness	correct / wrong
Format compliance	valid / invalid

Keep them separate because they fail for different reasons. A right answer in the wrong shape is still unusable in a pipeline. If you collapse both into one score, you will not see which one the examples actually fixed.

For a single-label task like sentiment, "partially correct" is a trap — it invites you to decide after the fact whether a mixed sentence "kind of" counts. Pick one label per case before you run, and score it correct or wrong. If a case is genuinely ambiguous, that is a finding about your fixture, not a reason to soften the scale.

Record the score per case, not just the total. A total tells you who won. A per-case table tells you where the run changed.

Case	Zero-shot	Few-shot	Notes
1	correct / valid	correct / valid	easy case
2	correct / invalid	correct / valid	format fixed
3	wrong	wrong	ambiguous input
4	correct	wrong	few-shot pulled it wrong

That table is the entire deliverable. Everything else is commentary.

Read the Cases, Not Just the Totals

Now interpret. Look for three patterns.

Where few-shot fixed format. This is the most common and most reliable benefit of examples. If most of your few-shot gains are format compliance, you have learned something useful: you may not need examples at all — you may need a stricter output contract.

Where few-shot made the answer worse. This usually happens on ambiguous inputs, where the model pulls the case toward the pattern of your examples instead of reading it on its own terms. That is overfitting to your examples, and it is the failure mode nobody warns you about.

Where your examples leaked the answer. If a test case looks suspiciously like one of your examples, the few-shot score is inflated for the wrong reason. Check for this before you celebrate.

If the totals are close, say so. A tie on a small fixture is a real result, not a failed experiment. It means the examples did not earn their place on this task.

Write one sentence naming the mechanism you believe caused the biggest difference. Then write one sentence on what would falsify it. That second sentence is what turns an observation into a testable claim.

Change One Thing and Run It Again

A single run gives you a result. A follow-up run gives you an explanation.

Pick one variable:

• Swap in a different example set.
• Add a fifth example.
• Replace one example with a harder edge case.

Predict the outcome before you run it. Then compare your prediction to the result. The gap between them is where the learning is — if you expected format to improve and correctness dropped instead, you just found the boundary of your mental model.

If you have access to a second model, run the same fixture on it. Note that the conclusion may not transfer. Different models have different tolerances for examples, and a result on one is not a result on all.

Keep the fixture and the scoring rule unchanged so the new run is comparable to the first.

What This Experiment Cannot Tell You

Be honest about the boundaries.

• Ten cases cannot separate a real effect from noise. Treat the result as a signal to investigate, not a verdict.
• Results are tied to this model, this wording, and this fixture. None of them transfer automatically.
• Your scoring is subjective at the margins. On ambiguous cases, a second reader would score some rows differently.
• Sampling is non-deterministic. A rerun can shift results even with no prompt change. Note whether you used a low-temperature setting, and if you did not, expect variance.

The practical takeaway is not the score. It is the fixture. Keep it. It becomes a regression test you can rerun every time you touch the prompt — and a prompt you cannot re-test is a prompt you cannot safely change.

The Rule to Reuse

Never ship a prompt change you have not measured on cases you wrote down first. That single habit separates people who guess from people who know.

Your next step: turn this fixture into a small reusable test set you rerun whenever you edit the prompt. Pair it with a structured-output contract so format failures become visible instead of silent — a right answer in the wrong shape should fail loudly, not quietly pass. Then run the same experiment on your own real task, and let the output tell you whether the examples earned their place.