Instructions, Examples, and the Evidence Between Them

Instructions and examples are both ways of conditioning a model, but they do different work. Instructions state goals and constraints; demonstrations provide concrete input-output patterns. The distinction matters because a prompt that seems “clear” to its author may leave the model to infer the expected format, boundary cases, or level of detail.

Eugene Yan offers a useful engineering mental model and warns that examples can narrow behavior if they fail to represent production inputs. OpenAI’s older cookbook frames instructions and demonstrations as distinct ways to steer a model. Read together, they invite a practical starting question: before adding examples, can the desired behavior be stated and evaluated clearly? That baseline leads to the harder question of what examples convey that prose does not.

A demonstration may communicate a preference more efficiently than a list of rules: tone, what counts as relevant, and how to handle a borderline case. But examples can also leave rules implicit, or encourage imitation of accidental features. That makes the choice less “instructions versus examples” than “which parts of the desired behavior are expressible as rules, and which need to be shown?”

Anthropic’s business case reports a 20% accuracy improvement after combining few-shot examples with expert-recommended workflow and other changes; it is a concrete deployment story, not an isolated experiment comparing prompt ingredients. LangChain’s experiments make the contrast more explicit: instruction optimization helped surface clear patterns in some tasks, while few-shot prompts did better at communicating nuance in another. Together, the readings suggest that the right representation depends on what the task’s “policy” contains—and set up the question of what any prompt can do when a model lacks the needed capability.

If a model already understands the task, examples may mainly steer format or preferences. If it lacks relevant knowledge or capability, additional context may help—but a prompt cannot reliably create an ability the model does not have. Distinguishing those cases is difficult: poor outputs can signal unclear instructions, missing task knowledge, weak reasoning, or an inadequate evaluation set.

DeepMind’s Flamingo post is a foundational example of few-shot prompting enabling adaptation across multimodal tasks, including results with only a handful of examples. Anthropic’s 2025 context-engineering discussion offers a later engineering perspective: begin with a minimal prompt and a capable model, then add instructions and examples in response to observed failures. These differ in emphasis—demonstrated model capability versus deployed-system guidance—but together they shift the choice from “which prompt trick?” to “what is the failure, and is it prompt-fixable?” The next session examines how examples themselves can be selected and arranged.

Once examples help, the next decisions are not trivial: how many, which ones, in what format, and in what order? More demonstrations can raise token costs, introduce contradictory patterns, or shift what the model attends to. Dynamically retrieving similar examples may help on diverse inputs, but similarity is not the same as representativeness.

LangChain reports that a few dynamically selected examples often matched or beat a larger fixed set in tool-calling experiments, with effects that differed across models and formatting choices. Hugging Face’s evaluation experiments complicate the apparent gains: minor formatting changes and even shot ordering can change benchmark scores. One source studies deployed tool behavior; the other studies evaluation sensitivity. Their tension is useful: are examples improving the model’s behavior, or sometimes changing the conditions under which we observe it? That concern makes evaluation design central to the final session.

A prompt change is only an improvement relative to a task definition, a dataset, and a measurement method. If examples used to tune the prompt also appear in its evaluation set, apparent gains may reflect overfitting. If an LLM judge is used, its own examples and criteria need validation too.

Hamel Husain argues for domain-expert labels, held-out evaluation, and error analysis when validating an LLM judge. OpenAI’s cookbook recipe demonstrates a prompt-optimization workflow that checks candidate instructions against labeled examples; it is useful as a method, though its archived recipe and model-specific details should not be treated as current model guidance. Together, they frame evaluation not as a final score but as the discipline that lets teams compare instructions, examples, and model choices without mistaking a prompt-specific win for robust improvement. The seminar ends where iteration begins: what should the next test change?