Ology

The Model at the Edge of Its Knowledge

Methods for evaluating whether large language models recognize gaps, uncertainty, and contradictions in learned knowledge

5 sessions · 13 readings · 3 views

Curated by Ned Horowitz

Missing Fact, Lost Keysession 1Confidence with Consequencessession 2Search Is a Decisionsession 3The Context Can Misleadsession 4Measure the Boundary, Not Just the Answersession 5

Opening

An incorrect answer can mean several different things: the model never encoded the fact, cannot retrieve it in this phrasing, has conflicting associations, or has no reliable evidence at all. Those causes call for different remedies. A newer line of work makes that distinction explicit by probing what models can reproduce with contextual cues versus what they can recall directly—while leaving open how cleanly such probes reveal what a model “knows.”

The engineering problem has also shifted. Retrieval-augmented systems can supply material absent from a model’s parameters, but they introduce new failure points: the search may miss the answer, retrieve misleading evidence, or provide a passage the model ignores. Tool use is not automatically reliable just because a tool is available; the model still has to decide when to use it and what to do with what it finds.

Even recognizing uncertainty is not a single skill. A model might estimate that a proposed answer is likely wrong yet still answer; it might abstain appropriately on unfamiliar questions but refuse answerable ones; or it might appear well-calibrated on familiar data and fail under shift. Evaluations that count only correct answers can obscure these tradeoffs—and may reward guessing.

This seminar follows the problem from internal knowledge signals to abstention, search, evidence handling, and evaluation. Its central question is: when should a model treat uncertainty as a reason to qualify, investigate, or withhold an answer—and how can we tell whether it chose well?

Five questions worth arguing about

  1. 1When a model misses a fact, what evidence distinguishes absent knowledge from knowledge it cannot reliably retrieve?
  2. 2Should confidence training optimize answer-or-abstain decisions, or teach graded uncertainty before refusal?
  3. 3Should a model search whenever confidence falls, or only when external evidence could improve the answer over abstaining?
  4. 4When retrieved passages are plausible but incomplete, how can systems detect missing support or contradictory evidence before answering?
  5. 5Do abstention scores measure calibrated uncertainty, or just a model’s ability to learn a narrow refusal policy?

The Sessions

1Session 1start here

Missing Fact, Lost Key

The problem

A wrong answer does not reveal whether the fact was absent from the model’s learned knowledge or simply inaccessible to the prompt. Anthropic’s experiments distinguish confidence in a proposed answer (“P(True)”) from a question-level estimate of whether the model knows the answer (“P(IK)”). They report promising self-evaluation, but also weaker calibration when P(IK) is transferred to new tasks.

Google Research’s newer knowledge-profiling approach separates encoding from recall: it tests whether facts can be elicited in pretraining-like contexts and whether they can be answered across differently phrased questions. That offers a richer diagnosis than accuracy alone, but the distinction depends on behavioral probes, not a direct readout of internal memory. If a cue changes performance, has it uncovered stored knowledge—or supplied a useful path to inference? That ambiguity leads to the next session: what should a system do with a fallible estimate of its own limits?

Readings

  1. Language models (mostly) know what they knowAnthropic
    Open ↗
  2. Empty shelves or lost keys? Recall is the bottleneck for parametric factualityNitay Calderon and Gal Yona
    Open ↗
2Session 2Confidence with Consequences2 readings3Session 3Search Is a Decision3 readings4Session 4The Context Can Mislead3 readings5Session 5Measure the Boundary, Not Just the Answer3 readings

Next Up