What Should the Agent Know? Choosing Documents by Their Marginal Value

Incremental selection often begins with compressed descriptions: titles, tags, abstracts, or short summaries. They make a large candidate pool tractable, but they can hide the very details that determine whether a source helps. Anthropic’s contextual-retrieval experiments sharpen this concern: generic document summaries added to chunks yielded limited gains, while chunk-specific context improved retrieval. That is a warning against treating any short description as a dependable proxy for downstream usefulness.

Google’s Open Knowledge Format approaches the compression problem from another direction: standardize lightweight metadata and descriptions so agents can navigate portable, curated knowledge. But a format for describing knowledge is not an experiment proving that descriptions rank candidate documents well. Put together, the posts distinguish a useful map from the territory it summarizes. The next question is what “useful coverage” should mean once a candidate is admitted.

A topic-breadth strategy tries to avoid building a library that knows one subject deeply and everything else poorly. But breadth can be a misleading objective if the agent’s actual tasks depend on a few precise facts or on multiple documents jointly supporting one answer. Hamel Husain’s discussion of FreshStack argues that RAG retrieval should be judged not only by relevance, but also by coverage and redundancy: a useful result set may need to span the evidence for an answer without returning near-duplicates.

Google’s RAG evaluation guidance makes the measurement problem concrete: test questions should cover a broad subset of the underlying data and vary in phrasing and complexity. That is an evaluation recommendation, not evidence that a broad-first acquisition policy wins. The tension is whether selection should optimize abstract topic spread or expected answer coverage; resolving it requires looking at what happens when the agent actually encounters missing knowledge.

Gap-driven selection sounds straightforward: observe where the agent struggles, then add a document that fills the gap. Google Cloud’s October 2026 Knowledge Catalog post describes a product loop that analyzes agent and user interactions, identifies missing or outdated context, and stages proposed changes for review. This is a current engineering direction, but its reported performance figures are internal benchmarks, not an independent comparison of gap-driven selection against random or breadth-first alternatives.

Meta’s “organizational second brain” makes the diagnosis harder and more useful. Its engineers distinguish a knowledge gap from a recipe or reasoning problem and from genuine ambiguity; they also separate frequently used, information-dense knowledge from situational sources that belong in retrieval. That distinction matters because adding a document cannot repair a failure when the agent already had the necessary evidence. The next session asks how to test that distinction without changing the model at the same time.

If a larger or different model is introduced alongside a new selection policy, an improved answer does not reveal which change mattered. Red Hat’s AutoRAG demonstration offers a useful controlled pattern: sweep retrieval configurations, score retrieval separately, then compare end-to-end answers using the same small model. Its example also shows why retrieval gains are not equivalent to task success: the selected configuration reduced context substantially, yet the model still missed many questions under the strict answer check.

Hamel Husain and Shreya Shankar make the complementary methodological point: retrieval and generation need distinct evaluation, with application-specific judges validated against human labels. Neither post studies document-acquisition policies directly, but together they suggest a disciplined test for them: hold the agent, prompts, and task set fixed; vary only the selected knowledge base; and measure both retrieval behavior and downstream answers. The remaining issue is whether those gains hold beyond the cases that guided selection.

A selection rule can appear successful because its evaluation questions mirror its assumptions. Google Cloud’s Discovery Bench discussion argues for varying query difficulty rather than relying on a single phrasing, revealing performance cliffs that a static score can conceal. Although it studies agent discovery rather than knowledge-base acquisition, the lesson transfers: measure how selection performs as questions become vaguer, more complex, or differently phrased—not just on a comfortable benchmark average.

Casey West’s evaluation-engineering lessons add a second caution: grading only the final response can miss whether the agent followed the right execution path. For a document-selection experiment, that means tracking whether the agent found and used the chosen evidence, not only whether its final answer happened to be correct. These posts do not establish a winning policy; they help define a harder standard for claiming one. The seminar ends with the open test: can breadth- or gap-driven selection beat random on held-out tasks, across difficulty levels, without quietly changing the agent?