Ology

From Question to Evidence: Building Agents That Can See, Select, and Defend

AI agents that answer questions by selecting, generating, critiquing, and explaining data visualizations

5 sessions · 13 readings · 1 view

Curated by Ben Lorenzo

The Agent Must Choose a Pathsession 1Evidence Before Aestheticssession 2When a Chart Becomes an Argumentsession 3Make the Chart Defend Itselfsession 4Trust the Trace, Not the Polishsession 5

Opening

Question-driven visualization is becoming an agent problem rather than a charting problem. Recent systems can inspect schemas, retrieve relevant data, execute code, generate notebooks, choose visual forms, and continue investigating when an intermediate result looks wrong. The important shift is that the agent is no longer merely rendering a chart from a prepared table; it is helping decide what the table should contain.

That shift makes the task deceptively difficult. A question such as “Which routes are most unreliable?” requires definitions, joins, filters, aggregations, and comparisons before visualization begins. The resulting chart may be technically correct yet answer the wrong question, conceal uncertainty, or omit the comparison that makes the evidence meaningful.

The newest systems also make their own workflows part of the product. OpenAI’s internal data agent exposes analysis traces and evaluates generated queries against expected results; notebook-based systems use executable artifacts as both working memory and evidence; other systems send generated plots back through a vision-language model for critique. These approaches disagree about how much autonomy, structure, and explanation an agent needs.

The seminar therefore follows one central problem: how can an agent move from an ambiguous question to a selective, well-encoded, self-criticized visualization without confusing plausible analysis with warranted evidence?

Five questions worth arguing about

  1. 1Should a visualization agent be autonomous, or should its analysis path be constrained by a workflow?
  2. 2Does better context selection matter more than a stronger model for finding the evidence a question actually needs?
  3. 3Can chart selection be treated as a reasoning decision rather than a final formatting step?
  4. 4What would make an agent’s critique of its own chart evidence-based instead of merely stylistic?
  5. 5How should we evaluate an agent when a plausible chart hides a wrong query, aggregation, or omitted comparison?

The Sessions

1Session 1start here

The Agent Must Choose a Path

The problem

A question-driven visualization system cannot begin with a chart type. It must decide whether to retrieve metadata, inspect the schema, search for definitions, write code, query a database, ask for clarification, or iterate after seeing an unexpected result. The first design choice is therefore architectural: should the system follow a predictable pipeline, or should the model decide dynamically what to do next?

Anthropic’s architectural distinction between workflows and agents provides the useful baseline. Fixed workflows offer predictability and easier testing, while autonomous loops offer flexibility when the number and order of analytical steps cannot be known in advance. OpenAI’s internal data agent shows why the latter can be valuable: open-ended questions require table discovery, contextual interpretation, query execution, and course correction rather than a single text-to-SQL conversion.

The tension is not autonomy versus simplicity in the abstract. It is whether the uncertainty lies in the user’s question, the data environment, or the analytical path. The next session examines how much of that uncertainty can be reduced before the model ever starts reasoning.

Readings

  1. Building Effective AI AgentsErik S. and Barry Zhang
    Open ↗
  2. Inside OpenAI’s in-house data agentBonnie Xu
    Open ↗
2Session 2Evidence Before Aesthetics3 readings3Session 3When a Chart Becomes an Argument3 readings4Session 4Make the Chart Defend Itself2 readings5Session 5Trust the Trace, Not the Polish3 readings

Next Up