Ology

The Upgrade’s Evidence Trail

Source-traceable knowledge graphs for dependency upgrades, code-change mapping, and predicting migration breakage

5 sessions · 10 readings · 3 views

Curated by Ben Lorenzo

Warnings Aren’t Impact Mapssession 1The Call-Site Neighborhoodsession 2Memory With a Maintenance Budgetsession 3Test Green, Behavior Unknownsession 4After the Pull Requestsession 5

Opening

Dependency assistants are moving from announcing that a version changed to proposing code changes in response. GitHub now lets developers send some Dependabot alerts to coding agents; meanwhile, migration systems described by Google and AWS use repository structure, validation results, or accumulated migration artifacts to guide work. The important shift is not simply that agents can edit more files: it is that they can draw on more evidence than a prompt and a changelog.

That evidence comes in different forms and at different scales. A package advisory can describe a vulnerability, but not whether a project reaches the affected API. Symbol references can locate call sites, but may miss behavior encoded in tests or conventions. Prior successful migrations can help future ones, but only if their lessons remain relevant and their provenance is inspectable.

The tradeoff is becoming practical: a one-off agent run may be quick to set up, while reusable playbooks or migration agents promise consistency across many repositories. Yet maintained knowledge has costs—curation, freshness, and the risk of turning an edge-case fix into a misleading rule. And a clean build is not proof that a migration preserved behavior.

These sessions follow the trail from evidence to impact, reusable memory, and evaluation. What should an assistant be able to show before its upgrade advice deserves trust?

Five questions worth arguing about

  1. 1When package advisories, source references, and maintainer guidance disagree, which evidence should govern an upgrade recommendation?
  2. 2Should an assistant retrieve only changed call sites, or enough neighboring code and tests to reconstruct local intent?
  3. 3Does migration knowledge become more trustworthy when distilled into maintained playbooks, or less faithful to edge cases?
  4. 4Which failures should evaluations catch beyond incorrect final code: missing evidence, skipped checks, unstable runs, or behavior regressions?
  5. 5When one migration takes weeks and another exposes compiled regressions, what should count as credible evidence of preserved behavior?

The Sessions

1Session 1start here

Warnings Aren’t Impact Maps

The problem

A package advisory describes a risk in the dependency; a project-specific recommendation has to establish whether the project actually uses the affected functionality. Google’s migration workflow starts from candidate locations gathered through static tools and human input, then expands them using symbol cross-references. GitHub’s Dependabot-agent workflow begins from an alert and repository dependency usage, then asks an agent to propose a fix and respond to test failures. These are complementary evidence paths, not interchangeable ones.

Neither post establishes that the agent has a complete account of every relevant code path. Google describes how it finds locations and related files; GitHub describes what the agent is asked to inspect and do. The distinction between an alert’s scope and a repository’s actual exposure is where the next question begins: how much surrounding code is needed to map the impact?

Readings

  1. Accelerating code migrations with AIStoyan Nikolov and Siddharth Taneja
    Open ↗
  2. Dependabot alerts are now assignable to AI agents for remediationGitHub
    Open ↗
2Session 2The Call-Site Neighborhood2 readings3Session 3Memory With a Maintenance Budget2 readings4Session 4Test Green, Behavior Unknown2 readings5Session 5After the Pull Request2 readings

Next Up

The Upgrade’s Evidence Trail · Ology