DiffBench

Anatomy of a task

How DiffBench works

1. 1 · Two snapshots, one question

   The agent gets the planning platform — a live working database, its point-in-time snapshots, and the report contract the comparison screen expects — with no engine behind it. The job: given any two moments, report what the business actually did between them. Row differences are not the answer. The weekly import rebuilds rows without changing anything; a renamed site is still the same site; a quantity re-split between storage slots leaves every total an operator reads untouched. The engine and the golden it is graded against are extracted from a shipped client system, and the original client asks from that engagement survive as companion fix tasks.

   -- the grader makes ONE change
   -- to a fresh copy of the database:
   update sites set name = 'Redrock 09B'
    where name = 'Redrock 09';
   
   -- so one correct account exists:
   --   1 changed · Site · name
   --   'Redrock 09' -> 'Redrock 09B'
   
   -- a raw row diff reports instead:
   --   8 records — one site and its three
   --   phases, each deleted-and-recreated.

2. 2 · The translation is company habit — and habits redraw

   What separates a real change from churn is never in the schema. It is a fact about how the deployment works: whether its imports recreate rows or update them, whether storage numbers name equipment or shelf slots, whether reviewers read totals or the splits beneath them. Each entity needs its own answer, evidenced only by the platform's flows and its past reports. Change the habits and every answer moves — which is why the family regenerates: variant worlds redraw the judgment at the same measured difficulty, and how much the shipped history reveals dials difficulty continuously.

   // world A — the import recreates rows,
   // so a phase's identity is its meaning:
   phases: { keyFields: ["site_customer_name",
                         "site_name", "name"] }
   
   // world B — the import updates in place,
   // so the row id is the identity:
   phases: { keyFields: ["id"] }
   
   // same schema, same ask, different habits.
   // a perfect world-A engine scores 0.4 on
   // world B. sol: 0.33 (A) vs 0.41 (B).

3. 3 · Graded against known answers

   The verifier builds two fresh copies of the database and makes one known change — renames a site, replays the import, moves quantity between slots, or touches nothing at all — so there is exactly one correct account of what the business did. The model's report and the shipped engine's report are compared record by record; naming and internal id formats are free, the drawn conclusions are not. Each scenario carries a weighted slice of the score, so partial understanding earns partial credit: frontier models land near 0.35 on fresh worlds, fail with consistent signatures, and never sweep.

Current evidence

Task breakdown

Average scores across matching versions of each public sample. The dataset leaderboard also includes private tasks.