InfraBench

Anatomy of a task

How InfraBench works

1. 1 · The agent gets a framework

   A runnable copy of n1x — the code-generation framework that stamps out client applications from composable plugins — plus one engineering instruction taken from real capability work.

   # Add composable AG Grid and AG Charts
   # Enterprise support to n1x
   
   Introduce a `web-ag` capability that requires
   `web` … registers AllEnterpriseModule exactly
   once, including across development reloads. …
   Keep the capability isolated and lifecycle-safe:
   - `web-ag` must work without `infra`.
   - Repeated apply operations must be idempotent.
   - Removing `web-ag` must clean its managed
     output without removing PostHog …

2. 2 · It ships a plugin

   The deliverable is a declarative plugin: a manifest plus managed templates, composed through the framework's existing seams. The core engine and unrelated capabilities stay untouched.

   {
     "name": "web-ag",
     "requires": ["web"],
     "replace": [ { "from": "files/src/_lib/ag.ts",
       "to": "web/src/_lib/ag.ts" } ],
     "mergeJson": { "web/package.json": {
       "dependencies": {
         "ag-charts-enterprise": "14.0.0",
         "ag-grid-enterprise": "36.0.0", … }
     } }, …
   }

3. 3 · Grading is generated behavior

   Generate → Execute → Recompose → Remove

   The verifier stamps out real projects from the modified framework — alone, combined with other plugins in both orders, reapplied, and removed — and runs the generated code with stubs. Structure is never string-matched against a reference patch.

   4% core engine and unrelated capabilities are unchanged 4% web-ag resolves web and renders its managed helper 1% web-ag pins ag-charts-community 1% web-ag pins ag-charts-enterprise + 28 more weighted checks

Current evidence

Task breakdown

Average scores across matching versions of each public sample. The dataset leaderboard also includes private tasks.