AllocBench

Anatomy of a task

How AllocBench works

1. 1 · The agent gets a planning cycle

   A supply-allocation scenario from an anonymized data-center operator: equipment lots with ERP pool noise and missing costs, deal-phase demand with required-on-site dates, and a config whose switches — supplier pins, the single-supplier rule, locked phases — are named but never explained. The retired engine's decision policy was never written down; the only spec is its run history, a salted archive of captured request/plan pairs the agent must reverse-engineer.

   {
     "config": {
       "objective": "timelineRisk",
       "enforceSingleSupplier": true,
       "supplierPins": [ { "phaseId": "phase-tus02-2",
         "equipmentTypeId": "eq-gen",
         "supplierId": "sup-duneland" } ],
       "lockedPhaseIds": ["phase-msn04-1"]
     },
     "demand": [ { "id": "req-msn04-1-pdu",
       "quantityRequired": 36,
       "needByDate": "2027-04-01", … }, … ],
     "supply": [ { "id": "lot-0016",
       "inventoryPool": " ops  spare ",
       "unitCost": null,
       "availabilityDate": "2027-06-15", … }, … ]
   }

2. 2 · It ships an allocation engine

   Not a plan — a solver. The deliverable is a pure function from request to assignments that reproduces the legacy engine's behavior: its lexicographic objective (maximize fill, then minimize lateness, then cost), its pricing of unpriced lots, its pool exclusions — every rule the archive evidences.

   # allocator/solve.py
   def solve(request: dict) -> dict:
       """Return an allocation response
       for an allocation request."""
       …
       return { "assignments": [
         { "demandId": "req-msn04-1-pdu",
           "lotId": "lot-0142",
           "quantity": 24 }, … ] }

3. 3 · Grading is what the plan achieves

   Run → Check rules → Compare → Perturb

   The verifier re-derives feasibility from the contract, then compares the plan's fill, lateness, and cost against the golden optimum — any optimal plan passes, whichever lots it picks. Then the world changes: new, cheaper lots land and one more phase locks. The engine re-runs, and locked phases must come back byte-for-byte even though re-optimizing them would look better. That freeze rule exists because a real operator once watched a locked phase move.

Current evidence

Task breakdown

Average scores across matching versions of each public sample. The dataset leaderboard also includes private tasks.