InfraBench

hard · software-engineering

Add a composable web Cloud Run job-runner capability to n1x

Add a composable web Cloud Run job-runner capability to n1x.

Task version0.3.0

Checksumf7c427d17d9c

Published attempts9

Full reward1.00

Current task results

Evaluation comparison

Each bar averages that model configuration's verifier rewards for this task. The latest published batch 2026-08-05__12-16-53 contains 9 attempts across 3 model configurations.

v9m-rl-learnability-tp8openai
96.0%± 0.0 pp SD
3/3 scored attempts
gpt-5.6-lunaopenai
93.0%± 1.7 pp SD
3/3 scored attempts
gpt-5.6-solopenai
84.3%± 22.9 pp SD
3/3 scored attempts

± is sample standard deviation across scored attempts. The line spans minimum to maximum; the heavier marker is the mean. Fixed 0–100 scale.

Latest published cohort

Verifier criteria

Pass rate across 9 attempts from 3 model configurations in batch 2026-08-05__12-16-53. Reward weight is how much each check contributes to one attempt's total score. Expand a criterion to see attempt outcomes by model configuration.

core engine and unrelated capabilities are unchanged3%3/933%

core engine and unrelated capabilities are unchanged attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 failed2/3
gpt-5.6-solopenai	Attempt 1 failed Attempt 2 failed Attempt 3 passed1/3
v9m-rl-learnability-tp8openai	Attempt 1 failed Attempt 2 failed Attempt 3 failed0/3

web-job-runner resolves its prerequisites without infra5%9/9100%

web-job-runner resolves its prerequisites without infra attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

Next tracing includes the project job roster5%9/9100%

Next tracing includes the project job roster attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

plain web keeps the shared seam and project-owned routes4%9/9100%

plain web keeps the shared seam and project-owned routes attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

standalone documents optional credentials and fail-closed use5%5/956%

standalone documents optional credentials and fail-closed use attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 failed Attempt 2 failed Attempt 3 failed0/3
gpt-5.6-solopenai	Attempt 1 failed Attempt 2 passed Attempt 3 passed2/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

infra derives a valid suffix-safe runner identity5%8/989%

infra derives a valid suffix-safe runner identity attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 failed Attempt 2 passed Attempt 3 passed2/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

Terraform grants the runner exactly the two-job roster12%8/989%

Terraform grants the runner exactly the two-job roster attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 failed Attempt 2 passed Attempt 3 passed2/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

runner key and Terraform outputs have the required semantics6%8/989%

runner key and Terraform outputs have the required semantics attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 failed Attempt 2 passed Attempt 3 passed2/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

Vercel receives region and sensitive runner credentials6%8/989%

Vercel receives region and sensitive runner credentials attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 failed Attempt 2 passed Attempt 3 passed2/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

local env workflow writes both values only when configured7%9/9100%

local env workflow writes both values only when configured attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

consumer composition is independent of requested plugin order5%9/9100%

consumer composition is independent of requested plugin order attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

shared policy has one neutral owner and both legacy moves7%9/9100%

shared policy has one neutral owner and both legacy moves attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

removing runner preserves the Gemini shared owner4%9/9100%

removing runner preserves the Gemini shared owner attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

removing Gemini preserves the runner shared owner4%8/989%

removing Gemini preserves the runner shared owner attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 failed Attempt 2 passed Attempt 3 passed2/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

removing both consumers cleans the shared owner4%9/9100%

removing both consumers cleans the shared owner attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

projects without credential consumers stay isolated5%9/9100%

projects without credential consumers stay isolated attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

repeated apply preserves project-owned roster, route, and edits5%9/9100%

repeated apply preserves project-owned roster, route, and edits attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

runner cleanup is complete while Gemini and project files remain4%9/9100%

runner cleanup is complete while Gemini and project files remain attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

downstream lint fixture exercises the generated capability1%0/90%

downstream lint fixture exercises the generated capability attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 failed Attempt 2 failed Attempt 3 failed0/3
gpt-5.6-solopenai	Attempt 1 failed Attempt 2 failed Attempt 3 failed0/3
v9m-rl-learnability-tp8openai	Attempt 1 failed Attempt 2 failed Attempt 3 failed0/3

repository smoke detects broken runner contracts3%9/9100%

repository smoke detects broken runner contracts attempt outcomes by model configuration
Model configuration	Attempts
gpt-5.6-lunaopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
gpt-5.6-solopenai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3
v9m-rl-learnability-tp8openai	Attempt 1 passed Attempt 2 passed Attempt 3 passed3/3

Published evidence

Individual attempts

01v9m-rl-learnability-tp8partial0.9640m2026-08-05__12-16-53 02v9m-rl-learnability-tp8partial0.9643m2026-08-05__12-16-53 03v9m-rl-learnability-tp8partial0.9650m2026-08-05__12-16-53 04gpt-5.6-solpartial0.5811m2026-08-05__12-16-53 05gpt-5.6-solpartial0.9615m2026-08-05__12-16-53 06gpt-5.6-solpartial0.9915m2026-08-05__12-16-53 07gpt-5.6-lunapartial0.943m2026-08-05__12-16-53 08gpt-5.6-lunapartial0.944m2026-08-05__12-16-53 09gpt-5.6-lunapartial0.914m2026-08-05__12-16-53