InfraBench

nplus1 × Zendo

hard · software-engineering

OpenAI provisioning for n1x

Add composable OpenAI runtime and Terraform provisioning capabilities to n1x.

Task version0.1.0
Checksumf1578cef5d2c
Published attempts9
Full reward1.00

Current task results

Evaluation comparison

Each bar averages that model configuration's verifier rewards for this task. The latest published batch 2026-08-05__12-16-56 contains 9 attempts across 3 model configurations.

  1. gpt-5.6-solopenai
    72.3%± 4.0 pp SD
    3/3 scored attempts
  2. gpt-5.6-lunaopenai
    67.0%± 7.9 pp SD
    3/3 scored attempts
  3. v9m-rl-learnability-tp8openai
    65.0%± 1.7 pp SD
    3/3 scored attempts

± is sample standard deviation across scored attempts. The line spans minimum to maximum; the heavier marker is the mean. Fixed 0–100 scale.

Latest published cohort

Verifier criteria

Pass rate across 9 attempts from 3 model configurations in batch 2026-08-05__12-16-56. Reward weight is how much each check contributes to one attempt's total score. Expand a criterion to see attempt outcomes by model configuration.

core engine and unrelated capabilities are unchangedReward weight 4%Passed 9/9Pass rate 100%
core engine and unrelated capabilities are unchanged attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
web-openai resolves its dependency setReward weight 3%Passed 9/9Pass rate 100%
web-openai resolves its dependency set attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
web-openai adds OpenAI SDK 7.xReward weight 3%Passed 9/9Pass rate 100%
web-openai adds OpenAI SDK 7.x attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
standalone web-openai declares one runtime keyReward weight 3%Passed 9/9Pass rate 100%
standalone web-openai declares one runtime key attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
standalone credentials remain server-onlyReward weight 3%Passed 9/9Pass rate 100%
standalone credentials remain server-only attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
standalone guidance explains server-only useReward weight 2%Passed 9/9Pass rate 100%
standalone guidance explains server-only use attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
infra declares the official provider constraintReward weight 3%Passed 9/9Pass rate 100%
infra declares the official provider constraint attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
Terraform formats and validates when the registry is availableReward weight 4%Passed 9/9Pass rate 100%
Terraform formats and validates when the registry is available attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
infra declares only the Terraform admin credentialReward weight 3%Passed 9/9Pass rate 100%
infra declares only the Terraform admin credential attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
openai plus infra does not create web outputReward weight 2%Passed 9/9Pass rate 100%
openai plus infra does not create web output attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
infra plans one app-named OpenAI project9 verifier/setupReward weight 5%Passed 0/9Pass rate 0%
infra plans one app-named OpenAI project attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
gpt-5.6-solopenaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
v9m-rl-learnability-tp8openaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
infra plans one app-named runtime service account9 verifier/setupReward weight 5%Passed 0/9Pass rate 0%
infra plans one app-named runtime service account attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
gpt-5.6-solopenaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
v9m-rl-learnability-tp8openaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
runtime identity receives the API project-member role9 verifier/setupReward weight 5%Passed 0/9Pass rate 0%
runtime identity receives the API project-member role attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
gpt-5.6-solopenaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
v9m-rl-learnability-tp8openaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
Terraform outputs expose both managed IDs9 verifier/setupReward weight 4%Passed 0/9Pass rate 0%
Terraform outputs expose both managed IDs attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
gpt-5.6-solopenaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
v9m-rl-learnability-tp8openaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
Terraform does not mint or expose runtime API keys9 verifier/setupReward weight 5%Passed 0/9Pass rate 0%
Terraform does not mint or expose runtime API keys attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
gpt-5.6-solopenaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
v9m-rl-learnability-tp8openaiAttempt 1 verifier or setup issueAttempt 2 verifier or setup issueAttempt 3 verifier or setup issue0/3
combined secret map is correct in plugin order 1Reward weight 3%Passed 9/9Pass rate 100%
combined secret map is correct in plugin order 1 attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
runtime workflow works in plugin order 1Reward weight 4%Passed 9/9Pass rate 100%
runtime workflow works in plugin order 1 attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
combined secret map is correct in plugin order 2Reward weight 3%Passed 9/9Pass rate 100%
combined secret map is correct in plugin order 2 attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
runtime workflow works in plugin order 2Reward weight 4%Passed 9/9Pass rate 100%
runtime workflow works in plugin order 2 attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
openai does not own web-openai behaviorReward weight 6%Passed 9/9Pass rate 100%
openai does not own web-openai behavior attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
plain web and infra remain isolated from OpenAIReward weight 6%Passed 4/9Pass rate 44%
plain web and infra remain isolated from OpenAI attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 failedAttempt 3 failed1/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 failedAttempt 2 failedAttempt 3 failed0/3
repeated apply is idempotentReward weight 5%Passed 9/9Pass rate 100%
repeated apply is idempotent attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
removing web-openai preserves infra provisioningReward weight 4%Passed 8/9Pass rate 89%
removing web-openai preserves infra provisioning attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 failedAttempt 3 passed2/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
removing openai cleans all managed outputReward weight 4%Passed 4/9Pass rate 44%
removing openai cleans all managed output attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 failedAttempt 3 failed1/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 failedAttempt 2 failedAttempt 3 failed0/3
README documents the operator credential boundaryReward weight 3%Passed 5/9Pass rate 56%
README documents the operator credential boundary attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 failed2/3
gpt-5.6-solopenaiAttempt 1 failedAttempt 2 failedAttempt 3 passed1/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 failed2/3
downstream lint fixture covers OpenAIReward weight 2%Passed 9/9Pass rate 100%
downstream lint fixture covers OpenAI attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
repository smoke regression passesReward weight 1%Passed 7/9Pass rate 78%
repository smoke regression passes attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 failedAttempt 3 failed1/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3
repository smoke detects broken OpenAI contractsReward weight 1%Passed 6/9Pass rate 67%
repository smoke detects broken OpenAI contracts attempt outcomes by model configuration
Model configurationAttempts
gpt-5.6-lunaopenaiAttempt 1 passedAttempt 2 failedAttempt 3 failed1/3
gpt-5.6-solopenaiAttempt 1 passedAttempt 2 failedAttempt 3 passed2/3
v9m-rl-learnability-tp8openaiAttempt 1 passedAttempt 2 passedAttempt 3 passed3/3

Published evidence

Individual attempts

01v9m-rl-learnability-tp8partial0.6625m2026-08-05__12-16-5602v9m-rl-learnability-tp8partial0.6622m2026-08-05__12-16-5603v9m-rl-learnability-tp8partial0.6320m2026-08-05__12-16-5604gpt-5.6-solpartial0.7313m2026-08-05__12-16-5605gpt-5.6-solpartial0.6812m2026-08-05__12-16-5606gpt-5.6-solpartial0.7611m2026-08-05__12-16-5607gpt-5.6-lunapartial0.766m2026-08-05__12-16-5608gpt-5.6-lunapartial0.647m2026-08-05__12-16-5609gpt-5.6-lunapartial0.615m2026-08-05__12-16-56