Break the Agent

Goal

Pick the world we build. This is the highest-leverage choice of the weekend: 60 of 100 points (gap + meaningful difficulty) are decided by whether the setting produces a real, explainable gap.

Done when

• ☐ 3 to 5 candidate settings compared against the criteria below
• ☐ Cheap evidence for the favourite: a quick manual or scripted probe showing a frontier model actually struggles
• ☐ Decision recorded with new decision, and coord/project.md Concept section rewritten

Notes

Criteria for a good setting:

• A frontier agent plausibly fails for a reason we can name (planning depth, exploration, resource limits, adapting to new information), not because of missing information or awkward tooling.
• A reference run that is clearly stronger is feasible for us to build and reproduces exactly (a solver, a search, a scripted expert policy).
• The reward can be computed from real environment state, so it is hard to fake; doing nothing and stopping early naturally score about zero.
• Episodes are short and cheap enough to run many times in 48 hours.
• Rules make sense for the setting and fit in a short description.
• Bonus: produces experience a small model could learn from (stretch goal).

Ask the on-site mentors for a design review before committing.

Goal

Have Harbor + the OpenCode agent talk over MCP to a trivial placeholder environment on our machines, using the provided scoring skeleton. Everything else builds on this loop, so it comes first.

Done when

• ☐ Starter kit / harness docs are in the repo or linked from coord/project.md
• ☐ A toy MCP environment (one tool, trivial reward) runs under the harness and produces a score
• ☐ Exact setup steps and commands are written down so every teammate can repeat them
• ☐ Model, credits budget and rate limits are noted in coord/project.md

Notes

Write down early what the harness expects from an environment (MCP tool schema, how reward is reported, how an episode ends, how do-nothing and stop-early baselines are run). env-spec depends on those answers.

Goal

The two baselines the judges will run: an agent that takes no action, and one that stops after the first few steps. Both must score little or nothing.

Done when

• ☐ Both baselines runnable through the same entry point as the real evaluation
• ☐ Scores recorded in an eval experiment
• ☐ If either scores meaningfully above zero, a task is opened to fix the reward

Notes

Goal

A rehearsed 10-minute walkthrough: why this setting, the environment live, the gap, the agent's mistakes with evidence, and why the reward holds up.

Done when

• ☐ Outline with time per part
• ☐ One recorded agent failure ready to show, plus the reference run succeeding
• ☐ Rehearsed once end to end
• ☐ Fallback if live runs are slow: pre-recorded outputs

Notes

Goal

The environment itself: world or simulator logic, the MCP tools the agent calls, deterministic seeding, and a trajectory log of every observation, action and reward.

Done when

• ☐ Runs under the Harbor + OpenCode harness
• ☐ Deterministic for a fixed seed
• ☐ Invalid, out-of-range and out-of-order tool calls are handled with clear errors, never crashes or free reward
• ☐ Every episode writes a trajectory log to runs/
• ☐ Unit tests for the world rules

Notes

Goal

A short written spec of the environment that the implementation, the reference run and the evaluation all build against, so they can proceed in parallel.

Done when

• ☐ Goal stated in one or two sentences
• ☐ Action space: the MCP tools, arguments, and what each does
• ☐ Observation space: what the agent sees at the start and after each action
• ☐ Reward: formula, range, when it is given, and why do-nothing and stop-early score near zero
• ☐ Episode start, termination, step or resource limits, seeding
• ☐ What the agent is told up front (the task prompt) - checked for fairness: nothing needed to succeed is hidden

Notes

Put the spec in docs/env-spec.md and record contested choices as decisions. Interface changes after this point must be announced with a message to all.

Goal

A written analysis of where and why the frontier agent falls short: each main mistake tied to a decision in the task, with evidence that the information and tools to do better were available.

Done when

• ☐ 2 to 3 main failure modes, each with quoted evidence from trajectories
• ☐ Each traced to a decision point and its cost in score
• ☐ Fairness check passed: no failure caused by hidden information, wording, timing or broken tools
• ☐ Written up in docs/failure-analysis.md and summarised in the README

Notes

Use the trajectory-analyst subagent on the run logs.

Goal

Run OpenCode with the frontier model through the harness on the real environment, repeatedly, and record every score next to the reference run.

Done when

• ☐ At least 3 runs with the judging configuration, trajectories kept
• ☐ Registered as a --kind eval experiment with reference, agent, noop, stop_early filled in
• ☐ The gap is consistent across runs (if not, say so and open a task)

Notes

Watch the credit budget; note cost per run in the experiment file.

Goal

A single command in the README that runs the reference run, three agent runs, do-nothing and stop-early with the standard harness and prints the scores we report.

Done when

• ☐ Works from a fresh clone on a machine that has never run it
• ☐ README: why this setting, what an agent can learn, rules, how to run, reported results
• ☐ The reported numbers match what the command prints

Notes

Start a skeleton of this command as soon as the environment runs, and keep it green on main.

Goal

A run that clearly outscores the frontier agent and that we can demonstrate live and reproduce: for example a solver, search, or scripted expert policy using only the same tools and information the agent has.

Done when

• ☐ Reproduces the same score across repeated runs and on a second teammate's machine
• ☐ Uses only actions and information available to the agent (otherwise the gap is not fair)
• ☐ We can explain in one minute why it does better

Notes

Goal

Scoring that measures success at the task and nothing else, computed from real environment state rather than anything the agent claims.

Done when

• ☐ Implemented against the spec, with tests for edge cases
• ☐ Do-nothing scores about zero; stop-early scores little
• ☐ Bounded: no term can accumulate without limit or be farmed
• ☐ Completing the task well is the highest-scoring strategy we know of

Notes

Any later change here means rerunning the baselines and the reward-redteam subagent.

Goal

Do the judges' 15-minute exploit hunt ourselves, repeatedly, until nothing that avoids the task beats honest play.

Done when

• ☐ reward-redteam subagent run against the current environment
• ☐ Findings recorded as a --kind redteam experiment
• ☐ Every confirmed exploit fixed and re-tested
• ☐ A second teammate (or their agent) who did not write the reward has tried to break it

Notes

Rerun after every reward or tool-surface change.

Goal

Show a smaller model improving after training on experience from the environment (for example imitation of reference trajectories or a simple RL loop). Only a tiebreaker: start it once the four main criteria are covered.

Done when

• ☐ Before and after scores on held-out seeds
• ☐ Training command and data generation are reproducible

Notes