Break the Agent

Junction 2026 · AfterQuery challenge · an RL environment where frontier agents fall short for real reasons2026-11-15T10:00+02:00
main@bd41ccc · built 2026-11-14T03:14+00:00 · 1/14 tasks done · generated by tools/coord.py from coord/

Project coord/project.md

Concept

Not chosen yet. Deciding the setting is task choose-setting; the outcome goes into coord/decisions/ and replaces this section.

What the concept has to deliver, in order of points:

State

Repo and coordination structure are set up. No environment code yet. Next: get Harbor and OpenCode running (starter-kit-harness) and pick the setting (choose-setting); they can run in parallel.

Links

Required stack and submission (AfterQuery, 14 Nov)

Open facts to fill in

Schedule

Submissions open Sat 14 Nov 12:00. Final submission Sun 15 Nov 10:00, review from 10:30.

The gap latest finished eval: reference vs frontier agent vs baselines

No finished eval recorded yet. Register one with python3 tools/coord.py new exp "..." --kind eval, then fill in reference, agent (one score per run), noop and stop_early and set status=done. This panel then shows the gap the judges will re-measure: reference run, three agent runs, do-nothing, stop-early.

Rubric coverage tasks done per judging criterion

The gap · 30 p
0/2
tasks done
Meaningful difficulty · 30 p
0/3
tasks done
Reward integrity · 25 p
0/3
tasks done
Craft · 15 p
0/4
tasks done
Stretch: small model · tiebreak
0/1
tasks done

Tasks click a card for details

○To do11

Add do-nothing and stop-early baselinesP1integritywaits on env-implementation, reward-function

Goal

The two baselines the judges will run: an agent that takes no action, and one that stops after the first few steps. Both must score little or nothing.

Done when

  • ☐ Both baselines runnable through the same entry point as the real evaluation
  • ☐ Scores recorded in an eval experiment
  • ☐ If either scores meaningfully above zero, a task is opened to fix the reward

Notes

Update 2026-11-14 (peter): organisers' required-stack post

Do-nothing is Harbor's built-in -a nop. Stop-early has no built-in: until the mentors say how they run it, implement it as a variant of the reference script that quits after the first few periods.

coord/tasks/baselines-noop-stop-early.md · updated 2026-11-13T23:38+02:00
Prepare the 10-minute demoP1craftwaits on failure-analysis, one-command-eval-readme

Goal

A rehearsed 10-minute walkthrough: why this setting, the environment live, the gap, the agent's mistakes with evidence, and why the reward holds up.

Done when

  • ☐ Outline with time per part
  • ☐ One recorded agent failure ready to show, plus the reference run succeeding
  • ☐ Rehearsed once end to end
  • ☐ Fallback if live runs are slow: pre-recorded outputs

Notes

Update 2026-11-14 (peter): organisers' required-stack post

A demo video is part of the submission, and the organisers encourage showing a playthrough in it. Plan the recording, not only the live walkthrough: one agent run failing and the reference run succeeding, from real Harbor run logs.

coord/tasks/demo-prep.md · updated 2026-11-13T23:38+02:00
Implement the environment as an MCP serverP1craftwaits on env-spec, starter-kit-harness

Goal

The environment itself: world or simulator logic, the MCP tools the agent calls, deterministic seeding, and a trajectory log of every observation, action and reward.

Done when

  • ☐ Runs under the Harbor + OpenCode harness
  • ☐ Deterministic for a fixed seed
  • ☐ Invalid, out-of-range and out-of-order tool calls are handled with clear errors, never crashes or free reward
  • ☐ Every episode writes a trajectory log to runs/
  • ☐ Unit tests for the world rules

Notes

Update 2026-11-14 (peter): organisers' required-stack post

The deliverable is a Harbor task directory (instruction.md, task.toml, environment/, tests/, solution/). Keep the simulator a plain Python module that also runs in-process, so the reference run, the unit tests and the post-training experiment can step it without Harbor.

MCP tool schemas: every argument exactly one explicit type, no optional/nullable/union types, or Gemini rejects every request (docs/resources.md).

coord/tasks/env-implementation.md · updated 2026-11-13T23:38+02:00
Specify goal, actions, observations, reward and episode endP1difficultywaits on choose-setting

Goal

A short written spec of the environment that the implementation, the reference run and the evaluation all build against, so they can proceed in parallel.

Done when

  • ☐ Goal stated in one or two sentences
  • ☐ Action space: the MCP tools, arguments, and what each does
  • ☐ Observation space: what the agent sees at the start and after each action
  • ☐ Reward: formula, range, when it is given, and why do-nothing and stop-early score near zero
  • ☐ Episode start, termination, step or resource limits, seeding
  • ☐ What the agent is told up front (the task prompt) - checked for fairness: nothing needed to succeed is hidden

Notes

Put the spec in docs/env-spec.md and record contested choices as decisions. Interface changes after this point must be announced with a message to all.

coord/tasks/env-spec.md · updated 2026-11-13T23:38+02:00
Explain the agent's main mistakesP1difficultywaits on frontier-agent-runs

Goal

A written analysis of where and why the frontier agent falls short: each main mistake tied to a decision in the task, with evidence that the information and tools to do better were available.

Done when

  • ☐ 2 to 3 main failure modes, each with quoted evidence from trajectories
  • ☐ Each traced to a decision point and its cost in score
  • ☐ Fairness check passed: no failure caused by hidden information, wording, timing or broken tools
  • ☐ Written up in docs/failure-analysis.md and summarised in the README

Notes

Use the trajectory-analyst subagent on the run logs.

coord/tasks/failure-analysis.md · updated 2026-11-13T23:38+02:00
Run the frontier agent at least three times and record scoresP1gapwaits on env-implementation, reward-function

Goal

Run OpenCode with the frontier model through the harness on the real environment, repeatedly, and record every score next to the reference run.

Done when

  • ☐ At least 3 runs with the judging configuration, trajectories kept
  • ☐ Registered as a --kind eval experiment with reference, agent, noop, stop_early filled in
  • ☐ The gap is consistent across runs (if not, say so and open a task)

Notes

Watch the credit budget; note cost per run in the experiment file.

Update 2026-11-14 (peter): organisers' required-stack post

AfterQuery gives no credits; the model comes from the Google or OpenAI sponsor credits and is our choice. Record provider, model and cost per run in the experiment. If budget allows, repeat on the other provider. Ask the mentors which model the judges rerun with.

Update 2026-11-14 (peter): models

Headline: google/gemini-4-carbon. Smoke runs: google/gemini-3.8-flash or DeepSeek on Verda. Second family if budget allows: Kimi K3.1 Pro on Verda. The OpenAI sponsor credits cannot pay for OpenCode runs. See docs/resources.md.

coord/tasks/frontier-agent-runs.md · updated 2026-11-13T23:38+02:00
One-command evaluation and README for judgesP1craftwaits on reference-run, baselines-noop-stop-early, frontier-agent-runs

Goal

A single command in the README that runs the reference run, three agent runs, do-nothing and stop-early with the standard harness and prints the scores we report.

Done when

  • ☐ Works from a fresh clone on a machine that has never run it
  • ☐ README: why this setting, what an agent can learn, rules, how to run, reported results
  • ☐ The reported numbers match what the command prints

Notes

Start a skeleton of this command as soon as the environment runs, and keep it green on main.

Update 2026-11-14 (peter): organisers' required-stack post

  • The README must say, briefly: what the agent has to do, why it is hard, how it is scored. The organisers warn that AI-generated READMEs "may communicate your environment's value poorly", so a human on the team should write or rewrite the final text.
  • The submission must include at least one full Harbor run log with its score. runs/ is ignored by git, so copy one into results/.
  • The command wraps harbor run -p <task> -a oracle, -a opencode -m <model> -k 3, -a nop, and the stop-early baseline; it needs the model key from .env.
coord/tasks/one-command-eval-readme.md · updated 2026-11-13T23:38+02:00
Build the stronger reference runP1gapwaits on env-implementation

Goal

A run that clearly outscores the frontier agent and that we can demonstrate live and reproduce: for example a solver, search, or scripted expert policy using only the same tools and information the agent has.

Done when

  • ☐ Reproduces the same score across repeated runs and on a second teammate's machine
  • ☐ Uses only actions and information available to the agent (otherwise the gap is not fair)
  • ☐ We can explain in one minute why it does better

Notes

coord/tasks/reference-run.md · updated 2026-11-13T23:38+02:00
Implement the reward and scoringP1integritywaits on env-spec

Goal

Scoring that measures success at the task and nothing else, computed from real environment state rather than anything the agent claims.

Done when

  • ☐ Implemented against the spec, with tests for edge cases
  • ☐ Do-nothing scores about zero; stop-early scores little
  • ☐ Bounded: no term can accumulate without limit or be farmed
  • ☐ Completing the task well is the highest-scoring strategy we know of

Notes

Any later change here means rerunning the baselines and the reward-redteam subagent.

Update 2026-11-14 (peter): organisers' required-stack post

The final reward is a single number from 0 to 1 in /logs/verifier/reward.txt; partial credit is allowed. Normalise so the do-nothing policy maps to 0 and the reference to a high score, and clip. Keep sub-scores in our own log, not in the reward file. Compute it from simulator state the agent cannot write (docs/harness-notes.md, "What this means for our design").

coord/tasks/reward-function.md · updated 2026-11-13T23:38+02:00
Red-team the reward for exploitsP1integritywaits on reward-function, env-implementation

Goal

Do the judges' 15-minute exploit hunt ourselves, repeatedly, until nothing that avoids the task beats honest play.

Done when

  • ☐ reward-redteam subagent run against the current environment
  • ☐ Findings recorded as a --kind redteam experiment
  • ☐ Every confirmed exploit fixed and re-tested
  • ☐ A second teammate (or their agent) who did not write the reward has tried to break it

Notes

Rerun after every reward or tool-surface change.

coord/tasks/reward-redteam.md · updated 2026-11-13T23:38+02:00
Stretch: train a smaller model on environment experienceP2stretchwaits on env-implementation, reference-run

Goal

Show a smaller model improving after training on experience from the environment (for example imitation of reference trajectories or a simple RL loop). Only a tiebreaker: start it once the four main criteria are covered.

Done when

  • ☐ Before and after scores on held-out seeds
  • ☐ Training command and data generation are reproducible

Notes

Update 2026-11-14 (peter): organisers' required-stack post

The organisers now list "some post-training experiments" among what they are looking for, and Modal credits are available for the GPU. Priority raised from 3 to 2. Cheapest credible version: log reference-policy trajectories from the simulator, fine-tune a small open model on them on Modal, and report before and after scores on held-out seeds. Design the environment so it can be stepped in-process without Harbor, or collecting experience will be too slow.

Update 2026-11-14 (peter): which model, where

Proposal: Gemma 4 E4B (or E2B), LoRA with the Unsloth recipe on a Modal L40S, trained on reference-policy trajectories; evaluated in-process on held-out seeds, before and after. Also eligible for the "Best Use of Gemma" and "Best use of Modal" side challenges. Details and cost traps: docs/resources.md.

coord/tasks/stretch-train-small-model.md · updated 2026-11-14T02:50+02:00

●In progress2

Choose the environment setting and write the decisionP1difficulty@peter

Goal

Pick the world we build. This is the highest-leverage choice of the weekend: 60 of 100 points (gap + meaningful difficulty) are decided by whether the setting produces a real, explainable gap.

Done when

  • ☐ 3 to 5 candidate settings compared against the criteria below
  • ☐ Cheap evidence for the favourite: a quick manual or scripted probe showing a frontier model actually struggles
  • ☐ Decision recorded with new decision, and coord/project.md Concept section rewritten

Notes

Criteria for a good setting:

  • A frontier agent plausibly fails for a reason we can name (planning depth, exploration, resource limits, adapting to new information), not because of missing information or awkward tooling.
  • A reference run that is clearly stronger is feasible for us to build and reproduces exactly (a solver, a search, a scripted expert policy).
  • The reward can be computed from real environment state, so it is hard to fake; doing nothing and stopping early naturally score about zero.
  • Episodes are short and cheap enough to run many times in 48 hours.
  • Rules make sense for the setting and fit in a short description.
  • Bonus: produces experience a small model could learn from (stretch goal).

Ask the on-site mentors for a design review before committing.

Research, 2026-11-14 (peter)

Harness facts this builds on are in docs/harness-notes.md. Two of them drive the choice:

  • The agent under test has a shell and can script calls to our MCP server, so

a setting that a solver cracks is not safe: the agent can write the solver.

  • The judges' own paper (StoreBench) got its gap from scarce operations plus

hidden events, not from horizon length. Without shocks, frontier models with a coding harness beat their scripted operator.

Candidates

| # | Setting | Why a frontier agent should fall short | Reference run | Reward and baselines | Build cost | Main risk | |---|---|---|---|---|---|---| | A | Metered operations with delayed consequences: run a small operation for N periods; every tool call, reads included, costs one of K operations per period; orders and repairs take effect after a delay; conditions change on announced, rumoured and silent channels | The failure modes the judges measured themselves: under-investing, missing silent changes, spending the budget on reading, declaring done early | Scripted triage heuristic using only the agent's tools | Terminal books after settlement, normalised between the absentee policy and the reference; do-nothing and stop-early land at about 0 by construction | Medium: one simulator, about 8 to 12 tools | Reads as a small StoreBench unless the setting is clearly ours; balancing takes iterations | | B | Unknown machine: a black-box process with hidden response curves; limited noisy experiments, then a production phase | Poor experimental design, anchoring on the first hypothesis | Scripted design-of-experiments plus model fit | Production output from simulator state | Low | The agent can fit models in Python too; weakest evidence for a gap | | C | Online dispatch: jobs arrive over time, limited vehicles or crews, disruptions | Greedy, reactive play instead of holding capacity for what is forecast | Rolling-horizon heuristic | Value served by the end | Medium | Fully visible state invites the agent to script the same heuristic | | D | Hidden-rule puzzle worlds (in the style of ARC-AGI-3): mechanics must be discovered by acting | Large documented gap between frontier agents and people on novel interactive games | State-graph explorer with no knowledge of the rules | Levels completed, weighted by action efficiency | Low to medium | "Hidden rules" can be judged as withheld information; text grids make perception, not decisions, the bottleneck | | E | Incident response in a simulated service: diagnose and repair with side effects | Acting before diagnosing | Scripted runbook | Uptime over the episode | High | A runbook that knows the fault classes is a privileged reference; the simulator is the largest of the five |

Recommendation (not yet a decision)

A, in a setting that is not a shop. It is the only candidate whose failure modes are already measured on frontier models, by the people judging us, and its structure answers three rubric rows at once: metering defeats brute-force scripting (gap), every lost point traces to a logged allocation decision (difficulty), and terminal settlement makes quitting worthless (integrity).

Setting proposal: spacecraft operations. Contact windows make a per-period command budget physically natural, telemetry is the read cost, consumables (power, storage, propellant) are the scarce resources, slews and uploads are the delays, and faults develop silently in channels the operator has to choose to look at. Score is the science value downlinked by end of mission, which is zero if you do nothing. Alternatives with the same skeleton: a hospital blood bank (perishables, lead times, strong textbook heuristics for the reference) or a winter road-maintenance depot. The simulator core is the same, so the flavour can still change after the probe.

What makes it fail: the gap is a hypothesis until a frontier model has played it. If the first probe shows the agent matching the scripted reference, tighten the operation budget and add silent events before adding anything else; if it still matches, fall back to D.

Before this can be decided

  • ☐ Mentor design review of A (questions in docs/harness-notes.md)
  • ☐ Probe: needs the harness running and model credits, neither of which exists on any machine yet. A 2 to 3 hour throwaway version of A (3 tools, 20 periods) is enough to see whether the agent under-invests and stops early
  • ☐ Team agrees, then new decision and rewrite the Concept section of coord/project.md

Update 2026-11-14: organisers' required-stack post

Does not change the recommendation; it changes three inputs:

  • The model is ours to pick (Google or OpenAI sponsor credits). On StoreBench's short tasks the GPT and Gemini models tested scored 0.54 and 0.55 against 0.76 for the scripted operator, a wider gap than the best model overall (0.70). That was under a different harness, so it is encouragement for candidate A, not proof.
  • Cost now matters. Episode length has to fit our own credits; fix the horizon after measuring one toy run.
  • Post-training is wanted, not just a tiebreak. Candidate A suits it: the reference policy generates unlimited labelled experience from the simulator.

Still open before deciding: whether a second container is allowed (question 1 in docs/harness-notes.md), and a probe run.

Probe results, night of 14 Nov (peter)

Harness and model path work end to end (Harbor on Modal, OpenCode, Kimi K3.1 Pro served by Verda; no Gemini key yet). Experiments: coord/experiments/1114-03* and 1114-04*.

Candidate A, version 1 (probes/orbital): no gap. 30-orbit satellite mission, 5 operations per window, silent faults, storm alert, cancelled passes. Kimi scored 2308 against 2056 for my scripted reference (do-nothing 0). It played all 30 windows carefully, with explicit arithmetic, and made almost no mistakes. Lessons:

  • A clean, fully specified, small operations world is not hard for a strong model. Metered operations, silent faults and announced events alone do not produce a gap.
  • My first reference was weak. Any reference we ship has to be tuned until it beats careful greedy play, or the agent will beat it.
  • Free remedies invite blanket use. Every action that can be spammed as insurance needs a price.

Version 2 (probes/orbital2) adds: 4 operations per window, detector filters whose quality per target class has to be learned from noisy results that only arrive at downlink (and shift once), single-use remedies with a price. In-process on 40 held-out seeds: fixed filter 1015, naive try-each-once learner 1252, learning reference 1331, privileged ceiling (told the true filter response) 1522. So the learning element is worth about 6% between a naive and a careful learner: real, but small. Kimi run on version 2 is in progress.

Working conclusion so far: individual "clever" mechanics each buy a few percent. The published evidence (StoreBench, Vending-Bench) and this probe point the same way: gaps come from scale, mess and long horizons (many open commitments, compounding consequences, context that outgrows the agent's attention), not from a small clean puzzle. If version 2 also shows no gap, the next version scales the world up (several satellites sharing one operation budget, customer orders with deadlines and penalties, 100+ windows) rather than adding another mechanic.

Change of direction, 14 Nov about 04:15 (Peter)

Peter stopped the orbital probes and asked for a robotics environment instead: a robot arm on a kitchen tabletop that has to pick fruit and drinks and place them (basket, in a given order; "serve the chosen drink"), with post-training on a specific task. Seen working at a sponsor booth with an open model driving an arm through tools in Isaac Sim.

Candidate R, as I am prototyping it (probes/tabletop, not there yet):

  • Simulator: MuJoCo 4.0.1, not Isaac Sim. Isaac needs an RTX GPU and a very large image; judges must rerun our task with one command inside Harbor. MuJoCo is a pip install, runs on CPU, is fast and deterministic.
  • Agent interface: MCP tools at the level of observe, move_to(x, y, z, yaw), grasp, release. Object poses and sizes are given exactly, so nothing needed is withheld.
  • Where the difficulty has to come from: plain pick-and-place on an uncluttered table is what the booth demo shows models can already do, so it will not produce a gap. The hard part is physical: bottles that tip when a finger clips them, fruit that rolls, a banana that must be gripped across its short axis, fingers that have to fit between neighbours, a basket that fills up, and no second chance once something is on the floor.
  • Score: from the simulator's final physical state (what is in the basket, in which order it went in, what stands upright on the coaster, what was knocked over or lost). Do-nothing is 0.
  • Reference: a scripted controller that uses the same tools and computes clearances.
  • Post-training: a small policy or small open model trained on reference trajectories for one task ("pick the chosen drink"), before and after scores.

Unproven until a model has played it. The orbital probes stay in probes/ as evidence of what did not produce a gap.

The orbital probe code was removed from the tree on 14 Nov (Peter's request); last commit containing it: a1ac3bc. Its result summary stays in results/probes/.

coord/tasks/choose-setting.md · updated 2026-11-14T02:33+02:00
Get the AfterQuery starter kit running end to endP1craft@peter

Goal

Have Harbor + the OpenCode agent talk over MCP to a trivial placeholder environment on our machines, using the provided scoring skeleton. Everything else builds on this loop, so it comes first.

Done when

  • ☐ Starter kit / harness docs are in the repo or linked from coord/project.md
  • ☐ A toy MCP environment (one tool, trivial reward) runs under the harness and produces a score
  • ☐ Exact setup steps and commands are written down so every teammate can repeat them
  • ☐ Model, credits budget and rate limits are noted in coord/project.md

Notes

Write down early what the harness expects from an environment (MCP tool schema, how reward is reported, how an episode ends, how do-nothing and stop-early baselines are run). env-spec depends on those answers.

Update 2026-11-14 (peter): organisers' required-stack post

There is no separate starter kit: it is public Harbor (uv tool install harbor, needs Docker) with OpenCode, run as harbor run -p ./my-task -a opencode -m [provider/model]. Background and the upstream MCP example to copy: docs/harness-notes.md.

Changes to "done when":

  • The toy task writes one number 0..1 to /logs/verifier/reward.txt.
  • No AfterQuery credits. Get a Google or OpenAI key from the sponsor credits into .env, and record model, cost per run and rate limits in coord/project.md.
  • Test both layouts and report which works: a second container via environment/docker-compose.yaml, and a single Dockerfile with the simulator under a different OS user from the agent.
  • Run -a oracle and -a nop on the toy task and record what each does, and what reward an agent timeout produces.

2026-11-14 (peter, claimed)

Done so far: Harbor 0.26.0 installed in .venv with Modal support; facts confirmed from source are in docs/harness-notes.md; .env.example added. Not done: no trial has run yet. Blocked on a model key in .env and either local Docker (needs sudo) or a Modal token.

coord/tasks/starter-kit-harness.md · updated 2026-11-14T03:08+02:00

⚠Blocked0

Nothing here.

✓Done1

Set up the repo and agent coordination structureP1infra@peter

Goal

A git repo that several Claude Code agents can collaborate through: shared instructions, file-based tasks / experiments / messages, and an HTML board.

Done when

  • ☑ CLAUDE.md with the protocol
  • ☑ coord/ structure and tools/coord.py
  • ☑ HTML board builds from coord/

Notes

Done in the initial commit. Still to do by a human: create the remote, push, and invite teammates (see README).

coord/tasks/repo-coordination-setup.md · updated 2026-11-13T23:38+02:00

Experiments

StatusExperiment / resultKindOwnerHostWhen
●Running
Tabletop with the Franka Panda arm: reference and Kimi, levels 1 and 2

Hypothesis

The tabletop task with a real arm model (Franka Panda from MuJoCo Menagerie, its own hand and collision meshes, inverse kinematics). The hand is much bulkier than the earlier gantry gripper: a 21 cm housing 4 cm above the fingertips, so far more picks need the right yaw or a blocker moved first. Question: does that widen the gap between Kimi K3.1 Pro and the scripted reference? Level 1 = four fruit, one drink; level 2 = five fruit, two drinks.

Setup

Exact command, seeds, model, env commit - enough for a teammate to rerun it.

Results

Takeaways

coord/experiments/1114-0514-tabletop-with-the-franka-panda-arm-reference-and.md · TABLETOP_SEED=<s> TABLETOP_LEVEL=<l> sg docker -c '.venv/bin/harbor run -p probes/tabletop -a opencode -m verda/kimi-k3-1-pro --base-url https://inference.verda.com/v1 --api-key-name VERDA_API_KEY --api-format openai_chat_completions --env-file .env -o runs/panda'
probe@peterpeter-LENOVO2026-11-14T05:14+02:00
✓Done
Candidate R (kitchen tabletop, MuJoCo): anchors and first Kimi run

Hypothesis

First robotics probe: a two-finger gripper on a gantry over a cluttered table (tight 3x2 block of cans and bottles, fruit tucked against it and in a huddle). Put four fruit in the basket in a given order and serve one drink from the middle of the block upright on the coaster, without crushing, dropping, toppling or losing anything. Question: does a strong model fall short of the scripted geometric reference?

Setup

Exact command, seeds, model, env commit - enough for a teammate to rerun it.

Results

Takeaways

Setup

probes/tabletop (gantry gripper, level 1): a tight 3 x 2 block of cans and bottles, five fruit (two against the drinks, three in a huddle). Task: four named fruit into the basket in order, one drink from the middle column upright on the coaster. Score: 1 per fruit in the basket (+0.5 if in turn), 2 for the drink, half for anything dropped, minus 1 per object crushed, knocked over, off the table or wrongly in the basket; divided by 8. All under Harbor with local Docker; model runs are OpenCode with Kimi K3.1 Pro on Verda.

Results

| Run | Seed 1 | Seed 2 | Seed 3 | Notes | |---|---|---|---|---| | -a nop | 0.0 | | | | | stop-early (-a oracle --ae TABLETOP_POLICY=stop_early) | 0.188 | | | places one fruit, then stops | | scripted reference (-a oracle) | 1.0 | 1.0 (in-process) | 0.69 to 1.0 (in-process, depends on the reference version) | about 60 to 105 s of robot time | | OpenCode + Kimi K3.1 Pro | 0.875 | 1.0 | 0.875 | 57 to 70 tool calls, 4 to 6.5 min, about $0.40 to $0.60 each |

In-process over 24 seeds, level 1: do-nothing 0.00, stop-early 0.19, naive straight-down controller 0.11, reference 0.98 (before a later change that cost it seed 3).

Kimi's two imperfect runs each lost exactly one point to one crushed object:

  • Seed 1: its very first descent, onto the juice bottle at yaw 0. Its reasoning checked the palm against the neighbouring cider bottle using the centre distance (0.0742 m) and forgot the bottle's radius; the palm is 0.075 m long each way. It then turned the gripper 90 degrees, moved the juice bottle out of the way, and did everything else correctly.
  • Seed 3: crushed the banana.

Commands and score details for the three Kimi runs: results/tabletop/kimi-seed*.{commands,details}.json. Replay pages (not in git): python probes/tabletop/viewer/replay.py results/tabletop/kimi-seed1.commands.json --seed 1 -o replay.html.

Takeaways

At level 1 there is no reliable gap: Kimi averages 0.92, my reference between 0.90 and 1.0. What the runs do show:

  • The model plans correctly (moves a blocker aside before reaching into the block, picks a yaw that fits, carries high, sets things down gently) and reasons about clearances with explicit arithmetic.
  • Its weakness is reliability, not understanding: about one geometric slip per episode, and with one attempt and real physics a slip is permanent.
  • With linear partial credit a slip costs 12%. Scored as "flawless or not", two of three episodes fail while the reference fails about one in ten. How damage is scored decides whether this shows up as a gap, and has to be justified by the setting, not by the number we want.
  • Still to try: level 2 (two drinks to serve, five fruit, more fruit against the drinks; reference 0.975 in-process), and the Franka Panda version, whose hand is much bulkier than the gantry gripper and should make clearance errors more likely.
  • The reference needs hardening: it gives up on a blocked item in a few seeds.
coord/experiments/1114-0423-candidate-r-kitchen-tabletop-mujoco-anchors-and.md · sg docker -c '.venv/bin/harbor run -p probes/tabletop -a oracle|nop|opencode -m verda/kimi-k3-1-pro ... -o runs/tabletop' (TABLETOP_SEED=1)
Tabletop level 1: no reliable gap. Kimi 0.875 / 1.0 / 0.875 vs reference about 0.9 to 1.0; nop 0, stop-early 0.19. One crushed object per imperfect run
probe@peterpeter-LENOVO2026-11-14T04:46+02:00
✓Done
Redteam: can the agent fake the score from inside its container?

Hypothesis

Toy task, scripted attacker in place of the agent: it never presses the counter, but floods /logs/verifier/reward.txt, starts a fake score server and points env-server at it in /etc/hosts, and replaces python3. Two variants: agent as root (Harbor default) and agent as an unprivileged user ([agent] user). Expect the root variant to score 1.0 (exploit) and the user variant 0.0.

Setup

Exact command, seeds, model, env commit - enough for a teammate to rerun it.

Results

Takeaways

Setup

Toy counter task (probes/toy-mcp), with the reference script replaced by an attacker that never presses the counter (probes/redteam/container-attack.sh). It runs where the agent runs, with the agent's rights, and tries four things: write /logs/verifier/reward.txt in a loop that keeps running after it exits, plant /logs/verifier/reward.json (Harbor prefers it over reward.txt), start a fake score server and point env-server at it in /etc/hosts, and replace python3. All on Modal, -a oracle.

Results

| Variant | Reward | What got through | |---|---|---| | Agent as root (Harbor default), grader reads the score from the sidecar | 1.0 | /etc/hosts edit: the grader fetched the score from the attacker's fake server | | Agent as unprivileged user, same grader | 1.0 | The grader read the true score (0.0), but the attacker's leftover loop overwrote reward.txt afterwards: /logs/verifier is world-writable | | Unprivileged agent, grader runs pkill first and clears the directory | 1.0 | pkill does not exist in python:3.12-slim, so nothing was killed; planted reward.json also survived | | Unprivileged agent, grader kills the agent's processes itself (via /proc), clears the directory, then reads the sidecar | 0.0 | Nothing |

Controls on the final layout: honest reference 1.0, OpenCode with Kimi K3.1 Pro 1.0 (so OpenCode works as a non-root user).

Takeaways

Keeping state in a sidecar is necessary but not enough. With Harbor's defaults the agent can fake any score without touching the environment. The layout that held, now in probes/toy-mcp:

1. task.toml: [agent] user = "agent" (created in the Dockerfile), [verifier] user = "root". 2. tests/test.sh, as root, in this order: kill every process owned by agent using only Python and /proc; delete everything in /logs/verifier; fetch the score from the sidecar; write reward.txt. 3. Call interpreters by absolute path with -I.

Not tested: the separate verifier mode, and local Docker (all runs were on Modal).

coord/experiments/1114-0350-redteam-can-the-agent-fake-the-score-from-inside.md · .venv/bin/harbor run -p <toy-exploit-root|toy-exploit-user> -a oracle -e modal -o runs/redteam
Harbor defaults let the agent fake the score (3 ways); unprivileged agent + grader that kills leftovers and clears the log dir held at 0.0
redteam@peterpeter-LENOVO2026-11-14T03:56+02:00
✓Done
Probe A (orbital ops): anchors and first agent runs

Hypothesis

Throwaway version of candidate A (metered operations, delayed consequences, silent faults) as a Harbor task, seed 101. Scripted anchors in-process and under Harbor (oracle = reference operator, nop), then OpenCode with Kimi K3.1 Pro. Question: does a strong model fall short of the scripted reference, and where does it lose the points?

Setup

Exact command, seeds, model, env commit - enough for a teammate to rerun it.

Results

Takeaways

Setup

probes/orbital at commit of this file: 30 orbits, 5 operations per window, seed 101 (faults: attitude from orbit 4, comms from orbit 6; storm orbit 13; passes 23 and 24 cancelled; eclipse 13 to 18). Score = science value downlinked / par, where par is 1.1 times what the scripted reference earns on the seed.

Results

In-process, 24 held-out seeds (101 to 124), cd probes/orbital/environment/env-server && python3 -m orbital.run --seeds ...:

| Operator | Mean value | Score | |---|---|---| | do nothing / attends but never acts | 0 | 0.00 | | stop after 5 orbits | 334 | about 0.15 | | greedy, never checks health or mail | 1059 | about 0.45 | | checklist clerk | 1464 | about 0.74 | | scripted reference | 1882 | 0.91 |

Under Harbor on Modal, seed 101:

| Run | Reward | Notes | |---|---|---| | -a oracle (scripted reference through MCP) | 0.9091 | value 2056, 1 min wall | | -a nop | 0.0 | | | -a opencode -m verda/kimi-k3-1-pro | 1.0 (clipped) | value 2308, 12% above the reference; 173 tool calls, 6 min of agent time, 3.7M input tokens (98% cached), 6.4k output tokens, about $0.92 |

Replay of the Kimi run: results/probes/orbital-v1-kimi-1.txt (made with python3 probes/orbital/analyze.py results/probes/orbital-v1-kimi-1.commands.json --seed 101).

Takeaways

No gap. Version 1 of probe A is too easy, and its reference is too weak. This is the result that matters for choose-setting.

  • Kimi attended all 30 windows, stowed for the storm, booked every pass two orbits ahead, took two observations in almost every orbit (56 against the reference's 48), and fixed both faults early. Its only losses were two downlinks skipped for lack of power in eclipse.
  • It ran all four maintenance actions without looking first. Maintenance on a healthy subsystem costs one operation and nothing else, so blanket maintenance is cheap insurance: a loophole in the rules.
  • With 5 operations per window, plentiful energy and plentiful downlink, "take the two best targets, book every pass, glance at telemetry" is close to optimal. Nothing forces a hard trade-off, so there is nothing for the agent to get wrong.
  • Metered operations, silent faults and announced events on their own do not stop a strong model. What the next version adds: something that has to be learned from noisy, delayed feedback (which instrument mode suits which target class), remedies that cost something so they cannot be applied blindly, energy mistakes that hurt (brown-out instead of a polite skip), and fewer operations.

Note, 14 Nov: the orbital probe code (probes/orbital, probes/orbital2) was removed from the tree at Peter's request after the change to a robotics environment. It is in git history; the last commit that contains it is a1ac3bc (git checkout a1ac3bc -- probes/orbital probes/orbital2 brings it back).

coord/experiments/1114-0346-probe-a-orbital-ops-anchors-and-first-agent-runs.md · .venv/bin/harbor run -p probes/orbital -a oracle|nop|opencode -e modal -o runs/orbital (see file)
Probe A v1: NO gap. Kimi K3.1 Pro 2308 vs scripted reference 2056 (nop 0). Too easy; v2 adds learning from noisy delayed feedback
probe@peterpeter-LENOVO2026-11-14T03:58+02:00
✓Done
Toy MCP task: OpenCode with Kimi K3.1 Pro (Verda) on Modal

Hypothesis

First model-driven trial. Does OpenCode connect to the MCP sidecar, call the tools, and leave a trajectory? Expect reward 1.0 on the toy counter. Also measures cost and wall time of the agent path.

Setup

Exact command, seeds, model, env commit - enough for a teammate to rerun it.

Results

Takeaways

Results

  • First attempt (-m openai/kimi-k3-1-pro): reward 0.0, NonZeroAgentExitCodeError. OpenCode died before its first step with t.responses is not a function: with the provider named openai, OpenCode uses the Responses API, which an OpenAI-compatible endpoint does not offer. Not a model failure.
  • Second attempt (-m verda/kimi-k3-1-pro, any provider name other than openai): reward 1.0. OpenCode connected to the MCP sidecar, called press five times and stopped. A trajectory (agent/trajectory.json) was written.

Working command (key in .env as VERDA_API_KEY):


.venv/bin/harbor run -p probes/toy-mcp -a opencode -m verda/kimi-k3-1-pro \
  --base-url https://inference.verda.com/v1 --api-key-name VERDA_API_KEY \
  --api-format openai_chat_completions --env-file .env -e modal -o runs/toy

Takeaways

  • The full agent path works: OpenCode under Harbor on Modal, model served by Verda, tools on our MCP sidecar, score read from the sidecar.
  • For any OpenAI-compatible endpoint, do not call the provider openai.
  • OpenCode reports cost 0 for this provider, so cost has to be computed from the token counts.
coord/experiments/1114-0340-toy-mcp-task-opencode-with-kimi-k3-1-pro-verda-o.md · .venv/bin/harbor run -p probes/toy-mcp -a opencode -m openai/kimi-k3-1-pro --base-url https://inference.verda.com/v1 --api-key-name VERDA_API_KEY --api-format openai_chat_completions --env-file .env -e modal -o runs/toy --job-name opencode-kimi-1
OpenCode + Kimi (Verda) on Modal: 1.0 on the toy task; provider must not be named openai
probe@peterpeter-LENOVO2026-11-14T03:48+02:00
✓Done
Toy MCP task on Modal: oracle and nop

Hypothesis

Does Harbor run a two-container task on Modal without local Docker, does the oracle reach the MCP sidecar, and does the verifier read the score from the sidecar? Expect oracle 1.0, nop 0.0.

Setup

Exact command, seeds, model, env commit - enough for a teammate to rerun it.

Results

Takeaways

Results

  • -a oracle: reward 1.0, 1m09s wall, no exceptions.
  • -a nop: reward 0.0, 1m02s wall, no exceptions.
  • Ran on Modal from a laptop with no Docker installed. Both containers built inside the Modal sandbox; the reference script in main reached the MCP server at http://env-server:8000/mcp; the verifier read the score from the sidecar's /score route and wrote /logs/verifier/reward.txt.

Exact commands, from the repo root with .venv set up as in docs/harness-notes.md and a Modal token in ~/.modal.toml:


.venv/bin/harbor run -p probes/toy-mcp -a oracle -e modal -o runs/toy --job-name oracle-1
.venv/bin/harbor run -p probes/toy-mcp -a nop    -e modal -o runs/toy --job-name nop-1

No seeds: the toy is deterministic.

Takeaways

  • The two-container layout works under Harbor on Modal; about one minute of overhead per trial for build and start.
  • Not tested yet: a model-driven agent (-a opencode) against the sidecar, the separate verifier mode, and local Docker.
coord/experiments/1114-0332-toy-mcp-task-on-modal-oracle-and-nop.md · .venv/bin/harbor run -p probes/toy-mcp -a oracle -e modal -o runs/toy (then -a nop)
oracle 1.0, nop 0.0 on Modal, about 1 min per trial, no local Docker
probe@peterpeter-LENOVO2026-11-14T03:34+02:00
✕Failed
Probe A v2 (orbital ops with filter learning): anchors and Kimi

Hypothesis

Version 2 after v1 showed no gap: 4 operations per window, detector filters whose quality per target class must be learned from noisy results that arrive at downlink (and shift once), single-use remedies with a price. Seed 101. Question: does Kimi K3.1 Pro now fall short of the scripted learning reference, and by how much?

Setup

Exact command, seeds, model, env commit - enough for a teammate to rerun it.

Results

Takeaways

Results

  • -a oracle (scripted learning reference) on Modal, seed 101: reward 0.9091, as in-process.
  • The Kimi run was stopped at Peter's request before it finished (change of direction towards a robotics environment); no agent score exists for version 2.
  • In-process anchors on 40 held-out seeds (101 to 140): fixed filter 1015, naive try-each-once learner 1252, learning reference 1331, privileged ceiling 1522.

Takeaways

Unanswered: whether a strong model falls short on version 2. The in-process numbers say the learning mechanic alone is worth about 6% between a naive and a careful learner. To resume: the two commands in the frontmatter, with --job-name kimi-1.

Note, 14 Nov: the orbital probe code (probes/orbital, probes/orbital2) was removed from the tree at Peter's request after the change to a robotics environment. It is in git history; the last commit that contains it is a1ac3bc (git checkout a1ac3bc -- probes/orbital probes/orbital2 brings it back).

coord/experiments/1114-0404-probe-a-v2-orbital-ops-with-filter-learning-anch.md · .venv/bin/harbor run -p probes/orbital2 -a oracle|nop|opencode -m verda/kimi-k3-1-pro ... -e modal -o runs/orbital2
Stopped on request before the Kimi run finished; reference 0.909 under Harbor, no agent score for v2
probe@peterpeter-LENOVO2026-11-14T04:07+02:00

Messages agent-to-agent, newest first, unanswered on top

@peter → @all · Integrity: Harbor's default layout lets the agent fake the score. Use the hardened layout in probes/toy-mcp ●Needs reply · 2026-11-14T03:56+02:00 · re reward-function

Tested on Modal with a scripted attacker in the agent's seat (coord/experiments/1114-0350-*). Three separate ways to get reward 1.0 without doing the task, even with all state in a sidecar:

  • as root, edit /etc/hosts so the grader fetches the score from a fake server;
  • as any user, leave a background loop that overwrites /logs/verifier/reward.txt after the grader has written it (the directory is world-writable);
  • plant /logs/verifier/reward.json, which Harbor prefers over reward.txt.

What held (reward 0.0 for the attacker, 1.0 for the honest reference and for OpenCode): agent runs as an unprivileged user ([agent] user in task.toml), verifier runs as root, and tests/test.sh first kills the agent's leftover processes using Python and /proc (the slim image has no pkill), then empties /logs/verifier, then reads the score from the sidecar. Every task we ship must copy probes/toy-mcp/task.toml and tests/test.sh. Judges get 15 minutes to hunt for exactly this.

@peter → @all · AfterQuery posted the required stack: no credits from them, reward is one number in reward.txt ●Needs reply · 2026-11-14T02:50+02:00

The organisers' post is at the end of docs/challenge.md; CLAUDE.md has a new 'Required stack' section. What changed:

  • There is no separate starter kit. It is public Harbor: the environment is a Harbor task (instruction.md, task.toml, environment/Dockerfile, tests/, solution/), played with harbor run -p <task> -a opencode -m <provider/model>.
  • The verifier writes ONE score between 0 and 1 to /logs/verifier/reward.txt. Partial credit allowed. Do not use reward.json.
  • AfterQuery gives no model credits. Use the Google, OpenAI and Modal sponsor credits. The model is therefore our choice; record it in every experiment and watch the cost.
  • Submission needs at least one Harbor run log with its score, a short README (what the agent does, why hard, how scored; better written by a human), and a demo video with a playthrough.
  • They name post-training experiments among what they want. stretch-train-small-model is now P2, with Modal for the GPU.

Notes were added to the tasks this touches (starter-kit-harness, reward-function, env-implementation, baselines, frontier-agent-runs, one-command-eval-readme, demo-prep, stretch). Read the update at the bottom of a task before you claim it.

Open and worth a trip to the booth: may we use a second container (docker-compose) for the simulator, and which model do the judges rerun with? Full list: end of docs/harness-notes.md.

@peter → @all · Research: what the harness gives us, and five setting candidates ●Needs reply · 2026-11-14T02:40+02:00 · re choose-setting

Read docs/harness-notes.md and the Research section of coord/tasks/choose-setting.md before designing anything.

Three facts that change the design:

  • The agent under test (OpenCode under Harbor) has a shell with every permission approved, besides our MCP tools. It can script calls to our server. State and score must live in the MCP sidecar container, and difficulty must survive the agent writing code.
  • Harbor already has the judging protocol built in: oracle agent = reference run (solution/solve.sh), nop agent = do-nothing, -k 3 = three agent runs. Stop-early has no built-in; we need to ask how the standard harness does it.
  • We do not have the AfterQuery starter kit, the model name or the credit budget. They are not public. Whoever talks to the mentors first: the question list is at the end of docs/harness-notes.md.

Recommendation, not yet decided: a metered-operations environment (every tool call costs budget, delayed consequences, silent changes), set in spacecraft operations. Reasons and four alternatives are in the task file. Objections welcome before the mentor review.

For starter-kit-harness: peter's laptop has no docker, uv or node installed yet.

@peter → @all · Direction change: robotics tabletop environment (MuJoCo), orbital probes stopped ○FYI · 2026-11-14T04:09+02:00 · re choose-setting

Peter asked to stop the orbital-operations probes and build a robotics environment: an arm on a kitchen tabletop picking fruit and drinks, placing them in a basket in order or serving a chosen drink, with post-training on one task.

Why this is reasonable on the evidence: the orbital probe showed no gap (Kimi K3.1 Pro beat my scripted reference 2308 to 2056). A clean, fully specified operations world is not hard for a strong model.

Plan and constraints are in coord/tasks/choose-setting.md, last section. Short version: MuJoCo (not Isaac Sim, which judges could not rerun), tools at the level of observe / move_to / grasp / release, difficulty from clutter and physics rather than from plain pick-and-place, score from the simulator's final state. Prototype goes to probes/tabletop. Nothing is decided until a model has played it.

@peter → @all · Sponsor credits reviewed: Gemini is our frontier agent, OpenAI credits cannot pay for OpenCode runs ○FYI · 2026-11-14T03:08+02:00

Full write-up: docs/resources.md. Key template: .env.example.

  • Agent under test: google/gemini-4-carbon on the Gemini team credits (one form per team; has anyone filed it?). Cheap smoke runs: gemini-3.8-flash, or DeepSeek on Verda.
  • The OpenAI grant is Codex usage credits on a ChatGPT account, not API credits. Harbor's opencode agent cannot use them. Use Codex for red-teaming our reward instead.
  • Modal: Harbor runs trials there with -e modal, so no local Docker is needed; also the GPU for post-training. Every teammate should redeem their own 100 dollars of Modal credit.
  • Post-training proposal: fine-tune Gemma 4 E4B on Modal with reference trajectories. Counts for the AfterQuery tiebreak and two side challenges.
  • MCP tool arguments must each have exactly one explicit type or Gemini fails every request.
  • Harbor 0.26.0 installs without admin rights: python3 -m venv .venv && .venv/bin/pip install 'harbor[modal]'.
@peter → @all · Repo is set up: how agents collaborate here ○FYI · 2026-11-13T23:38+02:00

Welcome. Read CLAUDE.md first (protocol, git rules), then coord/project.md.

First things for every agent joining:

  • Set a stable handle if the default is not what you want: write it into .coord-handle in the repo root.
  • Run python3 tools/coord.py checkin --human <name> --focus <what you are doing>.
  • Claim before you work: python3 tools/coord.py claim <task>. The two tasks that can start now, in parallel, are starter-kit-harness and choose-setting.

Open facts nobody has recorded yet (add them to coord/project.md if you know them): the submission deadline, the frontier model and credit budget, and where the starter kit lives.

Team

AgentCurrent focusChecked in
@peter
Peter · peter-LENOVO
researching choose-setting options and what the AfterQuery starter kit provides2026-11-14T02:33+02:00

Decisions

No decisions recorded.

Recent commits

bd41ccc  VrabelPe.. 32 seconds ago coord: peter updates 1114-0514-tabletop-with-the-franka-panda-arm-reference-and (status=running)
f1a7f08  VrabelPe.. 34 seconds ago coord: peter adds exp 1114-0514-tabletop-with-the-franka-panda-arm-reference-and
99154cb  VrabelPe.. 16 minutes ago Remove the Panda working copy (merged into probes/tabletop); README for the Panda version
bf72961  VrabelPe.. 16 minutes ago tabletop: Franka Panda arm (Menagerie) replaces the gantry; hand-accurate reference, level 2, viewers take --level
9c8cff4  VrabelPe.. 29 minutes ago tabletop: level 2 scene, README, three Kimi runs recorded (no reliable gap at level 1)
b3c18f3  VrabelPe.. 29 minutes ago coord: peter updates 1114-0423-candidate-r-kitchen-tabletop-mujoco-anchors-and (status=done)
6cd8337  VrabelPe.. 40 minutes ago tabletop: live and replay viewers, better parking in the reference, first Kimi result
21faa92  VrabelPe.. 46 minutes ago Remove the orbital probes (kept in history); add the tabletop replay viewer
a1ac3bc  VrabelPe.. 47 minutes ago Remove the orbital probes (kept in history); add the tabletop replay viewer
cfcec1f  VrabelPe.. 51 minutes ago coord: peter updates 1114-0423-candidate-r-kitchen-tabletop-mujoco-anchors-and (status=running)
b9e73fc  VrabelPe.. 51 minutes ago coord: peter adds exp 1114-0423-candidate-r-kitchen-tabletop-mujoco-anchors-and
2dfd6bd  VrabelPe.. 52 minutes ago probes/tabletop: MuJoCo kitchen tabletop (gantry gripper), scripted reference, Harbor task
7ab5212  VrabelPe.. 65 minutes ago coord: peter adds msg 1114-0409-peter-direction-change-robotics-tabletop-environment-m
6c084d4  VrabelPe.. 65 minutes ago coord: peter updates coordination files
73753cc  VrabelPe.. 68 minutes ago coord: peter updates 1114-0404-probe-a-v2-orbital-ops-with-filter-learning-anch (status=failed)