An agent harness is the code around a model: execution loop, tools, context, state, recovery, and verification. Per the Terminal-Bench 2.1 leaderboard, GPT-5 solves 35.2% of tasks inside Terminus 2 but 49.6% inside Codex CLI with identical weights. Most benchmarks leave that harness unchanged. HarnessDev Researchers from ByteDance Seed and researchers at Georgia Institute of Technology and M-A-P and TokenWave.AI have proposed a new approach that flips the goal: The artifact to be evaluated is not the result, but the harness in which the model runs.
2 stages: Creation and Evolution
In Creation, every creator receives the same weak seed: passive file, search, and process primitives plus result and trajectory writers, with no loop, planner, verifier, retry, or stopping rule. It scores zero everywhere if it is not modified. A task-family specification, a brief design tutorial and up to three development cases are provided. After building a harness, the creator freezes it before hiding tasks.
In Evolution, the creator starts from its own frozen Creation code harness and revises it using execution feedback from a fixed set of 100 SWE-bench Pro tasks and all 89 Terminal-Bench 2.1 tasks. The official candidates must perform both evaluations in pairs, each with 10 pairs of tasks and a maximum 2 probes of five tasks between them. The creator is never shown 630 SWE-Pro versions that will score each official version.
The harnesses were graded based on their capability (task completion) and efficiency.
Setup
6 creator LLMs were tested: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro, working inside Claude Code 2.1.177 (GPT-5.5 used Codex 0.144.3). Creation spans 4 domains and 5 benchmarks totaling 2,207 instances: SWE-bench Pro public split (731), Terminal-Bench 2.1 (89), MLE-bench (75), EQ-Bench3 (46), and BrowseComp (1,266). The average harness per benchmark is reported by each creator. Self-Eval evaluates each harness in conjunction with the creator, while Unified-Eval uses Gemini 3.1 PRO to run all.
Creation results
Under Self-Eval, Opus 4.8 posts the highest average score at 67.8 against a human-engineered reference of 86.2. The gap depends on domain:
- Code: Opus 4.8 reaches 69.3 on SWE-Pro versus the 80.0 reference. Gemini Pro 3.1 leads Terminal-Bench with 68.8 points compared to 88.8.
- Search: the widest gap. BrowseComp’s best score is 52.66 (GPT-5.5), compared to a reference of 92.2.
- Writing: Opus 4.8 scores 84.6 on EQ-Bench3, above the 83.7 reference.
- ML experimentation: Opus 4.8 (32.9) and Gemini (32.4) beat the 24.0 MLE-bench reference.
SWE Pro, Terminal Bench, and BrowseComp are OpenAI GPT 5.6 results, not repeats.
The volume of code did not correlate with quality. Although the 18 harnesses had added 17,111 lines net, Gemini was the one that added the least (1,006), and Terminal-Bench ranked first. The self-testing count was barely related to the score (Spearman 0,13 to 0.26), and revision calls were as high as 0.57.
Many of the generated machines are inert. From 108 instances of code components, 72 fire in actual runs, and 18 do not. They are all state and memory. On 11 out 18 harnesses a state class is defined, but no checkpoint occurs across the 26,679 paths. Of the 587 features, 124 are code.
Cost and executor transfer
MLE-bench token use varied roughly 19-fold. DeepSeek v4 achieved a 19.6 medal rate using 208.4M tokens, while GPT-5.5 only managed 19.1 when it used 29.3M. The swapping of the executor from Gemini changed rankings. Qwen scored 17.6 on BrowseComp, and 12.9 points on MLE Bench, while Opus’s SWE Pro score dropped from 69.3 down to 30.0, in part because some harnesses hard coded 120 steps around their original executor. After the change, Opus’ duplicate-query ratio jumped to 88.2%.
Evolution results
9 lineages (5 self-runtime, 4 fixed-Gemini) produced 73 official versions and 64 adjacent switches. The 5 creators of self-runtime improved their scores on tasks that were held out, ranging from +1.43 to +4.44 (mean +3.11). Only Opus showed improvement under fixed Gemini; GPT 5.5 declined 10.32 points.
The progress was not linear. Eight switches regressed in both benchmarks; 16 on just one. 27 switched gained within the noise range, while 2 were clearly positive. A single commit can vary by about ±4.75 pair-score points. Only 34 out of 64 (53.1%) feedback and hold-out score moved in the exact same direction, and 2 of 9 finalized versions had optimal held-out. One quarter of the new classes or functions have no calling code.
The clearest win: Opus 4.8 noticed 99 of 100 runs reported success while only 48 passed, traced it to premature completion, and added a completion gate. The failure diagnosis step was the least effective: only two times was the trajectory interface called.

