Close Menu
  • AI
  • Content Creation
  • Tech
  • Robotics
AI-trends.todayAI-trends.today
  • AI
  • Content Creation
  • Tech
  • Robotics
Trending
  • OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI
  • A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels
  • Collectively AI Discloses Immediate Leak After Coding Agent Uncovered Credential – Unite.AI
  • Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains
  • Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI
  • AWS Routes HyperPod Inference Throughout Areas From One Endpoint – Unite.AI
  • DailyAI Briefing: A Lean-checked Riemann proof, plus 4 extra
  • Anthropic Shares Drop 8.5% in Second Session Throughout AI Selloff – Unite.AI
AI-trends.todayAI-trends.today
Home»Tech»LLMs Can Engineer Their Own Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize

LLMs Can Engineer Their Own Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize

Tech By Gavin Wallace11/09/20265 Mins Read
Facebook Twitter LinkedIn Email
A Coding Implementation to Build an Interactive Transcript and PDF
A Coding Implementation to Build an Interactive Transcript and PDF
Share
Facebook Twitter LinkedIn Email

An agent harness is the code around a model: execution loop, tools, context, state, recovery, and verification. Per the Terminal-Bench 2.1 leaderboard, GPT-5 solves 35.2% of tasks inside Terminus 2 but 49.6% inside Codex CLI with identical weights. Most benchmarks leave that harness unchanged. HarnessDev Researchers from ByteDance Seed and researchers at Georgia Institute of Technology and M-A-P and TokenWave.AI have proposed a new approach that flips the goal: The artifact to be evaluated is not the result, but the harness in which the model runs.

2 stages: Creation and Evolution

In Creation, every creator receives the same weak seed: passive file, search, and process primitives plus result and trajectory writers, with no loop, planner, verifier, retry, or stopping rule. It scores zero everywhere if it is not modified. A task-family specification, a brief design tutorial and up to three development cases are provided. After building a harness, the creator freezes it before hiding tasks.

In Evolution, the creator starts from its own frozen Creation code harness and revises it using execution feedback from a fixed set of 100 SWE-bench Pro tasks and all 89 Terminal-Bench 2.1 tasks. The official candidates must perform both evaluations in pairs, each with 10 pairs of tasks and a maximum 2 probes of five tasks between them. The creator is never shown 630 SWE-Pro versions that will score each official version.

The harnesses were graded based on their capability (task completion) and efficiency.

Setup

6 creator LLMs were tested: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro, working inside Claude Code 2.1.177 (GPT-5.5 used Codex 0.144.3). Creation spans 4 domains and 5 benchmarks totaling 2,207 instances: SWE-bench Pro public split (731), Terminal-Bench 2.1 (89), MLE-bench (75), EQ-Bench3 (46), and BrowseComp (1,266). The average harness per benchmark is reported by each creator. Self-Eval evaluates each harness in conjunction with the creator, while Unified-Eval uses Gemini 3.1 PRO to run all.

Creation results

Under Self-Eval, Opus 4.8 posts the highest average score at 67.8 against a human-engineered reference of 86.2. The gap depends on domain:

  • Code: Opus 4.8 reaches 69.3 on SWE-Pro versus the 80.0 reference. Gemini Pro 3.1 leads Terminal-Bench with 68.8 points compared to 88.8.
  • Search: the widest gap. BrowseComp’s best score is 52.66 (GPT-5.5), compared to a reference of 92.2.
  • Writing: Opus 4.8 scores 84.6 on EQ-Bench3, above the 83.7 reference.
  • ML experimentation: Opus 4.8 (32.9) and Gemini (32.4) beat the 24.0 MLE-bench reference.

SWE Pro, Terminal Bench, and BrowseComp are OpenAI GPT 5.6 results, not repeats.

The volume of code did not correlate with quality. Although the 18 harnesses had added 17,111 lines net, Gemini was the one that added the least (1,006), and Terminal-Bench ranked first. The self-testing count was barely related to the score (Spearman 0,13 to 0.26), and revision calls were as high as 0.57.

Many of the generated machines are inert. From 108 instances of code components, 72 fire in actual runs, and 18 do not. They are all state and memory. On 11 out 18 harnesses a state class is defined, but no checkpoint occurs across the 26,679 paths. Of the 587 features, 124 are code.

Cost and executor transfer

MLE-bench token use varied roughly 19-fold. DeepSeek v4 achieved a 19.6 medal rate using 208.4M tokens, while GPT-5.5 only managed 19.1 when it used 29.3M. The swapping of the executor from Gemini changed rankings. Qwen scored 17.6 on BrowseComp, and 12.9 points on MLE Bench, while Opus’s SWE Pro score dropped from 69.3 down to 30.0, in part because some harnesses hard coded 120 steps around their original executor. After the change, Opus’ duplicate-query ratio jumped to 88.2%.

Evolution results

9 lineages (5 self-runtime, 4 fixed-Gemini) produced 73 official versions and 64 adjacent switches. The 5 creators of self-runtime improved their scores on tasks that were held out, ranging from +1.43 to +4.44 (mean +3.11). Only Opus showed improvement under fixed Gemini; GPT 5.5 declined 10.32 points.

The progress was not linear. Eight switches regressed in both benchmarks; 16 on just one. 27 switched gained within the noise range, while 2 were clearly positive. A single commit can vary by about ±4.75 pair-score points. Only 34 out of 64 (53.1%) feedback and hold-out score moved in the exact same direction, and 2 of 9 finalized versions had optimal held-out. One quarter of the new classes or functions have no calling code.

The clearest win: Opus 4.8 noticed 99 of 100 runs reported success while only 48 passed, traced it to premature completion, and added a completion gate. The failure diagnosis step was the least effective: only two times was the trajectory interface called.

Interactive explainer

ar bytedance
Share. Facebook Twitter LinkedIn Email
Avatar
Gavin Wallace

Related Posts

A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

22/11/2026

OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min

21/11/2026

Meet Pondering Machines Inkling 2 Mini: 26B Looped MoE Runs on a 32GB Mac

21/11/2026

Baseten Releases Bison-1 and Groundhog-1, Topping Choice Mannequin Index

21/11/2026
Top News

UK House Workplace Might Face Investigation Over Use of OpenAI Instruments

Anthropic: Claude Machines Should Cease If the Hyperlink Drops

Lisa Su, AMD’s CEO, says concerns about an artificial intelligence bubble are overblown

Hackathon Man vs. Machine: A Look Inside

DailyAI Briefing: Gemini might breach press rights, plus 4 extra

Load More
AI-Trends.Today

Your daily source of AI news and trends. Stay up to date with everything AI and automation!

X (Twitter) Instagram
Top Insights

Samsung Researchers Introduced ANSE (Active Noise Selection for Generation): A Model-Aware Framework for Improving Text-to-Video Diffusion Models through Attention-Based Uncertainty Estimation

30/05/2025

EMERGING You can also find out more about the following: Promethean Raise $300M Experience Fund With $500M Hard Cap – Unite.AI

16/08/2026
Latest News

OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI

22/11/2026

A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

22/11/2026
X (Twitter) Instagram
  • Privacy Policy
  • Contact Us
  • Terms and Conditions
© 2026 AI-Trends.Today

Type above and press Enter to search. Press Esc to cancel.