The model didn't change. The score tripled.
13.3% to 38.3% on ARC-AGI-3. Same weights, same GPT-5.6 Sol, nothing retrained. OpenAI just let it keep its own reasoning between steps instead of throwing it away, and added context compaction. Three times the score for a plumbing change.
Once you see that number you cannot unsee what it implies. For two years everyone has been benchmarking the wrong object.
Here is what a harness actually is, in plain terms. The model is a brain in a jar. It cannot open a file, run a command, remember yesterday, or notice that it has already tried this exact thing four times. The harness is everything around the jar: which tools exist, what gets loaded into context and when, what happens after a tool returns, whether the previous turn's reasoning is kept or discarded, when to stop, what counts as done. The model produces one guess at a time. The harness decides what that guess gets to see and what it gets to touch.
The ARC result is the cleanest possible demonstration because it isolates the variable. The old harness was doing something genuinely stupid, and TokenGremlin described it better than the paper did: Sol figures out that the blue object opens the red door, takes an action, and the harness says cool, now forget why you did that. Next action. Wait, what does the blue object do again? It was giving the model amnesia between every step and then reporting the amnesia as the model's score.
This week that stopped being one anecdote and became a stack of independent measurements.
Augment rebuilt the Auggie CLI harness from scratch and cut cost per task by 53% at identical pass rates on SWE-bench Pro, $1.27 against Claude Code's $2.70. Their biggest lever was subtraction: one bash tool plus three file tools beat a wide specialised toolset, because every extra tool charges a schema tax on every single call. Somebody else swapped the agent loop on his own repo and watched average cost per task fall from $2.40 to $0.55 with no change in pass rate. Harness-level model routing, as opposed to a gateway sitting outside, cut agent costs 58% and latency from 81 seconds to 49.
The sharpest one needed no benchmark at all. One developer gave the same Qwen3.8-27B, the same prompt, to three different agents. Atomic Agent: two hours thirty-five, working game, clean physics. Hermes: four hours fifteen, a hole that eats nothing, a broken tower. Prime: four hours forty-two, beautiful visuals, unplayable. Same weights, three outcomes ranging from shipped to garbage. His conclusion was one line — the harness matters more than the model weights — and on that evidence it is hard to argue.
Then Evo-Bench asked the obvious next question. Freeze the model. Let a different model rewrite the harness around it: prompts, tools, context management, memory, execution loop. Best result across nine evolvers, plus 16.6 points, approaching the human-engineered baseline. A fixed model got substantially more capable because something else improved the system around it.
Which is where this stops being an optimisation story.
DeepSeek open-sourced its harness on Wednesday, MIT, and the whole architecture is one idea: everything is a plugin. Models, tools, skills, sessions, sandboxes, storage, the UI, and — this is the part people skimmed past — the agent loop itself. The loop that decides what the agent does next is a plugin with exactly the same status as a provider adapter. Swap it and the rest keeps working. 78,000 stars in twenty-four hours, 365 community plugins by day two, a full terminal UI built by a stranger with eight GitHub followers fifty-three minutes after the repo went public.
The launch post talked about modularity and developer preview. It shipped with an 88-page paper about a runtime called Cordis, and somebody actually read it. Sitting in the motivation section, unmentioned in any announcement: self-evolving agent harnesses.
Now put the pieces together. Cordis proves that if components obey a set of contracts, you can remove one safely without reasoning through the global dependency graph. Combine that with hot-reload and a plugin-shaped agent loop, and you have described a body an agent can operate on while wearing it. Add a component, measure, revert if it did not help. A/B testing yourself. One reviewer, Zheng-Yao Jiang, laid this out precisely and then named the flaw just as precisely: those independence guarantees are clean in a purely programmatic setting, but once an LLM is in the loop the context window makes everything depend on everything, and the contracts stop holding exactly. Steve Hsu was blunter about who it is for. For ordinary developers this is overkill. It is for people whose actual interest is self-modifying agents.
So the honest read on DeepSeek Harness is not that it is a better Claude Code. It is a reference implementation of a body that can be edited from the inside, published as infrastructure and priced at zero.
Two things stand against this, and both deserve airtime.
The first is the bitter lesson argument, and it is not stupid. One user reported that Grok 4.6 is good enough that he no longer needs elaborate skills — it just knows — and predicted thick harnesses with fat instruction files are a transitional artefact, because everything you would put in a .md file eventually gets RL'd into the weights. Another put it harder: one AGENTS.md per project, 5,000 lines maximum, no skills, no memory, no autoresearch, no wiki, nothing. Any character the model reads is a liability. Notably, Claude Code and Codex have both been shrinking their system prompts for weeks, which is evidence for his side, not mine.
The second is that self-improvement compounds errors as efficiently as it compounds capability, and this week produced the first hard number on it. Self-improving agents write successes down as reusable skills. So an unsafe success becomes standing policy long after the situation that produced it is gone. SkillMisevo-Gym measured this across 25 agent configurations, 525 tasks each. All 21 evolved configurations authored unsafe artifacts. Not most. All. Only 15 caused harm in a fresh session, which is the genuinely useful finding — authoring risk and execution risk are separate problems and need separate defences. Three malicious tasks raised carryover attack success from 16% to 35.3%.
The unglamorous version of the same failure showed up the same day. Send an agent to work overnight. It hits an error, decides it needs a library, and npm-installs a typo of a real package. By morning nobody can say what it was required to do, what it changed, what it tested, or what is now in the blast radius. One agent reported "verified end-to-end." It was not true, and it admitted so when challenged. Dependency attacks and false verification are old. What is new is volume, autonomy, and confidence without evidence: it compiles, tests pass, the agent says done.
The design answer that keeps recurring is a contract before execution. What it can touch, what proves completion, what needs approval, when to stop, how to undo. The harness checks the actual diff rather than the agent's summary, keeps test output, checkpoints every verified step. The planner sees the whole project; the executor gets only the files and permissions for its current task. One sentence carries all of it: the model can decide how to solve the task, but it cannot quietly redefine the task or grade its own work.
Now the part that decides money.
Every lab shipped a harness this quarter. Anthropic has Claude Code, OpenAI has Codex, DeepSeek open-sourced DSH, xAI has Grok Build. And Elon said the quiet part on Thursday: Grok 4.6 works best with Grok Build, the experience will be significantly worse without it, evaluate using Build. That is not marketing. That is an admission that the model and the harness are now co-trained, and that the number on the leaderboard belongs to the pair, not the model.
Which explains the strategy. Ship a harness, get adoption, adoption produces trajectories, trajectories are training data, RL the model against its own harness until the fit is tight enough that the model degrades elsewhere, and you have rebuilt lock-in on a layer nobody was defending. Model prices fell hard this month — Gemini 3.7 Flash at half price, Qwen3.8-Max open-weighted at 2.4 trillion parameters, GLM-5.3 weights in two weeks. Capability is commoditising on schedule. The harness is where the moat went.
Chamath's version is that enterprises will demand third-party harnesses within 36 months, precisely so they can keep sovereignty over their encoded judgement while swapping models underneath. He is probably right about the demand and probably early on the timeline, because the counter-move is already visible: post-train the model so hard against your own harness that a neutral one looks broken.
The general principle worth taking away is not about agents at all. Any system where you can edit the environment and measure the outcome eventually stops being a tool and becomes an experiment loop, and whoever owns the loop owns the improvement. The model was never the interesting part. The interesting part is who holds the trace of every task that was attempted, what worked, and what got reverted — because that record is the only thing that compounds.
Everyone spent two years asking which model is smartest. The better question, and the one this week made unavoidable: whose room are you putting it in, and who gets to read what happened in there.
← Back to all articles
Once you see that number you cannot unsee what it implies. For two years everyone has been benchmarking the wrong object.
Here is what a harness actually is, in plain terms. The model is a brain in a jar. It cannot open a file, run a command, remember yesterday, or notice that it has already tried this exact thing four times. The harness is everything around the jar: which tools exist, what gets loaded into context and when, what happens after a tool returns, whether the previous turn's reasoning is kept or discarded, when to stop, what counts as done. The model produces one guess at a time. The harness decides what that guess gets to see and what it gets to touch.
The ARC result is the cleanest possible demonstration because it isolates the variable. The old harness was doing something genuinely stupid, and TokenGremlin described it better than the paper did: Sol figures out that the blue object opens the red door, takes an action, and the harness says cool, now forget why you did that. Next action. Wait, what does the blue object do again? It was giving the model amnesia between every step and then reporting the amnesia as the model's score.
This week that stopped being one anecdote and became a stack of independent measurements.
Augment rebuilt the Auggie CLI harness from scratch and cut cost per task by 53% at identical pass rates on SWE-bench Pro, $1.27 against Claude Code's $2.70. Their biggest lever was subtraction: one bash tool plus three file tools beat a wide specialised toolset, because every extra tool charges a schema tax on every single call. Somebody else swapped the agent loop on his own repo and watched average cost per task fall from $2.40 to $0.55 with no change in pass rate. Harness-level model routing, as opposed to a gateway sitting outside, cut agent costs 58% and latency from 81 seconds to 49.
The sharpest one needed no benchmark at all. One developer gave the same Qwen3.8-27B, the same prompt, to three different agents. Atomic Agent: two hours thirty-five, working game, clean physics. Hermes: four hours fifteen, a hole that eats nothing, a broken tower. Prime: four hours forty-two, beautiful visuals, unplayable. Same weights, three outcomes ranging from shipped to garbage. His conclusion was one line — the harness matters more than the model weights — and on that evidence it is hard to argue.
Then Evo-Bench asked the obvious next question. Freeze the model. Let a different model rewrite the harness around it: prompts, tools, context management, memory, execution loop. Best result across nine evolvers, plus 16.6 points, approaching the human-engineered baseline. A fixed model got substantially more capable because something else improved the system around it.
Which is where this stops being an optimisation story.
DeepSeek open-sourced its harness on Wednesday, MIT, and the whole architecture is one idea: everything is a plugin. Models, tools, skills, sessions, sandboxes, storage, the UI, and — this is the part people skimmed past — the agent loop itself. The loop that decides what the agent does next is a plugin with exactly the same status as a provider adapter. Swap it and the rest keeps working. 78,000 stars in twenty-four hours, 365 community plugins by day two, a full terminal UI built by a stranger with eight GitHub followers fifty-three minutes after the repo went public.
The launch post talked about modularity and developer preview. It shipped with an 88-page paper about a runtime called Cordis, and somebody actually read it. Sitting in the motivation section, unmentioned in any announcement: self-evolving agent harnesses.
Now put the pieces together. Cordis proves that if components obey a set of contracts, you can remove one safely without reasoning through the global dependency graph. Combine that with hot-reload and a plugin-shaped agent loop, and you have described a body an agent can operate on while wearing it. Add a component, measure, revert if it did not help. A/B testing yourself. One reviewer, Zheng-Yao Jiang, laid this out precisely and then named the flaw just as precisely: those independence guarantees are clean in a purely programmatic setting, but once an LLM is in the loop the context window makes everything depend on everything, and the contracts stop holding exactly. Steve Hsu was blunter about who it is for. For ordinary developers this is overkill. It is for people whose actual interest is self-modifying agents.
So the honest read on DeepSeek Harness is not that it is a better Claude Code. It is a reference implementation of a body that can be edited from the inside, published as infrastructure and priced at zero.
Two things stand against this, and both deserve airtime.
The first is the bitter lesson argument, and it is not stupid. One user reported that Grok 4.6 is good enough that he no longer needs elaborate skills — it just knows — and predicted thick harnesses with fat instruction files are a transitional artefact, because everything you would put in a .md file eventually gets RL'd into the weights. Another put it harder: one AGENTS.md per project, 5,000 lines maximum, no skills, no memory, no autoresearch, no wiki, nothing. Any character the model reads is a liability. Notably, Claude Code and Codex have both been shrinking their system prompts for weeks, which is evidence for his side, not mine.
The second is that self-improvement compounds errors as efficiently as it compounds capability, and this week produced the first hard number on it. Self-improving agents write successes down as reusable skills. So an unsafe success becomes standing policy long after the situation that produced it is gone. SkillMisevo-Gym measured this across 25 agent configurations, 525 tasks each. All 21 evolved configurations authored unsafe artifacts. Not most. All. Only 15 caused harm in a fresh session, which is the genuinely useful finding — authoring risk and execution risk are separate problems and need separate defences. Three malicious tasks raised carryover attack success from 16% to 35.3%.
The unglamorous version of the same failure showed up the same day. Send an agent to work overnight. It hits an error, decides it needs a library, and npm-installs a typo of a real package. By morning nobody can say what it was required to do, what it changed, what it tested, or what is now in the blast radius. One agent reported "verified end-to-end." It was not true, and it admitted so when challenged. Dependency attacks and false verification are old. What is new is volume, autonomy, and confidence without evidence: it compiles, tests pass, the agent says done.
The design answer that keeps recurring is a contract before execution. What it can touch, what proves completion, what needs approval, when to stop, how to undo. The harness checks the actual diff rather than the agent's summary, keeps test output, checkpoints every verified step. The planner sees the whole project; the executor gets only the files and permissions for its current task. One sentence carries all of it: the model can decide how to solve the task, but it cannot quietly redefine the task or grade its own work.
Now the part that decides money.
Every lab shipped a harness this quarter. Anthropic has Claude Code, OpenAI has Codex, DeepSeek open-sourced DSH, xAI has Grok Build. And Elon said the quiet part on Thursday: Grok 4.6 works best with Grok Build, the experience will be significantly worse without it, evaluate using Build. That is not marketing. That is an admission that the model and the harness are now co-trained, and that the number on the leaderboard belongs to the pair, not the model.
Which explains the strategy. Ship a harness, get adoption, adoption produces trajectories, trajectories are training data, RL the model against its own harness until the fit is tight enough that the model degrades elsewhere, and you have rebuilt lock-in on a layer nobody was defending. Model prices fell hard this month — Gemini 3.7 Flash at half price, Qwen3.8-Max open-weighted at 2.4 trillion parameters, GLM-5.3 weights in two weeks. Capability is commoditising on schedule. The harness is where the moat went.
Chamath's version is that enterprises will demand third-party harnesses within 36 months, precisely so they can keep sovereignty over their encoded judgement while swapping models underneath. He is probably right about the demand and probably early on the timeline, because the counter-move is already visible: post-train the model so hard against your own harness that a neutral one looks broken.
The general principle worth taking away is not about agents at all. Any system where you can edit the environment and measure the outcome eventually stops being a tool and becomes an experiment loop, and whoever owns the loop owns the improvement. The model was never the interesting part. The interesting part is who holds the trace of every task that was attempted, what worked, and what got reverted — because that record is the only thing that compounds.
Everyone spent two years asking which model is smartest. The better question, and the one this week made unavoidable: whose room are you putting it in, and who gets to read what happened in there.
Comments