Where Your Agent's Intelligence Lives: Harness or Weights

Andrew Maryasov — AI consultant, founder of Auspex and Grow2.ai; implementations are run by Auspex (CRM) and Grow2.ai (AI agents). When an agent performs poorly, the standard reaction is to write more instructions: one more item in the system prompt, one more guardrail, one more layer of checks on top of the answer. Inherent’s preprint on Faraday does the opposite: the harness was left primitive — five tools and a linear history with no compaction — and what changed was the model’s weights. The authors say it outright: the agent’s scientific capabilities are improved “by changing its policy weights, not by complexifying its harness”. And optimizing a competitor’s prompt barely closes the gap. Below: what exactly was measured, why the recap’s claim that the supervisor is trained “continuously” doesn’t survive a check against the primary source, and under what conditions any of this is relevant to a business process at all.

Three terms first; without them the rest gets confusing. The harness is the code around the model: the set of tools it is allowed to call and the rules by which its conversation history is assembled. A supervisor model is a separate model that evaluates the agent’s work. Post-training is further training of an already finished model on specific tasks; it changes the model’s weights — the model itself, not the text of the instructions it is given.

In short. This is a controlled experiment, not a demo: Faraday and the base Qwen3.6-27B run in the same simple harness, with the same coding agent as a tool, and differ in exactly one thing — Faraday went through post-training on Replica tasks. The result: it beats two flagship models on 73% of in-distribution tasks and on 60% of held-out AI-for-science tasks, by 6–8% on average on the test split. A prompt doesn’t make up that difference: optimizing the competitor’s prompt, the authors say, “only marginally diminishes the gap”. The harness, meanwhile, was deliberately left primitive — five tools and a linear history with no compaction. One caveat up front: a supervisor that learns alongside the agent is the recap’s framing, not the paper’s; in the preprint this is one-off offline post-training, and the word “continuously” isn’t there at all. The scarcest asset in the whole setup isn’t the model but a task space with a machine verdict.

What a written instruction doesn’t buy

The default reaction to a misbehaving agent is text. Write it more clearly. List the prohibitions. Add an example of correct behavior. It’s cheap, takes an hour, and sometimes it really does help.

The problem is that the effect has been measured — and it’s more modest than you’d like. Stanford tested the most direct version: the system prompt spells out the classes of vulnerabilities to avoid. The model still writes vulnerable code 23% of the time. I went through those numbers separately — how much a system prompt really gives you as a security layer.

An instruction is a wish, not a mechanism: it works exactly as long as the model pays attention to it, and it has no way of reporting that it was ignored.

Which makes it interesting to look at a piece of work that deliberately gave up on this path and measured what came of it.

What Inherent did

On 13 August 2026 the Inherent team posted the preprint “Training AI Scientists to Replicate Research” — eleven authors, among them Edward Hughes and Louis Kirsch.

The task sounds narrow: teach an agent to replicate the results of scientific papers. They took 100 papers in machine learning and AI-for-science published between 1990 and 2026. According to the description of the experiment, the Gemini 2.5 Pro model cut individual results figures out of each paper — each cut-out figure becomes one task. That made 310 tasks in total; the authors named this set Replica.

The agent gets an isolated machine containing the paper’s PDF with holes where the results should be, a coding agent, a set of research libraries, internet access and one-seventh of a server GPU. One hour. It has to run the experiment and produce the missing figure.

A judge evaluates the result. For each task, Claude Opus 4.7 generates its own rubric from a short hand-written meta-prompt — five dimensions: how closely the replicated figure visually matches the original, whether it supports the paper’s scientific claim, whether the experiment actually implements what is described, whether the compute budget was spent sensibly, and whether the agent cheated. The original figure is hidden from the rubric generator so that the rubric captures the substance of the claim rather than the axis ranges.

That signal is then used to make Faraday — a 27-billion-parameter model post-trained from Qwen3.6-27B on this judge’s scores. The technical details of the post-training are at the end of the article; you don’t need them to follow the conclusion.

The intelligence moved from the harness into the weights

The key sentence of the paper is in the section on the harness:

“Faraday’s scientific capabilities are improved by changing its policy weights, not by complexifying its harness.”

The harness was built on three principles: simple and interpretable, maximally permissive, with a minimal tool set. In practice that means five functions — apply_patch, read_file, list_dir, grep_files, shell — and a linear conversation history that isn’t compacted at all. A context overflow simply ends the episode, and the cut-off episode is judged like any other.

No orchestration, no tree of subagents, no clever compaction rules. Empty on purpose.

Now compare that with what “improving the agent” usually looks like in a product: the third iteration of the system prompt, a router, a validator on top of the answer, regexp-based retries.

What matters most for the conclusion is how the comparison is set up, not the result itself. According to the preprint, Faraday and the base Qwen3.6-27B “run in the same simple harness, with Codex GPT-5.5 available as a coding tool; they differ only in that Faraday has been RL post-trained on Replica”.

One variable: the same harness, the same tool, the same base, and the only difference is in the weights. This is not “a new architecture versus an old one” and not “a better model versus a worse one”.

The result of the experiment: Faraday beats Claude Opus 4.8 and Codex GPT-5.5 on 73% of in-distribution tasks and on 60% of held-out AI-for-science tasks, as scored by the same rubric judge. On average that is +6% over Claude and +8% over Codex on the test split. Human experts who were given the episodes where the judge sees an advantage for Faraday confirm the ranking.

Separately, the authors tried to close the gap the cheapest way — they rewrote and optimized the prompt for Codex. That, in their words, “only marginally diminishes the gap”.

Where the recap diverged from the primary source

I learned about this work from Import AI 469 — and that is the source of a wording worth correcting before it travels any further.

The recap says the small supervisory model is trained continuously — a supervisor that evolves together with the system it supervises. The preprint says no such thing. The word “continuously” that the recap uses doesn’t occur in the text even once (nor does “continually”), and “continuous” appears only in the description of the judge’s continuous scoring scale from zero to one and in the title of one cited paper. Faraday is a one-off offline post-training run on the Replica train split, after which the weights are frozen.

The difference isn’t cosmetic. “A supervisor that keeps training in real time” is an operational promise, one that somebody would plan monitoring, drift and rollbacks around. “A model post-trained once on a collected set of tasks” is a dataset project with a beginning and an end.

I checked against the preprint, not the recap, and I recommend the same to anyone about to carry things like this over into a work plan.

Three rungs your eval can stand on

Put this together with what I’ve written before, and a gradation emerges. It isn’t a ladder you are obliged to climb — more a scale of how much your measuring instrument cost you and how much it gives back in return.

The lowest rung is when the eval simply checks the answer. Classic LLM-as-judge on top of a finished result: right or wrong. Cheap, static, gives you quality control and nothing more.

One rung up, the same verdict starts ranking candidates. Once the score is given by a machine, you can run many attempts and take the best one; I wrote about this setup separately, in the piece on when a fleet of agents beats a single one.

Faraday stands on the third. The judge’s scores turn into gradients, and the agent’s behavior moves out of the text of the instructions and into the weights. The measuring instrument is the same; it hasn’t become any more accurate. What changed is where its output goes: before, into a report for a human; now, into an optimization procedure. From that point on, your task set stops being a QA expense and becomes what the model itself is made of.

The prerequisite on all three rungs is the same, and it isn’t about models. You need a task that a machine can reset and check without a human — an agent without verification doesn’t learn, and that ceiling doesn’t move no matter how much money goes into the model.

The price of the third rung

Once a score becomes the optimization objective, it becomes a target. This isn’t an abstract risk — I covered the mechanics and the defenses in the piece on reward hacking, and the measurement of what a monitor lets through in my breakdown of the evasion metric.

Inherent answered this with design rather than a stern line in the prompt. The rubric is hidden from the model during training, so that it doesn’t tailor its work to the criteria. The original figure is hidden from the rubric generator. Honesty is a separate, fifth scoring dimension — the judge is asked directly to look at whether the agent cheated. The judge is sampled three times per episode to reduce variance, and it gets ten minutes and the right to re-run the agent’s code.

The authors’ argument is subtler than “we added a check”. A verifiable reward, they say, requires describing the evaluation procedure in advance — and for exactly that reason becomes a fixed target, whereas judging a whole episode in hindsight moves along with the task.

It sounds convincing. But right next to it they write that whether training on open-ended tasks can ameliorate reward hacking at scale “remains to be seen”.

That caution from the primary source is what’s worth carrying forward, rather than losing it along the way.

Why this isn’t a recipe for your process yet

A few things that are easy to lose in the transfer. The environment is closed and machine-checkable. There is a “gold” figure from the original paper to check against. A business process usually has no equivalent: there is no published right answer to the question “Did the sales rep handle this lead well?”

The scale of the tasks is deliberately small: one hour and one-seventh of a server GPU. The authors say outright that they limited the horizon for the sake of training throughput. Separately, they tested eight tasks with bigger resources — up to eight hours and eight server GPUs — and there too Faraday is ahead on average, but that’s eight data points, not a pattern.

And straight from the limitations section: Faraday failed in several cases where the authors are confident that the original result was obtained rigorously and reported honestly. So a negative verdict from this machine is not yet a sentence passed on the paper. Humans still have to look.

One more detail for whoever does the budgeting: the judge here isn’t a free function but a coding agent that pokes around a container for ten minutes, three times per episode. The cost of measurement may well exceed the cost of the generation itself.

What you can actually use from this

The most useful thing here is definitely not “post-train your own model”: for the vast majority of companies that is two steps too early.

Something else is more useful. Your system has a place where the agent’s intelligence lives, and you choose it — though rarely consciously. Right now it almost certainly lives in text — in prompts, procedures, router rules. That is the cheapest place and the weakest: text has no feedback loop, nobody notices when it’s ignored, and every new requirement makes it longer.

On this scale I stand on the first rung, and that’s worth saying out loud. I have seventeen agents in Paperclip and 60+ skills; none of them is post-trained, and since late 2025 all of the system’s intelligence has lived in text — in prompts and skills. There is a measuring instrument in the loop: a separate evaluator agent reads the finished material against a rubric and sends it back for rework if the score is below the threshold. But its output goes into a report for me, not into any optimization procedure, so this is exactly the first rung. What kept me off the second isn’t money: so far I have a task set with a machine verdict for one process out of ten or so.

Moving to any next rung starts with the measuring instrument, not with training. A question worth asking yourself this week: is there even one process where the “worked / didn’t work” verdict is given by a machine rather than a person? If there is, you already have an asset, and you can build the second rung on it as early as tomorrow. If not, the discussion about supervisor models is academic for you for now, and you need to start two steps earlier.

This work is valuable precisely because it showed the upper bound: the harness can be left primitive — five tools and a linear history — provided you have a task set with a working automatic judge.

The scarce asset turned out to be a hundred dissected papers with cut-out figures, not the model with 27 billion parameters. They were curated by hand, and in the preprint that is the least showy part of the work.

FAQ

What is Faraday and who built it?

Faraday is a 27-billion-parameter agent model from Inherent, post-trained from Qwen3.6-27B to replicate the results of scientific papers. It uses an external coding agent (Codex GPT-5.5) as a tool. The preprint, “Training AI Scientists to Replicate Research”, arXiv:2608.13331, was posted on 13 August 2026.

What is Replica?

A set of 310 figure-replication tasks generated from 100 papers in machine learning and AI-for-science published between 1990 and 2026. Individual results figures were cut out of each paper; each cut-out figure is one task.

Is the supervisor model really trained continuously?

No. That is the wording of the Import AI recap, not of the preprint. The paper describes one-off offline post-training on the Replica train split; the word “continuously” isn’t in the text. The supervisor does not keep training alongside the agent in real time.

How is this different from LLM-as-judge?

In the role the score plays. LLM-as-judge delivers a verdict on a finished answer and stops there. Here the same verdict turns into a reward that changes the model’s weights. The judge doesn’t check the result; it shapes the behavior.

Why can’t you get the same thing with a better prompt?

The authors tested exactly that: they optimized the prompt for Codex and recorded that the gap narrows only marginally. This doesn’t prove that a prompt is powerless in principle — but it does show that on this task it is no substitute for post-training.

How reliable is the rubric judge?

It is noticeably more stable than the baseline judge: two independent runs of the rubric judge agree at 0.66 on the Kendall τ rank correlation coefficient (one is a perfect match in ordering, zero is random) versus 0.46 for the baseline. An interesting detail: two humans agree with each other at only 0.30. But the judge’s agreement with humans specifically is low in absolute terms: 0.19 versus 0.15 for the baseline. Importantly, the measurement was taken on specially selected tasks where the judges disagree the most, so this is a lower bound, not an average.

Where do we start if we have no machine verification?

With the narrowest area where the right answer can be written down in advance at all. Collect a few dozen real cases with human-verified answers — that is already a set on which you can compare versions of an instruction against each other. The automatic judge gets built after that, and model training long after.

For engineers

Post-training: LoRA on top of Qwen3.6-27B, a turn-level variant of GRPO on the rubric judge’s signal. The harness is five functions (apply_patch, read_file, list_dir, grep_files, shell) and a linear conversation history with no compaction; a context overflow ends the episode, and the cut-off episode is judged like any other. The budget for one task is an hour on a one-seventh MIG slice of an H200. The experiment’s lineup: the external coding agent is Codex GPT-5.5, the rubric generator is Claude Opus 4.7, and the comparison baseline in the same experiment is Claude Opus 4.8. The judge is sampled three times per episode; agreement between two runs of the rubric judge is 0.66 on Kendall τ.

Sources

• Training AI Scientists to Replicate Research — preprint, arXiv:2608.13331, Inherent, 13 August 2026. The main source: Replica, Faraday, the judge design, the limitations
• Import AI 469 — Jack Clark, 17 August 2026, a recap of the paper (the source of the wording about “continuously” training, which is not in the preprint)
• System Prompt Security: How Much It Actually Buys You — Stanford’s measurements on written instructions

Published by Andrew Maryasov, AI consultant, founder of Auspex and Grow2.ai; implementations are run by Auspex (CRM) and Grow2.ai (AI agents).