Harness results

What we recommend

1. InferOptimal OpenEvolve tool

   Recommended for well-defined tasks with a reward that can be hill-climbed: a score you can compute for every candidate and want to raise. Guide

2. Microsoft Agent Framework fan-out

   Recommended as the best overall harness we measured, with known failure modes where it fails completely: document retrieval, finding vulnerable files, a run that ends at its time cap with no answer, and very heavy token use. Guide

3. DeepSeek Harness on your endpoint

   Recommended as the safer alternative harness. That is our recommendation, resting on measured facts: one agent ended each of its short-task runs in minutes, none at the time cap, and was right 5 of 9 times on hard research. Its own measured weakness: one agent can run into the output limit on a long answer and end with none. Guide

OpenResearch add-on for the OpenEvolve tool: In development: still offered, but not recommended yet, because it is not performing well enough. Its measured results are below. Guide

Each tool is listed with its install and start commands in the console's Agent tools section, and your coding agent reads them with query action list_agent_tools.

What the runs show

1. What we recommend: the OpenEvolve tool for well-defined tasks with a reward it can climb; the Microsoft Agent Framework harness, the best overall but with known cases where it fails completely (0 on its retrieval and vulnerability runs); and DeepSeek Harness as the safer alternative. OpenResearch is still in development.
2. Hard research is where a swarm pays off. On the legislation questions the Microsoft Agent Framework was right 9 of 14 times and OpenResearch with the Sonnet decider 7 of 14. One plain GLM agent was right 1 of 15 times and GPT-6 Astra alone 1 of 5, though both answered in minutes.
3. On short, well-specified tasks one GLM agent did about as well as any swarm, faster and for much less: 3 or 4 of 4 right on tax law and data analysis in 1 to 6 minutes for under $2 a run. The swarms were also right on the items they reached, but took 4 to 40 minutes and $1.50 to $13 a run.
4. The box was torn down while the swarms were on their first item or two of each benchmark, so their short-task points rest on one or two runs each.
5. Document retrieval and finding vulnerable files stayed hard for every harness: the best averages were 22 of 100 on retrieval and 0.36 File F1 on vulnerabilities, the latter from a single run.
6. OpenEvolve steered only by the OpenResearch add-on’s judgment, with no cycle count in the search, made the Anthropic kernel 55.7 times faster for about $60 of box time. OpenEvolve with a numeric evaluator was not measured: its one ALE-Bench run, at high reasoning effort, produced no programs, and the low-effort runs never started.

The key players for each kind of task

For each kind of task: GLM-5.3-Flash with no harness (one agent) and with the swarm harnesses we tested on every task, beside one frontier model from another provider for reference.

Every harness we tested

The same charts with every harness we ran and every published frontier score we found.

Not measured before the box was torn down

ALE-Bench: our one finished run, a plain GLM agent on ahc039, had wrong answers on 7 of 150 cases, so it scores 0, below the 753 the unchanged starting programs score. OpenEvolve’s low-effort runs never started.

Other published reference points

Scores that are neither a model on our endpoint nor another provider's frontier model, shown here instead of on the charts.

How cost is counted

• GLM-5.3-Flash box (all GLM harnesses): $20 per hour of run time, whatever the tokens. Source: owner, 2026-10-19.
• GPT-6 Astra API: $10 per million input tokens, $1 cached, $50 output. Source: eesel.ai and anotherwrapper.com (citing OpenAI pricing, 2026-10-11).
• Claude Sonnet decider and brief writer (OpenResearch): cost the Claude CLI reports at API prices; brief writer averaged at $1.16 a run in Experiment 5. Source: converge results and writer-sonnet.jsonl.

Where the numbers come from

Every bar is computed from the data files runs.json, published.json, kinds.json, findings.json, rates.json, manifest.json; the manifest records how the data was compiled, from which runs and under which owner records. Logos name a harness's maker or a model provider; their sources and licences are recorded with the site's brand assets.