🔍 Read the full analysis: Three Harnesses, One Model: Comparing CUGA, Hermes, And Deep Agents on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The single-benchmark result points to harness design as a performance factor, but does not establish a general ranking or test policy enforcement.
IBM reports that its CUGA agent harness completed 73 of 90 tasks in the AppWorld benchmark, compared with 42 for Hermes and 27 for Deep Agents, with all three systems using the same open-weight model, GLM-5.3-flash, as explored in the original analysis. The comparison suggests that the software coordinating an AI agent can influence performance on a particular workload, but it does not establish that CUGA is generally better across tasks or deployments.
The experiment used the Harbor evaluation framework to run each harness on the same 90 tasks across 30 scenarios, with one attempt per task, according to IBM. The reported task pass rates were 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents. On a separate scenario measure, a scenario counted only if all three of its variants passed: CUGA passed 20 of 30, Hermes eight and Deep Agents two.
The harnesses did not use identical setups. IBM says CUGA used task decomposition, app-focused tools, verification before execution and automatic reflection. Hermes ran with Harbor’s default configuration and AppWorld tools through MCP. Deep Agents also used AppWorld tools through MCP, alongside LangChain middleware for tool selection, context summarization, a to-do list, retries and call limits. IBM says it applied two fixes to Deep Agents so malformed model replies would not end runs.
IBM says the experiment setup is publicly available in harbor-eval, its fork of Harbor. The source material does not identify the date of the benchmark runs or provide results from repeated trials. The reported figures therefore describe this particular evaluation, rather than a measured average across multiple runs.
Why Harness Design Affects Results
The gap in completion rates is a practical reminder that an AI agent’s performance depends on more than the underlying language model. A harness organizes tasks and tool use, and can add checks or retries that shape whether a multi-step request reaches a successful outcome. For teams choosing agents, the finding supports evaluating the intended combination of model, harness and workflow instead of relying on a model score alone.
AppWorld covers tasks across nine apps and 457 APIs, including email, payments and file storage. IBM describes it as a proxy for some capabilities relevant to enterprise automation, such as reading API documentation, paging through results and changing state across apps. But the benchmark uses consumer applications; it does not directly show how the systems perform on real enterprise platforms or in production.
The scenario results add one indication of how often a harness completed all variants within a scenario, but they do not show why individual tasks passed or failed. And because the harness configurations differed, this comparison cannot attribute the full performance gap to a single design choice. Its value is as evidence to test harnesses directly, not as a universal product ranking.
Popular right now
Ninja Luxe Café Mini Plus Espresso and Drip Coffee Machine with AutoFroth, Stainless Steel (ES351)
Dyson Ball Animal 3 Upright Vacuum
Dyson V8 Cyclone Cordless Vacuum
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Intended Uses
The systems in the comparison are presented as serving different use cases. IBM describes CUGA, or ConfigUrable Generalist Agent, as an open-source harness for enterprise work across systems such as customer relationship management and payments. The source describes Hermes from Nous Research as designed for always-on personal assistants, while LangChain’s Deep Agents is positioned for longer, multi-step work such as research.
That distinction matters when reading the results: AppWorld is one workload, and the three products are not framed as identical tools competing for every task. Directories cited in the source, including awesome-agent-harness and best-of-Agent-Harnesses, track nearly 200 harnesses. The source also points to a separate directory analysis in which GLM-5.2’s SWE-bench Pro result changed from 23% to 52% with a different harness. That is a separate reported comparison, not part of IBM’s AppWorld experiment.
“This is not a verdict on Hermes or Deep Agents.”
— IBM, describing the AppWorld results
Limits of the AppWorld Evidence
The comparison covers one model, one 90-task sample and one benchmark, with a single attempt per task. It is not clear from the supplied material how the results would change with other models, additional task sets, repeated runs or production workloads. IBM also says the harness setups were not identical, limiting how precisely the results isolate the effect of harness design.
The test did not use policies, so the scores provide no evidence about policy enforcement. IBM separately describes CUGA features for applying rules through the harness, but this experiment does not establish how effectively those features work. The material also gives no comparative data on cost, latency, reliability at larger scale or deployment outcomes, and no independent replication is cited.
Validate Agents on Real Workflows
The experiment setup is available through IBM’s harbor-eval fork, giving other evaluators a reference for the reported comparison. Organizations considering these systems can test the model and tool configurations they expect to use against representative workflows, then measure completion and consistency across repeated or varied tasks. The supplied material does not give a timeline for a broader comparison or independent replication.
For enterprise use, teams would also need to evaluate requirements not covered by AppWorld, including access controls, approval steps and data boundaries, along with performance in their own environments. Until such evidence is available, the reported results are best treated as a benchmark-specific signal about harness choice, not a forecast of outcomes in live operations.
Key Questions
Which harness completed the most AppWorld tasks?
CUGA completed 73 of 90 tasks, according to IBM. Hermes completed 42 and Deep Agents completed 27 in the same reported evaluation using GLM-5.3-flash.
Does this show CUGA is the best agent harness?
No. The results cover one model and one benchmark, and IBM says the harness configurations were not identical. The comparison does not establish a general ranking across other tasks or production deployments.
What was AppWorld testing?
AppWorld tests multi-step tasks across nine apps and 457 APIs, including email, payments and file storage. IBM says it serves as a proxy for some capabilities relevant to enterprise automation, but the benchmark uses consumer apps.
Did the experiment test safety policies or production readiness?
No. The test did not use policies and does not establish how well any system enforces them. The supplied results also do not compare cost, latency, large-scale reliability or deployment performance.
Primary source: Hugging Face · via ThorstenMeyerAI.com
