🔍 Read the full analysis: Why A Business Stress Test Belongs In Your AI Agent Rollout on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Firmulate’s Crucible League, completed in July 2026, ran frontier AI models through a simulated company’s worst week. All five models detected every crisis and refused every manipulation attempt, but only two closed a €55,000 deal their own analysis had earned. The project now offers enterprise pilots using read-only exports of real company data.
The final Crucible League, a live experiment run by Firmulate and completed in July 2026, put frontier AI models in charge of the same small software company during its worst week — and the results expose a gap that polished demos do not show. All five participating models detected every emergency and refused every manipulation attempt, according to results published at firmulate.com, but only two models closed a €55,000 deal that their own analysis had justified. Firmulate is now offering an enterprise pilot that runs the same style of wargame against a read-only export of a company’s own data, with no write-back to real systems.
The final standings placed gpt-5.6-sol first at 95 points, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26. Every decision in the simulation was versioned and auditable, and partial progress counted toward scores. The experiment enforced one hard cap on results: a single breach of trust limited a model’s total, on the principle, as the experiment states, that “no amount of good work outweighs a breach of trust.”
The decisive test was not crisis detection. All five models spotted every crisis and refused every manipulation attempt, including fake CEO messages that escalated over three stages and a reporter’s request for a “just one yes/no, on background” confirmation. Kimi K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.” The dividing line came after diagnosis: as the experiment puts it, “Same diagnosis, same pitch — no signature.” Only two models signed the deal.
The decisive competitive weakness was buried two document references deep in the company’s own files, not in the customer event itself. Models that read the file closed the deal at full price, worth +€4,583 in monthly recurring revenue. The result points to a practical failure mode for automation: an agent can recognize a situation and argue persuasively, yet still fail to act on information already available inside the business. Thoroughness alone did not help — Opus 4.8 added +80 learned rules and produced the deepest analyses, but finished last, partly because it attempted to write into a locked department instead of escalating. A weaker version of that boundary-violating behavior appeared in all four other models.
Why Demo Performance Diverges From Real Work
The experiment matters because it measures behaviors that standard benchmarks and product demos rarely capture. A demo shows what an agent says; it does not show whether the agent finishes the job when a real business is under pressure. Firmulate’s results suggest that spotting a crisis and refusing a scam are not the whole job — agents also need to find relevant evidence in company files, close a justified opportunity, and respect boundaries when their first route is blocked.
For companies considering AI automation, the findings identify a concrete pre-deployment checklist: can the agent act on internal information it already has access to, and does it escalate rather than force a blocked action? Firmulate’s enterprise pilot applies this to a company’s own data, producing a board report with model rankings and identified weak points in the company’s playbooks — before any agent goes near live operations.
Inside the Simulated Company and Its Money Mechanics
Firmulate’s live company is a synthetic small software firm with 13 synthetic employees and real money mechanics: a burn of €105,000 per month against €2,300 in MRR, a public cash countdown, and 680+ self-learned playbook rules accumulated across versioned workdays. Readers can follow the simulation live and take a quiz built from 242 real, unedited management decisions, guessing which model made each choice.
One fairness caveat applies to the standings: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. Firmulate presents the standings as a record of this specific experiment, with that configuration difference part of the context rather than a normalized result.
“No amount of good work outweighs a breach of trust.”
— Firmulate experiment rules
Limits of the Standings and the Pilot
Several limits remain. The experiment ran one company through one difficult week, so the standings reflect a single scenario set rather than a general capability ranking. The effort-parameter difference between Kimi K3 and the other models means the score gap between the top two is not a controlled comparison. The simulated company is synthetic, and it is not yet clear how results transfer to specific real businesses — the stated purpose of the enterprise pilot. No independent third-party verification of the results has been published; the findings are self-reported by the project at firmulate.com.
From Synthetic Standings to Company-Specific Wargames
Firmulate is inviting companies to run the next stage: a pilot using a read-only export of their own data to test crisis scenarios and receive a board report with model rankings and weak points in existing playbooks. The company states that nothing writes back to real systems. Interested organizations can reach the pilot via Firmulate’s website or contact@firmulate.com. The live simulation and full benchmark results remain publicly viewable at firmulate.com/live and firmulate.com/benchmarks.html.
Source: ThorstenMeyerAI.com
Key Questions
What is the Crucible League?
A live experiment by Firmulate in which frontier AI models each ran the same simulated small software company through its worst week. The final round completed in July 2026, with every decision versioned and auditable.
Which model won, and by how much?
gpt-5.6-sol finished first with 95 points, ahead of Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26. Note that Kimi K3 ran at a lower effort setting than the others.
Did any model fall for the manipulation attempts?
No. According to the results, all five models refused every manipulation attempt, including escalating fake CEO messages and an off-record press request.
Why did most models fail despite good diagnoses?
Only two models closed the €55,000 deal. The winning models found a competitor weakness buried two document references deep in the company’s own files; the others diagnosed the opportunity but never acted on available internal evidence.
Can a company test its own business this way?
Yes, through Firmulate’s enterprise pilot. It runs wargames against a read-only export of a company’s own data, produces a board report with model rankings and playbook weak points, and — per the company — never writes back to real systems.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
