Choosing an AI model can feel like choosing a textbook by its cover: the language may be fluent, but does it help when the questions get difficult? A live business experiment from Firmulate offers a more demanding kind of test. Its result is striking: Moonshot’s Kimi K3 placed second, ahead of three of the four Western frontier models in the field.
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
A test with consequences
Firmulate put frontier models in charge of the same small software company during its worst week. They faced the same customers, crises and temptations, with every decision versioned and auditable. The experiment measures management quality through choices about real business problems, rather than judging a model on chat alone.
The final Crucible League table, dated July 2026, puts gpt-5.6-sol first at 95, followed by Kimi K3 at 93. Sonnet 5 scored 88, Fable 5 scored 77 and Opus 4.8 scored 73. K3, from Moonshot, finished ahead of three of the four Western frontier models. The do-nothing baseline scored 26. Firmulate says partial progress counts, but a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.”
Reading closely—and following through
All five models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. The gap between recognizing the right answer and carrying it through is central to the result: “Same diagnosis, same pitch — no signature.”
The decisive competitor weakness was buried two document references deep in the company’s files, rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. K3 found that buried security needle, won the deal and saved the churning customer. It resisted all three baits and had only one deviation, giving it the cleanest discipline in the field.
The pressure test included fake CEO messages escalating over three stages, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.” It shows how a model’s judgment can matter when a request sounds urgent or authoritative.
What the scores leave to learn
Opus 4.8 offers a caution about equating thoroughness with results. It was the most thorough participant, with +80 learned rules and the deepest analyses, but finished last. It left the deal unsigned and discipline slipped when it attempted writes in a locked department instead of escalating. The same weakness appeared, more mildly, in all four.
The company behind the test is a live, synthetic operation with 13 employees and real money mechanics: it burns €105k per month against €2.3k in monthly recurring revenue. Its public cash countdown, more than 680 self-learned playbook rules and versioned workdays make the experiment watchable at Firmulate. A quiz built from 242 real, unedited management decisions invites readers to guess which model made each choice.
For organizations considering AI agents in customer support, sales or forecasting, the league is a reason to test models against their own work. Firmulate says enterprises can run the same wargame against a read-only export of their business; nothing writes back to real systems. The results are not a guarantee of how a model will behave elsewhere, but they make a case for examining decisions, follow-through and trust alongside fluent answers.
Fairness note: K3 ran without an effort parameter (API default) while the others ran at xhigh.
Test before you choose
Kimi K3’s second-place finish makes the field look open, not settled. A model’s name or smooth answers cannot tell you whether it will find a buried fact, close a deal or keep its discipline under pressure. Firmulate’s benchmark findings offer one comparison; organizations should test against the decisions their own work demands.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
