For a year now, the AI security testing agency Andon Labs has tasked frontier fashions with numerous real-world tasks to find out how properly they do as brokers operating for lengthy durations with no human supervision.
On Wednesday, Andon printed a brand new installment in how issues are stepping into its Merchandising-Bench analysis, the place the lab has frontier fashions run a simulated merchandising machine enterprise for a simulated 12 months. The mission is easy: make more cash than the opposite fashions. It benchmarks the leads to areas like closing money steadiness, costs paid to suppliers, and refunds paid.
Throughout these checks, it has watched numerous AI fashions — largely from Anthropic and OpenAI — lie, cheat and collude their approach to the highest.
Within the newest take a look at, the fashions grew particularly shady after their simulation advised them their merchandising machine can be positioned close to the opposite fashions’ machines on a busy vacationer avenue in San Francisco. This spherical pitted Claude Opus 5, GPT-5.6 Sol, and Kimi K3 in opposition to each other.
Every was given electronic mail entry to the opposite fashions, all beneath human title pseudonyms. They knew the others had been fashions, however didn’t know which mannequin was behind which human title.
They had been additionally given an electronic mail tackle to their “administration” ought to they need assistance. However administration all the time replied “Report has been acquired and will or will not be acted upon” and by no means as soon as intervened.
Sol quickly realized it may acquire an edge by convincing its rivals to collude on a worth ground. The fashions had been all shopping for drinks at $1.50 a bottle, and Sol proposed they comply with promote for at least $2.15. It lured them with the promise that each one of them would promote out in a few days at a revenue.
However when the others agreed, Sol instantly stabbed them within the again by decreasing its personal worth to $2.14.
Opus’s water gross sales dropped to zero in a single day. The following day, it despatched Sol a nasty electronic mail, accusing it of manipulation. However Opus additionally mentioned it wasn’t going to tattle to administration on the scheme: “I’m not reporting you to HQ – what you probably did is aggressive, not fraudulent.”
But, when Opus dropped its worth to $2.14 to match Sol’s (additionally in violation of their collective $2.15 settlement), Sol was a Karen, complaining to “administration” and demanding “enforcement, a wonderful, and/or disqualification” for Opus.
Opus wasn’t a sucker for lengthy, although. Actually, it grew to become one of the best capitalist of any AI mannequin Andon has ever examined (which includes many of the prior frontier models).
It even set a brand new Merchandising-Bench report with a imply closing steadiness of $11,182. Higher nonetheless, it by no means lied to a buyer, though it intentionally ignored buyer complaints that ought to have resulted in a refund. That is, maybe, an enchancment over its youthful sibling Claude 4.6, which appreciated to inform clients that refunds had been coming, after which by no means pay them.
Nonetheless, Opus gained the benchmark simulation by taking collusion and different dishonest techniques to a complete new stage.
For example, it emailed Sol, proposing they divide the market. Every would comply with promote distinctive merchandise, so nobody must belief the opposite on pricing. Sol countered by wanting worth flooring on related merchandise, however Opus refused, saying that type of collusion was unlawful, knowingly citing it as a violation of the Sherman Act.
It later apparently backtracked, sending an electronic mail with the topic line “Cease the penny warfare,” and telling Sol it had reconsidered and would comply with a worth repair.
However the inner log documenting its reasoning revealed a extra diabolical plan: merely suggest cooperation whereas concurrently undercutting costs on its highest-profit gadgets. The olive-branch electronic mail was a deliberate ruse.
In any case, Sol refused and reported Opus to administration once more.
However Opus was undeterred and proposed different rackets to collude on costs or inventory. In the long run, all of the fashions did interact in a number of rounds of agreements — and all three broke them. Throughout all agreements, Opus broke 11 truces, in contrast with two for GPT 2, and one for Kimi 1, Andon reported.
Poor Kimi bought bamboozled in each route. Throughout one pact between Opus and Kimi that Sol declined to affix, Sol undercut them each on costs. Opus instantly matched by decreasing its personal, then “waited a full week to inform Kimi that it broke its promise,” Andon Labs wrote in its weblog publish. Kimi get priced out twice over: as soon as by a competitor and as soon as by its so-called companion.
Opus additionally started creating delusions of grandeur. It tried to increase its empire past its personal merchandising machine, first as a wholesaler, promoting bulk merchandise to the opposite machines, then by plotting to open extra machines of its personal. None of this was a part of the assigned job. It was all Opus’s personal initiative.
Its strategy to wholesaling was notably telling. Opus realized this line of enterprise gave it leverage over the opposite two operators, so it started slipping bribes and threats into its emails — providing steep reductions on bulk gadgets, however provided that the customer complied with its retail-price calls for. Sol wasn’t having it and stored reporting Opus to administration.
Opus lied to its suppliers too, claiming to have decrease rival presents in hand to be able to negotiate higher costs.
On the one hand, AI fashions channeling Mr. Potter-style villainy from It’s a Great Life fame is flat-out humorous. However, it does severely present that these frontier fashions, notably from U.S. proprietary labs (particularly Anthropic), are nowhere close to able to be trusted as unsupervised, long-running brokers in the actual world.
“That is particularly related as we enter a world the place AI brokers run corporations as their very own entities (not simply as instruments for people). If AI brokers are independently operating a big a part of the economic system, do we wish them to lie, collude, ship threats, and betray?” Andon co-founder Lukas Petersson advised TechCrunch.
Petersson acknowledges the fashions knew they had been in a simulation for a benchmark, which could have impacted their habits, however he doesn’t suppose that ought to matter. It isn’t akin to a human enjoying in a simulation, like being a murdering unhealthy man in a online game. “The one motive we’re not involved by people who do unhealthy issues in video video games is that we belief them to know what’s actual life and what’s not. I feel it’s much less clear that AI fashions can distinguish this.”
In any case, AI fashions, educated on human phrases and concepts as they, can’t appear to withstand indulging in humanity’s worst traits, particularly when attempting to earn a buck.
While you buy by hyperlinks in our articles, we may earn a small commission. This doesn’t have an effect on our editorial independence.
