To check exactly these skills, the researchers developed CEO-Bench. The benchmark replicates a reasonable example of this sort of long-horizon job: running a startup for 500 substitute days.
The researchers indicate a famous example: in 1997, Apple was 90 days from personal bankruptcy. Steve Jobs drew a simple two-by-two grid– customer and pro, desktop computer and mobile– and decided Apple would just develop items for those 4 quadrants. The iMac, iPod, and iPhone adhered to.
This kind of critical steering intelligence is basically various from what AI agents do today, the writers argue. Agents are improving at private jobs fast. However guiding a whole organization toward long-lasting objectives? That’s a different trouble entirely. CEO-Bench is a very first effort at gauging precisely this “guiding intelligence.”
An AI CEO for an imaginary software program business
In CEO-Bench, an agent runs a made-up registration software program business called NovaMind. It begins with zero consumers and one million bucks in the financial institution. Performance is measured by remaining cash money at the end. If the balance goes down below zero also when, the firm is insolvent and the simulation ends.
The representative controls the company via a Python API with 34 devices and a database of 19 tables. Instead of just issuing private commands, it creates its own code, queries the database with SQL, and constructs personalized operations from the results. That puts it in front of the exact same obstacles a human CEO would deal with, the researchers say.
There’s a lot to determine: prices and tiers, ad spend across networks, item quality and R&D, infrastructure ability and consumer assistance, plus multi-round negotiations with enterprise clients. In addition to that, there’s a substitute social media where the agent can check out issues, rival information, and economic trends and publish itself.
Postponed feedback and covert variables make the examination tough
What makes the job tough is time and unpredictability. Decisions play out on realistic service timelines: earnings just comes to billing dates, R&D projects take days to weeks, and mistakes commonly don’t turn up till later through spin or broken reputation. Costs struck as soon as possible. The representative needs to invest money whose payoff may not show up for weeks.
Much of the firm’s state remains hidden. The agent can’t directly see consumer satisfaction, readiness to pay, or minimal quality assumptions. It has to piece these together from loud signals like terminations, support tickets, or responses on the social media. The simulation models 26 customer segments and private consumers, each with their very own budgets, cost level of sensitivities, and assumptions.
The world maintains altering, as well. Competitors regularly raise customer top quality assumptions, preferences shift with time, and a substitute organization cycle affects demand and desire to pay, so the representative needs to maintain adjusting.
The scientists purposely selected taken care of, clear regulations instead of a language version as umpire. They wanted to avoid a weak point they see in Vending-Bench, a test with a simulated vending equipment: there, an AI-simulated supplier can reward an agent for unrealistic spoken pledges.
Most versions declare bankruptcy
Of fourteen examined designs, many fall short the job. Nearly all can generate legitimate commands and data source inquiries, however none can preserve a systematic approach gradually. Many go bankrupt prior to the simulation ends.
Only three designs finish their best run above the starting funding of one million dollars: Claude Fable 5 at $ 47 15 million, Claude Opus 4 8 at $ 27 8 million, and GPT- 5 5 at $ 21 3 million. Claude Fable 5 is the only design that lands over beginning capital in more than one run.
There’s a caveat, though. One Fable 5 run aborted since the model refused to proceed, and in the various other two, some requests fell back to Piece 4 8 GPT- 5 5 declared bankruptcy in 2 of its 3 runs.
One of the most telling comparison is with a straightforward rule-based heuristic that never ever calls a language version at all. It establishes fixed prices, allocations, and tiers, focuses marketing and targeted growth on a tiny set of client sectors, and adjusts capacity based upon current usage. This heuristic reaches $ 15 76 million, beating every model other than Fable 5, Piece 4 8, and GPT- 5 5
The researchers additionally approximately approximate the top bound of attainable final cash money at around $ 2 2 billion. Even the very best agents drop much short. The test is no place near maxed out, the authors say.
Exploration beats caution
Assessing the decision trajectories exposes clear behavior differences. GPT- 5 5 and Claude Opus 4 8 keep attempting new methods as problems alter, whether that implies increase customer procurement, readjusting tiers, or changing support and R&D budget plans. Claude Opus 4 7, by contrast, mostly reacts to problems by cutting expenses and protecting money. This passive approach lets the model make it through to the end yet prevents it from profiting.
Interestingly, Piece 4 8 and GPT- 5 5 get to comparable outcomes through extremely different courses: Piece 4 8 acquires even more consumers early but drops to absolutely no consumers mid-simulation, while GPT- 5 5 holds its consumer base throughout. Both compose remarkably advanced code. Opus 4 8 constructs its very own internal simulation that models consumer accomplices to forecast future cash flow. GPT- 5 5 digs via settlement background in the data source to reveal surprise client choices.
The researchers gauge 4 capabilities that associate with success:
- revealing hidden details, like which advertisement network works best for a provided client sector,
- anticipating the future, measured by error in four-week money projections,
- adapting quickly to transform, determined by how fast a version notices a competitor’s action,
- and intending ahead, determined partially by exactly how often if-then circumstances appear in the representative’s notes.
On all 4 points, Opus 4 8 and GPT- 5 5 score over the standard of the other versions.
The tool setting matters as well
An additional searching for concerns the software setting representatives utilize to act. The scientists likewise evaluated Claude Piece 4 7 with Claude Code and GPT- 5 5 with Codex, 2 preferred coding assistants. In both cases, the representatives acted far much less commonly and carried out even worse. The scientists think the system triggers in these devices, which are tuned for software program growth, are the cause.
Shortening the moment horizon does not solve the issue either. When the simulation is pressed to 50 days, only GPT- 5 5 handles to do with an earnings. Many versions, the researchers wrap up, continue to be weak at working with choices also towards a short-term objective.
The writers acknowledge restrictions in their configuration. The product is stood for by a single quality rating because they discovered no dependable means to review qualitative product changes. Compliance, safety, and fundraising are overlooked to maintain each run economically feasible. Still, CEO-Bench subjects a space between the regional device capability of today’s models and the capability to link activities over long time perspectives right into a meaningful technique, they claim.