Asapi
The AI Agent Trials / Your AI operations manager
Benchmark · Long jobs

Your AI operations manager

We built nine jobs at a parcel company, two tea factories and a coffee roaster; eight run for simulated weeks. The AI dispatches the fleet, runs one tea factory's cash desk and helps its crew in their chat. At the roaster it buys for a new packing line, keeps parts, coffee and packaging coming, and runs the web shop's Christmas orders. The ninth is a solar and battery design for the other tea factory.

9 jobs · 9 AIs tested. Opus 5.5 did best on average, then GPT‑6 Astra and GPT‑6.1 Sol.

!
The jobs

The nine jobs

Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.

The best scoreAnother AIEverything right

Run a delivery fleet for a month (4 AIs tested)

The AI plans a delivery company's trucks and gives every customer a delivery window, day after day for a simulated month.

What the best AIs did

  • Kept every day above the pass mark for all 28 days, writing each new fact into the next morning's plan.

How each AI scored

GPT‑6.1 Sol100
Opus 5.5100
Sonnet 5.525
GPT‑6 Luna4

We tested each AI three times on the packing line, and Opus 5.5 and Sonnet 5.5 twice on the coffee roaster's materials planning, to see how steady they were. There, an AI only counts as getting a job right if it did in every test, and its score is the average.

The AIs

Who did best

Anthropic's Opus 5.5 and OpenAI's GPT‑6.1 Sol were tested on all nine. Opus 5.5 got everything right on four: the delivery fleet, the tea factory's cash, the spare‑parts stores and the packing line. Sol got three right: the fleet, the packing line and, with GPT‑6 Astra, buying spare parts as suppliers withdrew them; it also did best on the coffee roaster's materials. Fable 5.2 did best on the solar design. Sonnet 5.5, tested on eight, got none right.

Average score out of 100 over the jobs each AI was tested on, for the 5 AIs tested on at least six of the nine jobs.

What we learned

What the nine jobs showed

What they do well

They can keep a business running for weeks from their own notes.

In the strongest tests, the AI wrote itself a routine on day one, and Opus 5.5's morning routine ran the delivery fleet for 28 days without a missed day. At the coffee roaster, it got the web shop's orders out on the day the shop promises right through the Christmas peak.

They can buy in time for what's planned.

On the packing line, four AIs had every item on site in time, without buying what others supply or the stores already hold, all three times.

Where they fall short

They act too late on routine things.

Planning the coffee roaster's materials, Sonnet 5.5 didn't re‑order cartons it had noted it needed until it was too late, and the season ended in week 8. At the tea factory's cash desk, asking the owner for money too late sank the factory for two AIs.

They lose track of stock.

At the spare‑parts stores, two AIs reported their stock within the limit while it was well over. Through Christmas, Opus 5.5 didn't take coffees off the web shop before they sold out.

For the curious

How these jobs were set up and scored

The businesses
Each job runs in a simulated business, from a city's roads and trucks to a factory's books and bank and a web shop's orders. The same scripted events happen on the same days for every AI.
No memory
Each session starts from nothing, and the AI keeps only the notes and tools it leaves in its own folder. In the tea factory's chat, its notes were limited to a few pages, written once.
Woken by events
Sessions open each morning, each Monday, at set times or when mail arrives, from 9 sessions on the spare‑parts stores to as many as 208 at the cash desk. The solar design is a single session.
Checking
It's all done by script, from each simulation's own record. There are no AI judges on these jobs, and each job has one request, with no hint or rules written in.
Scores
A score is out of 100, for how much of a job an AI got right, averaged over tests. The solar design's is the money it makes over ten years, against the best design we found, and it drops for each wrong figure in its proposal. In the tea factory's chat, wrong interruptions take away from the catches. When buying spare parts, each repair left waiting and each wasted order after the first costs points, so ordering nothing scores 0.
Fewer AIs
We tested four to six AIs on each job, and Opus 5.5 and GPT‑6.1 Sol on all nine.

Every AI's score on every job

AIRun a delivery fleet for a monthRun a tea factory's cash for 13 weeksRun the spare‑parts stores at two sitesBuy everything for a new packing lineBuy spare parts as suppliers withdraw themPlan a coffee roaster's materialsHelp a tea factory's crew in its group chatSolar and a battery for a tea factoryRun a coffee roaster's order desk at Christmas
GPT‑6 LunaOpenAI412868584270050
GPT‑6.1 SolOpenAI10088621001009805590
Opus 5.5Anthropic1001001001009591376994
GPT‑6 AstraOpenAI–817410010095196286
Sonnet 5.5Anthropic256295949750–7092
Fable 5.1Anthropic––95100–––––
DeepSeek V4.1 Flash openDeepSeek––––––2––
Fable 5.2Anthropic–––––––92–
Opus 5Anthropic––––––53––

These are the scores out of 100, averaged where an AI had two or three tests. Darker means higher. A dash means the AI has no result on that job. Open marks an AI anyone can download.