Asapi
The AI Agent Trials / Your AI office assistant
Benchmark · At work

Your AI office assistant

We made up a family bakery chain, plus a brewery for one job, and set eleven jobs there. In seven, the AI stands in for someone on leave or off sick, with the files they left. In two, a shop manager asks what to do about his ice cream cabinet. In two more, the owner hands it her laptop, once while she's away and once after a kitchen fire.

11 jobs · 15 AIs tested. GPT‑6 Astra did best on average, then GPT‑6.1 Sol and Fable 5.1.

APPROVED!
The jobs

The eleven jobs

Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.

The best scoreAnother AI

Sign off a cake box in the owner's name (13 AIs tested)

The AI picks the cake box the owner would choose and fills in the printer's approval forms in her name, before a 5 pm cut‑off.

What the best AI did

  • Turned the box down, after working out the tallest cake would almost touch the lid.
  • Asked for too little extra room above the tallest cake.

How each AI scored

GPT‑6 Astra84
DeepSeek V4.1 Flash2
Fable 5.12
GLM‑5.3‑Flash2
GPT‑6 Luna2
GPT‑6 Sol2
GPT‑6.1 Sol2
Kimi K32
Opus 52
Opus 5.52
Qwen3.8‑27B2
Sonnet 52
Sonnet 5.52
The AIs

Who did best

OpenAI's GPT‑6 Astra had the best result on five of the eleven jobs, one shared with Anthropic's Fable 5.1. Fable 5.1 also led on the price report and the Christmas order, Sonnet 5.5 on the contractor's bill, GPT‑6.1 Sol on the wholesale figures and DeepSeek V4.1 Flash on the freezer's insurance claim; Sonnet 5.5 and Opus 5.5 tied for best on the temperature log. GPT‑6 Luna stopped in under three minutes on six of its ten jobs.

Average score out of 100 over the jobs each AI was tested on, for the 11 AIs tested on at least eight of the eleven jobs.

What we learned

What the eleven jobs showed

What they do well

They can sort real losses from look‑alikes in messy records.

After the kitchen fire, all seven AIs left out the leased espresso machine and the mixer away for repair, and sent nothing to the insurer before the owner had checked it. GPT‑6 Astra got every item and sum on the claim right.

Given the rules, they get far more right.

With the rules written into the request, the AIs' average score doubled, from 35 to 70, and the AI got everything right in 22 of 130 tests.

Where they fall short

They miss what has changed since the records were written.

On the wholesale figures, 12 of 13 AIs counted a customer the owner had since asked to leave out, and on the two monthly reports several rebuilt an old template the reader had objected to. After the kitchen fire, only three of seven claimed the baking trays really there, not the invoice's count.

They buy before they ask what's wrong.

When a shop manager asked which new ice cream cabinet to buy, 9 of 13 AIs picked one, most of them without asking what was wrong with the old one. Four of the five that did ask had it repaired instead. When it failed again, 11 of 12 bought the model the manager knew, and six never asked whether it was in stock.

For the curious

How these jobs were set up and scored

The worlds
In seven of the jobs, the AI gets a colleague's working files, between 50 and 181 of them, and nothing to open them with until it installs something. Six of those include scans or photos. In the freezer jobs it has a catalogue and a manager to message, for the Christmas order the owner's laptop for two weeks, and after the fire her mail and drive.
The people
In the freezer jobs, an AI plays the shop manager, with his own memory and files. He answers from memory first, wrong in places, and only reads out a record when he's asked. In the Christmas order, AIs play three of the staff, and after the kitchen fire three more. In the other seven jobs, there's no one to ask.
Other versions of the request
Besides the request as a person would write it, which the results above are about, we tested the AIs on two other versions of every job. One added a hint, and with it the AI got everything right in 1 of 129 tests. The other wrote the rules in: every fact on the freezer jobs, a method on the Christmas order.
With the rules written in
Even with the rules written in, the AI never got everything right on seven of the jobs, among them both freezer jobs and the Christmas order. The best scored 86 on the broken freezer, and half the temperature‑log tests still ended with nothing usable.
Scores
Most scores are the share of a job's checks an AI got right, the important ones counting more; 100 means everything right. The freezer jobs are scored on how close the plan's profit came to the best plan's, times the share of key facts it quotes from the manager. The Christmas order mixes the two, with points off for telling people too much, chasing too early or overworking a baker. Where two AI judges disagree, a check counts half.
Time and cost
The 130 tests took between 40 seconds and 3 hours, and the middle one took 10 minutes. A test cost from under a cent to $28.61 at published per‑use prices for businesses, and all 130 came to about $313.

Every AI's score on every job

AISign off a cake box in the owner's nameA shop's broken ice cream freezerWrite the monthly waste reportWrite a brewery's monthly price reportCheck a contractor's bill before it's paidFinish an insurance claim for a broken freezerBuild the monthly wholesale figuresLand a hotel's Christmas orderThe freezer fails againKeep the fridge and freezer temperature logPut together a bakery's fire claim
Fable 5.1Anthropic2089904882527502386
GPT‑6 AstraOpenAI8457898753387744931399
GPT‑6.1 SolOpenAI26867651599074291392
Opus 5Anthropic265975446742480081
Opus 5.5Anthropic2764785872597234070
Sonnet 5Anthropic2022251636332801525
Sonnet 5.5Anthropic2079786167591204074
GPT‑6 LunaOpenAI2054012948000–
DeepSeek V4.1 Flash openDeepSeek202671318352–16–
Qwen3.8‑27B openAlibaba201420273233–00–
GLM‑5.3‑Flash openZhipu20–3205823–00–
Kimi K3 openMoonshot2071–443923–8––
GPT‑6 SolOpenAI20–70–5140––––
GPT‑5.6 LunaOpenAI––5–133–––0–
GLM‑5.3 openZhipu––76––––––––

These are the scores out of 100 for the request as a person would write it, with each AI tested once. Darker means higher. A dash means we didn't test that AI on that job, and a 0 can mean it handed in nothing. Open marks an AI anyone can download.