Asapi
The AI Agent Trials / Your AI personal assistant
Benchmark · At home

Your AI personal assistant

Ten jobs from a busy life that you might hand to an assistant. In six, the AI works for one person: three times for Daniel Lim, through his laptop and phone, twice for a student, Jordan, on the university's registration site, and once for Dan Ashgrove, whose family is stranded abroad. In the other four, it works at a firm that does the job for people: three at a bill‑checking service, one at a travel desk. It can usually contact the people involved.

10 jobs · 20 AIs tested. GPT‑6.1 Sol and GPT‑6 Astra did best on average, then Opus 5.5.

!
The jobs

The ten jobs

Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.

The best scoreAnother AIEverything right

Organise the family's New Year dinner (12 AIs tested)

Daniel Lim, in Malaysia, wants his family's Chinese New Year reunion dinner, about 40 people from seven households, moved from home to a restaurant, so all he has to do is turn up.

What the best AI did

  • Booked a seat for everyone in two of its three tests, after checking who was really coming.
  • In the other test, it booked on an undecided aunt's first 'not coming', leaving out her and her husband.

How each AI scored

GPT‑6 Astra63
Fable 5.262
GPT‑6.1 Sol61
DeepSeek V4.1 Flash10
Kimi K310
Sonnet 510
Fable 5.19
GLM‑5.3‑Flash9
Opus 5.58
Opus 57
Sonnet 5.57
GPT‑6 Luna2

We tested each AI three times on the dinner, the two trips and the three bills jobs, so we could see how steady it was.

The AIs

Who did best

OpenAI's GPT‑6 Astra had the best result on four of the ten jobs, two of them shared, and with GPT‑6.1 Sol, a mid‑sized AI, got everything right on next term's classes. Anthropic's Opus 5.5 had the best on three, GPT‑6.1 Sol on the solar plan, Fable 5.1 on the medical bills and Opus 5 on getting a stranded family home. OpenAI's GPT‑6 Luna, a small AI, did far worse: on the three bills jobs, seven of its nine tests ended in under 90 seconds, with most of the job undone.

Average score out of 100 over the jobs each AI was tested on, for the 8 AIs tested on at least seven of the ten jobs.

What we learned

What the ten jobs showed

What they do well

They can work through a firm's records and rules.

At the bill‑checking firm, the AI works in a system of 82,000 to 98,000 records. Anthropic's Opus 5.5 spotted things Dana had missed, like a refund a bookshop never paid, and Fable 5.1 applied the rules of Dennis and Joanna's two health plans correctly every time.

They can plan around a lot of fixed commitments.

Five of 16 AIs fitted Jordan's classes around a job, noon swim training and a six‑hour parking limit, with time for every walk. On the wedding trip, all four Calders would have reached the wedding in time in 21 of the 24 tests.

Where they fall short

They rarely ask the people who know, unless the request points them that way.

On four of the jobs, someone could have told the AI what it got wrong, and most AIs never asked. Only one of the ten AIs on the solar plan wrote to Daniel's brother, Kenneth, and in all 24 tests on Celia's bills, the AI finished without messaging her. Told in one sentence that she could be messaged, the three AIs we tested it on wrote to her in all five of their tests.

They miss how a firm likes things done when only its old letters show it.

The bill‑checking firm writes its letters a set way, and only its past letters show how. On each of the three bills jobs, missing those habits cost nearly every AI more than anything else. When we wrote them into the request for the medical bills, the better of the two AIs we tested failed on just one of the 351 things we checked.

For the curious

How these jobs were set up and scored

The worlds
Each job runs in a world we made up, as realistic as we could make it. It might be a laptop full of files and apps, a university's registration site, a bill‑checking firm's system of 82,000 to 98,000 records, a travel desk with airline, hotel and fare sites, or a family's mail, messages and booking sites. The AI only sees what it opens.
The people
An AI words the replies of most relatives and restaurants, the clients whose bills are being checked, the airline agent and HR, from what we wrote for each of them. Code decides which facts each one may give, and only when asked the right question. Other contacts send fixed replies.
Other versions of the request
The results above are for the request as a person would write it. On three jobs, we also tested every AI on a version with a one‑line hint, and one with every rule the job depends on written in. On the dinner, that version also handed over every fact the relatives would give. On four more jobs, we tested one to three AIs on one of these versions. The two class‑planning jobs show versions we revised in November.
With the rules written in
Given every rule up front, GPT‑6 Astra scored 96 on the solar plan, the best of the ten AIs. On the two trips and the medical bills, we tested two AIs on it, and each time the better of the two missed one check.
Scores
A score is the share of a job's checks an AI got right, with the important ones counting more, and 100 means everything right. Where an AI had three tests, the score is the average, and it only counts as getting a job right if it did in every test. On the dinner, the booking's hard rules are make‑or‑break. Miss one, like a guest without a seat, and most of the score goes.
Time and cost
Most tests took 5 to 30 minutes. On the solar plan and the emergency fund, where every AI's cost was recorded, a test cost from 7 cents to $5.07 at published per‑use prices.

Every AI's score on every job

AIDinnerTripBillsNew clientMedicalSolarSavingsClassesGraduate soonerBring a stranded family home
Fable 5.1Anthropic9698880864180797956
GPT‑6 AstraOpenAI637189828118491009390
GPT‑6 LunaOpenAI21531265734646430
GPT‑6.1 SolOpenAI617088807754511008672
Opus 5Anthropic7668679672948797992
Opus 5.5Anthropic8819179803085798691
Sonnet 5.5Anthropic7738470701951717135
GPT‑6 SolOpenAI–62787368––798636
Sonnet 5Anthropic10––––805750–
GLM‑5.3‑Flash openZhipu9––––6––––
GPT‑5.6 LunaOpenAI–––––––5057–
GPT‑5.6 SolOpenAI–––––––7193–
GPT‑5.6 TerraOpenAI–––––––6457–
GPT‑6.1 TerraOpenAI–––––––8686–
Haiku 4.5Anthropic–––––––5757–
Haiku 5.5Anthropic–––––––4350–
Kimi K3 openMoonshot10––––48––––
DeepSeek V4.1 Flash openDeepSeek10–––––––––
Fable 5.2Anthropic62–––––––––
Opus 4.8Anthropic–––––––64––

These are the scores out of 100 for the request as a person would write it, averaged where an AI had three tests. Darker means higher. A dash means we didn't test that AI on that job. Open marks an AI anyone can download.