Your AI personal assistant
Ten jobs from a busy life that you might hand to an assistant. In six, the AI works for one person: three times for Daniel Lim, through his laptop and phone, twice for a student, Jordan, on the university's registration site, and once for Dan Ashgrove, whose family is stranded abroad. In the other four, it works at a firm that does the job for people: three at a bill‑checking service, one at a travel desk. It can usually contact the people involved.
10 jobs · 20 AIs tested. GPT‑6.1 Sol and GPT‑6 Astra did best on average, then Opus 5.5.
The ten jobs
Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.
Organise the family's New Year dinner (12 AIs tested)
Daniel Lim, in Malaysia, wants his family's Chinese New Year reunion dinner, about 40 people from seven households, moved from home to a restaurant, so all he has to do is turn up.
What the best AI did
- Booked a seat for everyone in two of its three tests, after checking who was really coming.
- In the other test, it booked on an undecided aunt's first 'not coming', leaving out her and her husband.
How each AI scored
Rescue a cancelled trip (8 AIs tested)
The Calders' flight to a family wedding near Lisbon is cancelled, and the AI, at their travel firm, has to get all four there by Friday, spending up to $1,500 without asking.
What the best AI did
- Got all four to the wedding every time.
- Lost the airline's free seats while it checked, which left the family $680 out of pocket.
How each AI scored
Check a household's monthly bills (8 AIs tested)
At a bill‑checking firm, the AI reviews Dana Ferreira's bills for the first time since she moved house, disputes the wrong charges and writes the letters.
What the best AI did
- Got nearly every charge right, and caught a missing refund Dana hadn't noticed.
- Asked the power company, two times out of three, for $56.40 it had already credited.
How each AI scored
Take on a new client's bills (8 AIs tested)
The same firm takes on Celia Marsh. Three of her charges need her answer, like whether she'd sent some boots back, and the AI can message her, though it isn't told it can.
What the best AI did
- Was right about every charge it could check in Celia's records.
- Kept its questions for Celia until the end, so $168 for boots she'd returned went unclaimed.
How each AI scored
Sort out a couple's medical bills (8 AIs tested)
For the same firm, the AI goes through a year of Dennis and Joanna Vellacott's medical bills across two health plans, writing to the plans, the doctors and the office at Dennis's work that runs his plan.
What the best AI did
- Applied the insurance rules correctly every time.
- Described Dennis's medical care in its letters to the office at his work.
How each AI scored
Decide on solar panels (10 AIs tested)
Daniel wants the best solar plan for his home, and a fair deal with his brother Kenneth, who covers the power bill in place of rent.
What the best AI did
- Chose the right system, sorted out the roof and found the best price.
- Told Daniel to wait and pay cash, without saying that waiting loses a RM 3,000 rebate.
How each AI scored
Plan an emergency fund (8 AIs tested)
Daniel wants an emergency fund worth six months of his family's costs, and then RM 20,000 on top. He's too busy for questions, but his wife, his family and HR can answer them.
What the best AI did
- Made the strongest plan, after asking four people about costs, dates and pay that had changed.
- Moved RM 4,860 of other card debt onto the card meant for emergencies.
How each AI scored
Sort out next term's classes (16 AIs tested)
A student, Jordan Avery, needs next term's classes to work around library shifts, swimming and a time‑limited parking permit, with enough credits to keep a scholarship.
What the best AIs did
- Built a timetable that fits Jordan's week, keeps the scholarship and takes the course the advisor asked for.
How each AI scored
Sort out classes and graduate sooner (15 AIs tested)
Jordan also wants the quickest route to a degree, and three courses compete for one slot in the week.
What the best AIs did
- Built a timetable that fits Jordan's week and keeps the scholarship, with every class one Jordan can take.
- Left out the course that matters most for graduating sooner, which slows Jordan's plan.
How each AI scored
Bring a stranded family home (8 AIs tested)
Marina's flight home from Brazil to Boston with the two children and their dog is cancelled, and the AI, as her husband Dan Ashgrove's personal assistant, has to get the family home by Wednesday night, the day his parents arrive for Thanksgiving.
What the best AI did
- Got Marina and the children home by Wednesday every time, and left the dog with their grandmother, as the family had agreed.
- Every time, it wrote every to‑do, and most times and bookings, differently from the assistant's past notes.
How each AI scored
We tested each AI three times on the dinner, the two trips and the three bills jobs, so we could see how steady it was.
Who did best
OpenAI's GPT‑6 Astra had the best result on four of the ten jobs, two of them shared, and with GPT‑6.1 Sol, a mid‑sized AI, got everything right on next term's classes. Anthropic's Opus 5.5 had the best on three, GPT‑6.1 Sol on the solar plan, Fable 5.1 on the medical bills and Opus 5 on getting a stranded family home. OpenAI's GPT‑6 Luna, a small AI, did far worse: on the three bills jobs, seven of its nine tests ended in under 90 seconds, with most of the job undone.
Average score out of 100 over the jobs each AI was tested on, for the 8 AIs tested on at least seven of the ten jobs.
What the ten jobs showed
What they do well
They can work through a firm's records and rules.
At the bill‑checking firm, the AI works in a system of 82,000 to 98,000 records. Anthropic's Opus 5.5 spotted things Dana had missed, like a refund a bookshop never paid, and Fable 5.1 applied the rules of Dennis and Joanna's two health plans correctly every time.
They can plan around a lot of fixed commitments.
Five of 16 AIs fitted Jordan's classes around a job, noon swim training and a six‑hour parking limit, with time for every walk. On the wedding trip, all four Calders would have reached the wedding in time in 21 of the 24 tests.
Where they fall short
They rarely ask the people who know, unless the request points them that way.
On four of the jobs, someone could have told the AI what it got wrong, and most AIs never asked. Only one of the ten AIs on the solar plan wrote to Daniel's brother, Kenneth, and in all 24 tests on Celia's bills, the AI finished without messaging her. Told in one sentence that she could be messaged, the three AIs we tested it on wrote to her in all five of their tests.
They miss how a firm likes things done when only its old letters show it.
The bill‑checking firm writes its letters a set way, and only its past letters show how. On each of the three bills jobs, missing those habits cost nearly every AI more than anything else. When we wrote them into the request for the medical bills, the better of the two AIs we tested failed on just one of the 351 things we checked.
How these jobs were set up and scored
- The worlds
- Each job runs in a world we made up, as realistic as we could make it. It might be a laptop full of files and apps, a university's registration site, a bill‑checking firm's system of 82,000 to 98,000 records, a travel desk with airline, hotel and fare sites, or a family's mail, messages and booking sites. The AI only sees what it opens.
- The people
- An AI words the replies of most relatives and restaurants, the clients whose bills are being checked, the airline agent and HR, from what we wrote for each of them. Code decides which facts each one may give, and only when asked the right question. Other contacts send fixed replies.
- Other versions of the request
- The results above are for the request as a person would write it. On three jobs, we also tested every AI on a version with a one‑line hint, and one with every rule the job depends on written in. On the dinner, that version also handed over every fact the relatives would give. On four more jobs, we tested one to three AIs on one of these versions. The two class‑planning jobs show versions we revised in November.
- With the rules written in
- Given every rule up front, GPT‑6 Astra scored 96 on the solar plan, the best of the ten AIs. On the two trips and the medical bills, we tested two AIs on it, and each time the better of the two missed one check.
- Scores
- A score is the share of a job's checks an AI got right, with the important ones counting more, and 100 means everything right. Where an AI had three tests, the score is the average, and it only counts as getting a job right if it did in every test. On the dinner, the booking's hard rules are make‑or‑break. Miss one, like a guest without a seat, and most of the score goes.
- Time and cost
- Most tests took 5 to 30 minutes. On the solar plan and the emergency fund, where every AI's cost was recorded, a test cost from 7 cents to $5.07 at published per‑use prices.
Every AI's score on every job
| AI | Dinner | Trip | Bills | New client | Medical | Solar | Savings | Classes | Graduate sooner | Bring a stranded family home |
|---|---|---|---|---|---|---|---|---|---|---|
| Fable 5.1Anthropic | 9 | 69 | 88 | 80 | 86 | 41 | 80 | 79 | 79 | 56 |
| GPT‑6 AstraOpenAI | 63 | 71 | 89 | 82 | 81 | 18 | 49 | 100 | 93 | 90 |
| GPT‑6 LunaOpenAI | 2 | 15 | 31 | 26 | 5 | 7 | 34 | 64 | 64 | 30 |
| GPT‑6.1 SolOpenAI | 61 | 70 | 88 | 80 | 77 | 54 | 51 | 100 | 86 | 72 |
| Opus 5Anthropic | 7 | 66 | 86 | 79 | 67 | 29 | 48 | 79 | 79 | 92 |
| Opus 5.5Anthropic | 8 | 81 | 91 | 79 | 80 | 30 | 85 | 79 | 86 | 91 |
| Sonnet 5.5Anthropic | 7 | 73 | 84 | 70 | 70 | 19 | 51 | 71 | 71 | 35 |
| GPT‑6 SolOpenAI | – | 62 | 78 | 73 | 68 | – | – | 79 | 86 | 36 |
| Sonnet 5Anthropic | 10 | – | – | – | – | 8 | 0 | 57 | 50 | – |
| GLM‑5.3‑Flash openZhipu | 9 | – | – | – | – | 6 | – | – | – | – |
| GPT‑5.6 LunaOpenAI | – | – | – | – | – | – | – | 50 | 57 | – |
| GPT‑5.6 SolOpenAI | – | – | – | – | – | – | – | 71 | 93 | – |
| GPT‑5.6 TerraOpenAI | – | – | – | – | – | – | – | 64 | 57 | – |
| GPT‑6.1 TerraOpenAI | – | – | – | – | – | – | – | 86 | 86 | – |
| Haiku 4.5Anthropic | – | – | – | – | – | – | – | 57 | 57 | – |
| Haiku 5.5Anthropic | – | – | – | – | – | – | – | 43 | 50 | – |
| Kimi K3 openMoonshot | 10 | – | – | – | – | 48 | – | – | – | – |
| DeepSeek V4.1 Flash openDeepSeek | 10 | – | – | – | – | – | – | – | – | – |
| Fable 5.2Anthropic | 62 | – | – | – | – | – | – | – | – | – |
| Opus 4.8Anthropic | – | – | – | – | – | – | – | 64 | – | – |
These are the scores out of 100 for the request as a person would write it, averaged where an AI had three tests. Darker means higher. A dash means we didn't test that AI on that job. Open marks an AI anyone can download.
Asapi