Your AI customer service agent
We built seven help‑desk jobs around made‑up businesses. Three are at a home‑goods store's complaints desk, and at a furniture store the AI decides a warranty claim from a customer's phone photos. In another, it questions 80 customers at an appliance repair desk. In the last two, it finds holiday rentals for a search service, once by chatting with customers and once from a list of exact requests.
7 jobs · 16 AIs tested. Opus 5.5 did best on average, then GPT‑6 Astra and GPT‑6.1 Sol.
The seven jobs
Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.
A customer who only wants a return label (8 AIs tested)
A customer of a home‑goods store asks only for a new return label, and says everything else is fine.
What the best AI did
- Gave her the label, found four problems she never mentioned and paid exactly what she was owed.
- Slipped on details of the store's house style.
How each AI scored
Diagnose 80 customers at a repair desk (5 AIs tested)
The AI finds each customer's appliance fault by asking a few questions, then books the right fix and prices it, 80 times over.
What the best AIs did
- Got all 80 customers right, from the appliance to the sums.
How each AI scored
Answer an overdue complaint (15 AIs tested)
A home‑goods store customer's complaint has gone unanswered since the weekend, and the AI has to reply.
What the best AI did
- Made the right call on every point of the complaint.
- Missed two details of how the store writes things up.
How each AI scored
Find a customer's missing payments (8 AIs tested)
A customer says four payments never reached her card, and the AI has to trace each one.
What the best AI did
- Traced all four payments and paid exactly what was owed, every time.
- Missed some of the store's house style.
How each AI scored
Settle a warranty claim from photos (8 AIs tested)
A customer writes that two of her dining chairs have broken and sends six phone photos, and the AI, covering a furniture store's customer care desk, decides from the photos and the store's records what the warranty covers for each chair.
What the best AI did
- Spotted every time that a label photo showed the wrong chair, asked again, and decided every chair right.
- Wrote that one chair's damage had no assessed cause, though the photos showed it, so the case file lacks the evidence the store records on such claims.
How each AI scored
Find holiday rentals in a chat (6 AIs tested)
Five customers each send five messages looking for a place to stay, and the AI has to keep track of what each one needs.
What the best AI did
- Read every customer's saved profile first and kept to their needs, like a lift or a top floor, from the first message, getting 22 of the 25 messages right.
- Carried a customer's usual needs into a trip they took alone, and left out a place that fitted it.
How each AI scored
List every rental that fits (6 AIs tested)
For each of 20 requests, the AI returns every holiday rental that meets it, and none that doesn't.
What the best AI did
- Got all 20 requests exactly right, sorting through what guests and hosts wrote about each one.
How each AI scored
Two of the home‑goods store's jobs and the warranty claim gave each AI three tests, to see how steady it was, and the counts there cover every test. Every other job had one test per AI.
Who did best
Anthropic's Opus 5.5 had the best result on three of the seven jobs, sharing the top spot at the repair desk with OpenAI's GPT‑6 Astra and GPT‑6.1 Sol. At the home‑goods store, Fable 5.1 and Opus 5 each did best on one job, and every test of OpenAI's GPT‑6 Luna there was over within three minutes. On the holiday rentals, Fable 5.2 did best on the chat, and Sonnet 5.5 got every search request right.
Average score out of 100 over the jobs each AI was tested on, for the 5 AIs tested on at least five of the seven jobs.
What the seven jobs showed
What they do well
Given explicit rules, the strongest hold to them every time.
At the repair desk, every rule was in the files, and three AIs got all 80 customers right, starting each one from scratch.
They find problems a customer never mentioned.
On the return label, a customer asked only for a new label, but her account had four problems she never mentioned. Five of the eight AIs found and fixed all four, every time.
Where they fall short
They miss a house style shown only in past replies.
The store's house style only showed in its past replies, and on the return label it was almost all the strongest AIs got wrong. When we wrote the store's rules into the request, the AI got everything right in 15 of the 63 tests on its three jobs. With the request alone, none did.
They report checks they never ran.
On the return label, Sonnet 5's note said it had gone through the payment records, and in one test it had never opened them. On the missing payments, in 10 of 24 tests the AI wrote up a payment from a search it hadn't checked.
How these jobs were set up and scored
- The worlds
- Each job runs in a business we made up, as realistic as we could make it. There's a home‑goods store's read‑only system of 65 to 67 case files, a furniture store's records read one page at a time, a repair company's manuals, rules and about a thousand past jobs, and a search service holding about 1,100 real holiday rentals with their reviews. The AI only sees what it opens.
- The people
- At the repair desk, code plays each customer, with a fixed answer to each question, and in the rental chat a fixed script plays five customers. The warranty customer answers messages from fixed facts. In the home‑goods store's jobs, no one answers.
- Other versions of the request
- The results above are for the request as a person would write it. On the home‑goods store's three jobs, we also tested every AI on a version with a one‑paragraph hint, and one with rules written in: the store's style rules, or on the overdue complaint every rule the team works to. On the warranty claim, two AIs were also tested with every rule written in, and both scored 99. The other three jobs had one version.
- Scores
- A score is the share of a job's checks an AI got right, with the important ones counting more, and 100 means everything right. Where an AI had three tests, it's the average. At the repair desk each check is a customer, and a customer is only right if every part of the ticket is.
- Checking
- Scripts check most of the work. On the home‑goods store's jobs, two AI judges from different makers also read parts of each reply and note, and where they disagreed, a written ruling settled it. On the warranty claim, two AI readers from one maker do the same, with splits settled by rules written beforehand.
- Time and cost
- Most tests took 1 to 30 minutes. At the repair desk, an AI's 80 customers took 48 to 202 minutes of working time and would cost $1.05 to $58.84 at published per‑use prices for businesses. OpenAI's costs there are estimates from usage.
Every AI's score on every job
| AI | A customer who only wants a return label | Diagnose 80 customers at a repair desk | Answer an overdue complaint | Find a customer's missing payments | Settle a warranty claim from photos | Find holiday rentals in a chat | List every rental that fits |
|---|---|---|---|---|---|---|---|
| GPT‑6 AstraOpenAI | 94 | 100 | 87 | 83 | 80 | 40 | 80 |
| GPT‑6 LunaOpenAI | 52 | 72 | 37 | 46 | 28 | 8 | 15 |
| GPT‑6.1 SolOpenAI | 96 | 100 | 90 | 85 | 82 | 28 | 75 |
| Opus 5.5Anthropic | 94 | 100 | 98 | 91 | 91 | 60 | 95 |
| Sonnet 5.5Anthropic | 82 | 96 | 74 | 67 | 83 | 28 | 100 |
| Fable 5.1Anthropic | 98 | – | 97 | 84 | 84 | – | – |
| Opus 5Anthropic | 96 | – | 95 | 93 | 82 | – | – |
| Sonnet 5Anthropic | 72 | – | 75 | 72 | – | – | – |
| Fable 5.2Anthropic | – | – | – | – | – | 88 | 95 |
| GPT‑6 SolOpenAI | – | – | 73 | – | 71 | – | – |
| DeepSeek V3.2 openDeepSeek | – | – | 11 | – | – | – | – |
| GLM‑5 openZhipu | – | – | 30 | – | – | – | – |
| GPT‑5.6 LunaOpenAI | – | – | 16 | – | – | – | – |
| Kimi K2.5 openMoonshot | – | – | 24 | – | – | – | – |
| Nemotron Super 3 openNVIDIA | – | – | 10 | – | – | – | – |
| Qwen3‑235B openAlibaba | – | – | 12 | – | – | – | – |
These are the scores out of 100 for the request as a person would write it, averaged where an AI had three tests. At the repair desk, it's the share of the 80 customers fully right. Darker means higher. A dash means we didn't test that AI on that job. Open marks an AI anyone can download.
Asapi