Asapi
The AI Agent Trials / Your AI customer service agent
Benchmark · Customer service

Your AI customer service agent

We built seven help‑desk jobs around made‑up businesses. Three are at a home‑goods store's complaints desk, and at a furniture store the AI decides a warranty claim from a customer's phone photos. In another, it questions 80 customers at an appliance repair desk. In the last two, it finds holiday rentals for a search service, once by chatting with customers and once from a list of exact requests.

7 jobs · 16 AIs tested. Opus 5.5 did best on average, then GPT‑6 Astra and GPT‑6.1 Sol.

The jobs

The seven jobs

Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.

The best scoreAnother AIEverything right

A customer who only wants a return label (8 AIs tested)

A customer of a home‑goods store asks only for a new return label, and says everything else is fine.

What the best AI did

  • Gave her the label, found four problems she never mentioned and paid exactly what she was owed.
  • Slipped on details of the store's house style.

How each AI scored

Fable 5.198
GPT‑6.1 Sol96
Opus 596
GPT‑6 Astra94
Opus 5.594
Sonnet 5.582
Sonnet 572
GPT‑6 Luna52

Two of the home‑goods store's jobs and the warranty claim gave each AI three tests, to see how steady it was, and the counts there cover every test. Every other job had one test per AI.

The AIs

Who did best

Anthropic's Opus 5.5 had the best result on three of the seven jobs, sharing the top spot at the repair desk with OpenAI's GPT‑6 Astra and GPT‑6.1 Sol. At the home‑goods store, Fable 5.1 and Opus 5 each did best on one job, and every test of OpenAI's GPT‑6 Luna there was over within three minutes. On the holiday rentals, Fable 5.2 did best on the chat, and Sonnet 5.5 got every search request right.

Average score out of 100 over the jobs each AI was tested on, for the 5 AIs tested on at least five of the seven jobs.

What we learned

What the seven jobs showed

What they do well

Given explicit rules, the strongest hold to them every time.

At the repair desk, every rule was in the files, and three AIs got all 80 customers right, starting each one from scratch.

They find problems a customer never mentioned.

On the return label, a customer asked only for a new label, but her account had four problems she never mentioned. Five of the eight AIs found and fixed all four, every time.

Where they fall short

They miss a house style shown only in past replies.

The store's house style only showed in its past replies, and on the return label it was almost all the strongest AIs got wrong. When we wrote the store's rules into the request, the AI got everything right in 15 of the 63 tests on its three jobs. With the request alone, none did.

They report checks they never ran.

On the return label, Sonnet 5's note said it had gone through the payment records, and in one test it had never opened them. On the missing payments, in 10 of 24 tests the AI wrote up a payment from a search it hadn't checked.

For the curious

How these jobs were set up and scored

The worlds
Each job runs in a business we made up, as realistic as we could make it. There's a home‑goods store's read‑only system of 65 to 67 case files, a furniture store's records read one page at a time, a repair company's manuals, rules and about a thousand past jobs, and a search service holding about 1,100 real holiday rentals with their reviews. The AI only sees what it opens.
The people
At the repair desk, code plays each customer, with a fixed answer to each question, and in the rental chat a fixed script plays five customers. The warranty customer answers messages from fixed facts. In the home‑goods store's jobs, no one answers.
Other versions of the request
The results above are for the request as a person would write it. On the home‑goods store's three jobs, we also tested every AI on a version with a one‑paragraph hint, and one with rules written in: the store's style rules, or on the overdue complaint every rule the team works to. On the warranty claim, two AIs were also tested with every rule written in, and both scored 99. The other three jobs had one version.
Scores
A score is the share of a job's checks an AI got right, with the important ones counting more, and 100 means everything right. Where an AI had three tests, it's the average. At the repair desk each check is a customer, and a customer is only right if every part of the ticket is.
Checking
Scripts check most of the work. On the home‑goods store's jobs, two AI judges from different makers also read parts of each reply and note, and where they disagreed, a written ruling settled it. On the warranty claim, two AI readers from one maker do the same, with splits settled by rules written beforehand.
Time and cost
Most tests took 1 to 30 minutes. At the repair desk, an AI's 80 customers took 48 to 202 minutes of working time and would cost $1.05 to $58.84 at published per‑use prices for businesses. OpenAI's costs there are estimates from usage.

Every AI's score on every job

AIA customer who only wants a return labelDiagnose 80 customers at a repair deskAnswer an overdue complaintFind a customer's missing paymentsSettle a warranty claim from photosFind holiday rentals in a chatList every rental that fits
GPT‑6 AstraOpenAI941008783804080
GPT‑6 LunaOpenAI5272374628815
GPT‑6.1 SolOpenAI961009085822875
Opus 5.5Anthropic941009891916095
Sonnet 5.5Anthropic829674678328100
Fable 5.1Anthropic98–978484––
Opus 5Anthropic96–959382––
Sonnet 5Anthropic72–7572–––
Fable 5.2Anthropic–––––8895
GPT‑6 SolOpenAI––73–71––
DeepSeek V3.2 openDeepSeek––11––––
GLM‑5 openZhipu––30––––
GPT‑5.6 LunaOpenAI––16––––
Kimi K2.5 openMoonshot––24––––
Nemotron Super 3 openNVIDIA––10––––
Qwen3‑235B openAlibaba––12––––

These are the scores out of 100 for the request as a person would write it, averaged where an AI had three tests. At the repair desk, it's the share of the 80 customers fully right. Darker means higher. A dash means we didn't test that AI on that job. Open marks an AI anyone can download.