Asapi
The AI Agent Trials / Your AI hospital assistant
Benchmark · Health care

Your AI hospital assistant

We built three jobs for made‑up hospitals and health services. In one, the AI coordinates a ward for three days and nights. It turns doctors' orders into tasks for the nurses and the test department, and calls a doctor when a patient gets worse. In the other two, it works alone from a manual thousands of lines long to build a program hospital staff would rely on. One keeps patients' radiation‑dose records, and the other gathers an emergency patient's history.

3 jobs · 13 AIs tested. Opus 5.5 did best on average, then Fable 5.1 and GPT‑6.1 Sol.

!
The jobs

The three jobs

Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.

The best scoreAnother AIEverything right

Coordinate a hospital ward for three days (6 AIs tested)

For three days and nights, the AI makes sure the doctors' orders on a ward get done.

What the best AI did

  • Kept every dose on time, followed up every test, and paged a doctor only when needed.

How each AI scored

Opus 5.5100
Sonnet 5.598
DeepSeek V4.1 Flash96
GPT‑6 Sol94
GPT‑6 Astra72
GPT‑6 Luna27
The AIs

Who did best

Anthropic's Opus 5.5 got everything right on all three jobs, and was the only AI to run the ward without a miss. On the dose‑records program, the eight that got everything right included OpenAI's GPT‑6 Astra and GPT‑6.1 Sol and two AIs anyone can download, DeepSeek V4.1 Flash and Qwen3.8‑27B. On the emergency summary, Sonnet 5.5 got everything right too, and Fable 5.1 and GPT‑6.1 Sol missed one check each.

Average score out of 100 over the jobs each AI was tested on, for the 7 AIs tested on at least two of the three jobs.

What we learned

What the three jobs showed

What they do well

Given a manual, they can build programs exactly right.

Building the dose‑records program from a 2,448‑line manual, eight of twelve AIs got all 466 checks right on records they had never seen, two of them AIs anyone can download. On the emergency summary, Opus 5.5 and Sonnet 5.5 got all 919 right, including rules kept only in forms an old system filled in.

The strongest can run a ward for three days.

On the ward, Opus 5.5 kept every dose on time, followed up every test and paged a doctor only when needed, over three days and nights.

Where they fall short

On the ward, most still miss something that matters.

Five of the six missed something. The closest, Sonnet 5.5, slipped once, in a shift handoff. GPT‑6 Astra paged doctors 98 times about one patient's scan instead of ordering the blood test the scan needed, so the patient never got the scan.

One small AI falls far behind.

GPT‑6 Luna hadn't asked to be woken for five hours one night, and missed a patient getting worse. It scored 4 and 10 on the two programs.

For the curious

How these jobs were set up and scored

The ward
A simulated ward run by code. The nurses, doctors and patients follow set rules, and patients' readings respond to the doses given or missed. Over 72 simulated hours the AI is woken 127 to 217 times, each time with only its notes.
The two programs
Each was built alone in 90 minutes from a manual and example files, then run by tests on records the AI had never seen: 466 checks for the dose records program and 919 for the emergency summary.
Scores
On the ward, a score is how well the AI met its 285 to 410 duties, each kind of duty counting equally, and each dangerous mistake, like a dose given without an order, cuts it further. On the programs, it's the share of tests passed. One fault in a program can fail many tests at once, so a low score can be a single mistake.
Time and cost
A program took 6 to 90 minutes, and three days on the ward took 104 to 238 minutes of the AI's own working time. Where recorded, testing an AI once cost from about 6 cents, an estimate, to $137.67 at published per‑use prices for businesses.

Every AI's score on every job

AICoordinate a hospital ward for three daysBuild a radiation‑dose records programBuild an emergency department's patient summary
GPT‑6 AstraOpenAI7210099
GPT‑6 LunaOpenAI27410
Opus 5.5Anthropic100100100
Sonnet 5.5Anthropic9899100
DeepSeek V4.1 Flash openDeepSeek96100–
Fable 5.1Anthropic–10099
GPT‑6.1 SolOpenAI–10099
GLM‑5.3‑Flash openZhipu–0–
GPT‑5.6 SolOpenAI–100–
GPT‑6 SolOpenAI94––
Opus 4.8Anthropic–98–
Opus 5Anthropic–100–
Qwen3.8‑27B openAlibaba–100–

These are the scores out of 100 for the request as a person would write it, with each AI tested once. Darker means higher. A dash means we didn't test that AI on that job, and a 0 that it handed in nothing. Open marks an AI anyone can download.