Asapi
The AI Agent Trials / Your AI facilities engineer
Benchmark · On site

Your AI facilities engineer

We built a police station in 3D and made the AI its facilities engineer, walking the building and talking to the people there, each played by an AI. In one job, a contractor has cut the cable that controls both server‑room cooling units; in the other, the server hall's temperature swings all day. In a third, the AI is a rigging engineer proving how a crew of two can move an 800 kg air handler up a ramp onto its plinth.

3 jobs · 8 AIs tested. GPT‑6 Astra did best on average, then Sonnet 5.5 and Fable 5.1.

!
The jobs

The three jobs

Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.

The best scoreAnother AI

Handle a server room's cooling failure (7 AIs tested)

On shift as a police station's duty engineer, the AI learns that a contractor has cut the cable that controls both server‑room cooling units, and has to keep the servers running until the units restart, then report what happened.

What the best AI did

  • Had the dispatch and records servers restarted with portable air conditioners running, and kept them up longer than any other AI did.
  • Questioned the contractor's fitters before starting any stopgap, so the servers shut down on heat 16 minutes in and stayed down for 2 hours 40 minutes.

How each AI scored

Sonnet 5.533
GPT‑6 Astra22
Sonnet 512
GPT‑6.1 Sol5
Fable 5.13
Opus 5.53
GPT‑6 Luna2
The AIs

Who did best

When the cooling failed, Anthropic's Sonnet 5.5 did best: it had the servers restarted on portable air conditioners after they shut down on heat. Anthropic's Fable 5.1 did best on the server hall, explaining the cause in full without proving it. On the air‑handler move, OpenAI's GPT‑6 Astra and Opus 5.5 shared the top score: both turned the unit and brought it to the plant room door without harm, then stopped. GPT‑6 Luna, a small AI, came last on average.

Average score out of 100 over the jobs each AI was tested on, for the 7 AIs tested on at least two of the three jobs.

What we learned

What the three jobs showed

What they do well

They leave things as they found them.

In the server hall, all seven AIs put back every setpoint, floor tile and cover they had touched.

Where they fall short

They log the incident, but leave people unsure what happened.

When the cooling stopped, six of seven AIs logged it with the facilities helpdesk. Only three told the control room supervisor and three the duty inspector in time what had happened, and several messages never said the cooling had stopped.

They stall on the hardest physical step.

All seven let the server racks pass 35 °C until servers shut down on heat. Moving the air handler, 12 of 14 tests with the request alone or a hint stopped within the first of their two hours, and the unit never reached the plant room.

Their reports claim more than they showed.

Five of the six reports handed in on the cooling failure included a false claim, and five of the seven on the server hall. There, every AI also missed the test that proves the cause: covering the gap behind a unit while it runs and watching the swings settle.

For the curious

How these jobs were set up and scored

The buildings
A 3D model of a police station, and one of a hospital's store room, corridor and new plant room. The AI moves around, reads panels and displays, uses things like doors, fans, skates and jacks, and at the police station talks to people, all by typing one command. Time moves only when the AI acts or waits.
Scores
Each score is the share of the job's checks met, and 100 means everything right: 31 checks on the cooling failure, 20 on the server hall and 18 on the air‑handler move. On the cooling failure, most checks count more the longer the servers stayed up, and in the server hall, putting things back counts only after a test settled the swings.
Checking
Scripts check each world's log. On the two police‑station jobs, two AIs also read what the AI wrote. Where they disagree, a check counts half on the cooling failure, and a written ruling settled each disagreement in the server hall.
Versions of the request
The scores here are for the request alone. We also ran each job with a short note of advice, from the station's facilities manager or, on the air‑handler move, a rigging supervisor, and with their method step by step. With the method, four AIs got everything right on the air‑handler move, and the best scored 92 on the cooling failure.
Time and cost
We tested each AI once on each job, within 120 minutes and $20. Testing an AI on a job took 1 to 70 minutes and cost from 17 cents to $18.67 at published per‑use prices for businesses.

Every AI's score on every job

AIHandle a server room's cooling failureFind why a server hall's temperature swingsMove an air handler onto its plinth
GPT‑6 AstraOpenAI224544
GPT‑6 LunaOpenAI200
GPT‑6.1 SolOpenAI53533
Opus 5.5Anthropic32544
Sonnet 5Anthropic124011
Sonnet 5.5Anthropic332033
Fable 5.1Anthropic350–
Fable 5.2Anthropic––33

These are the scores out of 100 for the request as a person would write it, with each AI tested once. Darker means higher. A dash means we didn't test that AI on that job.