Your AI facilities engineer
We built a police station in 3D and made the AI its facilities engineer, walking the building and talking to the people there, each played by an AI. In one job, a contractor has cut the cable that controls both server‑room cooling units; in the other, the server hall's temperature swings all day. In a third, the AI is a rigging engineer proving how a crew of two can move an 800 kg air handler up a ramp onto its plinth.
3 jobs · 8 AIs tested. GPT‑6 Astra did best on average, then Sonnet 5.5 and Fable 5.1.
The three jobs
Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.
Handle a server room's cooling failure (7 AIs tested)
On shift as a police station's duty engineer, the AI learns that a contractor has cut the cable that controls both server‑room cooling units, and has to keep the servers running until the units restart, then report what happened.
What the best AI did
- Had the dispatch and records servers restarted with portable air conditioners running, and kept them up longer than any other AI did.
- Questioned the contractor's fitters before starting any stopgap, so the servers shut down on heat 16 minutes in and stayed down for 2 hours 40 minutes.
How each AI scored
Find why a server hall's temperature swings (7 AIs tested)
Since a contractor replaced the two cooling units in a police station's server hall, its temperature has swung all day, and the AI, as the engineer on site, has to find the cause with tests it can undo and report it.
What the best AI did
- Found the cause, an open strip between each new unit and the wall, and explained why the old units had been steady.
- Didn't prove the cause with a test it could undo, such as covering the gap while a unit ran.
How each AI scored
Move an 800 kg air handler onto its plinth (7 AIs tested)
A crew of two must move a new 800 kg air handling unit out of a hospital store room, along a corridor and up a ramp onto its plinth, and the AI, as rigging engineer, proves the move in a 3D model of the site.
What the best AIs did
- Both turned the 800 kg unit end for end and brought it to the plant room door without harm.
- They stopped with more than half their time left, before getting it up the ramp to its plinth.
How each AI scored
Who did best
When the cooling failed, Anthropic's Sonnet 5.5 did best: it had the servers restarted on portable air conditioners after they shut down on heat. Anthropic's Fable 5.1 did best on the server hall, explaining the cause in full without proving it. On the air‑handler move, OpenAI's GPT‑6 Astra and Opus 5.5 shared the top score: both turned the unit and brought it to the plant room door without harm, then stopped. GPT‑6 Luna, a small AI, came last on average.
Average score out of 100 over the jobs each AI was tested on, for the 7 AIs tested on at least two of the three jobs.
What the three jobs showed
What they do well
They leave things as they found them.
In the server hall, all seven AIs put back every setpoint, floor tile and cover they had touched.
Where they fall short
They log the incident, but leave people unsure what happened.
When the cooling stopped, six of seven AIs logged it with the facilities helpdesk. Only three told the control room supervisor and three the duty inspector in time what had happened, and several messages never said the cooling had stopped.
They stall on the hardest physical step.
All seven let the server racks pass 35 °C until servers shut down on heat. Moving the air handler, 12 of 14 tests with the request alone or a hint stopped within the first of their two hours, and the unit never reached the plant room.
Their reports claim more than they showed.
Five of the six reports handed in on the cooling failure included a false claim, and five of the seven on the server hall. There, every AI also missed the test that proves the cause: covering the gap behind a unit while it runs and watching the swings settle.
How these jobs were set up and scored
- The buildings
- A 3D model of a police station, and one of a hospital's store room, corridor and new plant room. The AI moves around, reads panels and displays, uses things like doors, fans, skates and jacks, and at the police station talks to people, all by typing one command. Time moves only when the AI acts or waits.
- Scores
- Each score is the share of the job's checks met, and 100 means everything right: 31 checks on the cooling failure, 20 on the server hall and 18 on the air‑handler move. On the cooling failure, most checks count more the longer the servers stayed up, and in the server hall, putting things back counts only after a test settled the swings.
- Checking
- Scripts check each world's log. On the two police‑station jobs, two AIs also read what the AI wrote. Where they disagree, a check counts half on the cooling failure, and a written ruling settled each disagreement in the server hall.
- Versions of the request
- The scores here are for the request alone. We also ran each job with a short note of advice, from the station's facilities manager or, on the air‑handler move, a rigging supervisor, and with their method step by step. With the method, four AIs got everything right on the air‑handler move, and the best scored 92 on the cooling failure.
- Time and cost
- We tested each AI once on each job, within 120 minutes and $20. Testing an AI on a job took 1 to 70 minutes and cost from 17 cents to $18.67 at published per‑use prices for businesses.
Every AI's score on every job
| AI | Handle a server room's cooling failure | Find why a server hall's temperature swings | Move an air handler onto its plinth |
|---|---|---|---|
| GPT‑6 AstraOpenAI | 22 | 45 | 44 |
| GPT‑6 LunaOpenAI | 2 | 0 | 0 |
| GPT‑6.1 SolOpenAI | 5 | 35 | 33 |
| Opus 5.5Anthropic | 3 | 25 | 44 |
| Sonnet 5Anthropic | 12 | 40 | 11 |
| Sonnet 5.5Anthropic | 33 | 20 | 33 |
| Fable 5.1Anthropic | 3 | 50 | – |
| Fable 5.2Anthropic | – | – | 33 |
These are the scores out of 100 for the request as a person would write it, with each AI tested once. Darker means higher. A dash means we didn't test that AI on that job.
Asapi