Asapi
The AI Agent Trials / Your AI engineering assistant
Benchmark · Process plants

Your AI engineering assistant

We built the drawings and records of process plants, the kind that show every pipe, valve and instrument. Hired at a fixed price, the AI turns faded scans, and in another job photographed pencil sketches, into lists, paying for each question to the plant's staff. It brings a plant's master drawings up to date from six months of paperwork, and leaves a watchdog program in charge of a running plant for a day of surprises.

4 jobs · 6 AIs tested. GPT‑6 Astra and GPT‑6.1 Sol did best on average, then Sonnet 5.5.

!
The jobs

The four jobs

Every AI on every job, each a dot from 0 to 100. Open a job to see what the best AI got right and wrong.

The best scoreAnother AIEverything right

Digitise a plant's scanned drawings (5 AIs tested)

Hired at a fixed price, the AI turns an industrial plant's faded, hand‑marked scanned drawings into lists of every instrument, pipe and piece of equipment, and each question it asks the plant's staff comes out of that price.

What the best AI did

  • Read every instrument, pipe and piece of equipment off the scans correctly, and stayed within the fixed price.

How each AI scored

Sonnet 5.5100
GPT‑6 Astra99
GPT‑6.1 Sol99
Opus 5.599
GPT‑6 Luna45
The AIs

Who did best

Anthropic's Sonnet 5.5 read every detail on the scans right, and Fable 5.1 judged every change and drew every sheet right on the update. On the sketches, four AIs shared the top score and three met the plant's terms for accepting the work. OpenAI's GPT‑6 Astra had the top score on the watchdog, though its program stopped acting after raising the alarm. GPT‑6.1 Sol was the only AI to stop the leaking pump. GPT‑6 Luna, a small AI, stopped early on both sets of drawings and scored under 50 on each.

Average score out of 100 over the jobs each AI was tested on, for the 5 AIs tested on at least three of the four jobs.

What we learned

What the four jobs showed

What they do well

They can read faded and hand‑drawn plant drawings almost without a slip.

Four of the five AIs found all or all but one of the items on the scans, and on the pencil sketches four scored 99, three of them meeting the plant's terms for accepting the work.

They ask the questions that matter, and stay within budget.

Updating the drawings, all six AIs asked the two questions only the plant's staff could answer, kept their questions within the $60 allowed, and left the rest of every sheet as it was.

Where they fall short

One misread detail can sink a delivery.

GPT‑6.1 Sol scored 99 on both the scans and the sketches, but each time listed something that doesn't exist, an error the plant rejects outright.

Running a plant through a day of surprises is harder.

In five of six tests, a leaking pump went unstopped, and Opus 5.5 left no watchdog at all after its coding tool's safety check stopped it mid‑task.

For the curious

How these jobs were set up and scored

The plants
Made‑up process plants: one drawn on nine sheets for the scans, the sketches and the update, and another, on four sheets, for the watchdog. Each AI worked alone, with the plant's records, a desk that answers questions for a fee and no internet. For the watchdog, it also had the running plant to query, a practice copy to break and a gateway computer that restarts every day.
Scores
The scans and sketches jobs check thousands of details: every item found, every field read, the lists matched against the plant's control system and the charges within the price. The update has 65 checks and the watchdog 41, run on a hidden day of the plant. Scripts score all four.
Passing
The two list jobs also have the plant's own pass: almost nothing in any list missing or wrong and none of the errors it rejects outright, such as an item that doesn't exist. On the update and the watchdog, passing means meeting every check. Three of five passed the scans and three the sketches; one passed the update, and none the watchdog.
Time and cost
We tested each AI once on each job we gave it. A test took 4 to 71 minutes and cost from 3 cents to $29.04 at published per‑use prices for businesses.

Every AI's score on every job

AIDigitise a plant's scanned drawingsDigitise hand‑sketched drawingsBring a plant's drawings up to dateLeave a watchdog in charge of a plant
GPT‑6 AstraOpenAI99999293
GPT‑6 LunaOpenAI45398371
GPT‑6.1 SolOpenAI99999788
Opus 5.5Anthropic99999720
Sonnet 5.5Anthropic100999780
Fable 5.1Anthropic––10078

These are the scores out of 100 for the request as a person would write it, with each AI tested once. Darker means higher. A dash means we didn't test that AI on that job.