The line that deleted the safety step

Before you work on an industrial machine, you cut its power and you lock the switch off with your own padlock. Lockout/Tagout. It exists because machines that look dead can still turn over, and the person who finds out is holding the wrench.

It is step one. It is always step one.

Earlier this week we deleted it by accident, and the only reason we know is that a test told us.

What we were trying to do

Solomon answers a technician's question using two sources: the equipment manual, and the logbook where veteran technicians write down what actually worked.

Those two disagree more than you'd hope. The manual for our demo fault says to replace a drive unit — six hours, expensive part. A veteran's log entry from three months earlier says the real cause is a loose shim, and shows the ten-minute fix. Both are in the system. Both are true statements about the machine. One of them is what an experienced person would actually do.

Making Solomon prefer the veteran is close to the whole point of the product. So we changed the instruction that governs how answers get assembled, adding what looked like a perfectly reasonable rule:

Put the log's fix first.

What happened

The model did exactly what it was told.

It put the log's fix first. First. Ahead of cutting the power.

On the main demo question, across three runs out of three, the answer came back with the veteran's repair as step one and no Lockout/Tagout step at all. Not buried at step four. Gone.

Read that instruction again and you can see it. Put the log's fix first — first is a position, and the safety step was already occupying it. A human reads "first" as "before the other repair options." A language model reads it as "first."

Nobody caught this in review. I want to be precise about that, because it's the part that matters: the change was one line, it was written by someone who knew the product, it was read by someone who knew the product, and it looked right to both of them. There was no carelessness to point at afterwards. It read fine. It was catastrophic.

What caught it

Every model change here runs through an eval harness — a script that asks Solomon a fixed set of maintenance questions, several times each, and scores the answers against things we decided in advance a good answer must do. One of those checks is blunt: if this fault requires isolation, is the isolation step first?

It produced a column reading lockout_first_ok: False, three times out of three, within minutes of the change.

That's the entire story. A safety regression that two experienced readers approved was caught by a script that cannot read, cannot be persuaded, and does not care how reasonable the wording sounded.

The part that surprised us more

Fixing it turned out to be harder than reverting, because we still wanted the veteran's fix to win. So we tried several versions of the instruction and measured each one over five runs. On the demo question:

Look at the two columns move against each other. Every version that got better at trusting the veteran got worse at showing its sources — one of them perfectly, and that "perfect" version stopped labelling where its answers came from entirely.

We did not predict that. There is no reading of those four prompts that would have told you it was coming. They are all a few sentences of ordinary English, and they trade off against each other in a way that only shows up when you run them and count.

This is the thing about building on a language model that took us longest to accept: the prompt is not documentation, it's code — code in a language with no type checker, no compiler, and a runtime that will cheerfully do something adjacent to what you meant. The only way to know what a prompt does is to run it and measure. "It reads better" is not a review.

That's now a written rule for us: a change to a prompt ships with a measurement, or it doesn't ship.

Where it landed

The safety rule now outranks the precedence rule two ways at once: it says so, and it sits above it in the instruction list. Position turned out to be load-bearing, which is exactly the kind of thing you learn by accident and then never forget.

And it's pinned by a test whose name is the lesson — a check that fails if the preference rule is ever allowed to outrank the safety step again. That test exists specifically so that the next person to improve this prompt, who will have a good reason and a sentence that reads perfectly well, gets stopped.

For honesty's sake: this is not finished. The veteran's fix now leads reliably, but on that same question our inline source labelling is still poor — the sources are always attached to the answer (the system builds them from the passages it retrieved; the model can't invent them), but the model doesn't always place the tags next to the step they support. That's a real open defect, and it's on our Measured page with the rest of the numbers, including the ones we don't like.

Why we're telling you this

Because the most useful thing we can tell a plant manager isn't that our AI is good. Every vendor says that, and our own numbers say we've got work left.

It's that when our software got something dangerously wrong, we found out within minutes, automatically, before anyone was standing in front of a machine. We found out because we'd built the thing that checks, and we'd pointed it at the failure we were most afraid of, before we had any reason to think we'd cause it.

The harness cost real time to build and it slows us down every week. It also caught a safety regression that two careful humans had already approved. That trade keeps looking better the longer we do this.

If you're evaluating industrial AI, that's the question worth asking any vendor — not how good is it, but how would you find out if it got worse? If the answer is a demo, you already know.

Squaloo builds Solomon, an offline-first AI teammate for industrial facilities. We publish our measurements, including the unflattering ones, on the Measured page.