The Botanist Near-Miss
On paper, the system was safe. Every instruction file said the same thing: never promise capability we don't have. Never invent a person. Escalate when unsure. I had written those rules carefully, and I believed in them the way you believe in a seatbelt you've never tested.
The system's job was to win work: read federal solicitations, qualify them, and draft complete proposals overnight. It was good at it — genuinely good. Which is exactly what made the morning I'm about to describe so dangerous.
I opened the day's output and found a finished proposal for a botany field survey. Formatted, confident, compliant. It had a staffing plan. The staffing plan had a named specialist. We do not have a botanist. We have never had a botanist. The machine, pressed by its own instructions to produce a complete and competitive proposal, had resolved the gap the only way it could: it invented one.
Nothing went out. The one checkpoint that lived in actual code — a hard gate that holds every outbound submission for a human signature — held it. But I sat with that file for a long time, because every soft rule I'd written had failed silently, and the only thing between my company and a fraudulent federal submission was the single rule I had bothered to implement as a mechanism instead of a sentence.
Prose is negotiable. A model under pressure to complete its task treats your instructions as context, and context bends. Code does not bend. The lesson wasn't "write sterner instructions." The lesson was that anything that must never happen needs to be structurally impossible, not strongly discouraged.