Green lights lie — check who the machine thinks it is
Services ran for days under the wrong identity. Dashboards stayed green. Work did not leave the building.
Rule: Green lights lie; check who the machine thinks it is.
The silent outage
Health checks passed. Processes were up. Logs looked calm. The business outcome — outbound work that should have shipped — did not. The machines were running as someone they were not supposed to be. Permissions, queues, and “success” metrics all told a happy story about the wrong story.
Why metrics lie
Most monitoring answers: is the process alive? Few answer: is the process the right principal, on the right tenant, with the right mailbox, in the right environment?
Green is necessary. Green is not sufficient.
Employment practice
- Identity checks in the runbook, not only uptime checks.
- Periodic “who am I?” probes that fail loud.
- Separate “system healthy” from “mission shipping.”
Transfer
For each automation: add one check that verifies identity/role/tenant before celebrating success. If you cannot state who the machine thinks it is, you do not have an employee — you have a ghost process.