The Root-User Silent Outage
For four days, every dashboard was green. Services running, timers firing, logs flowing. And for four days, three of my systems produced absolutely nothing — no outreach, no pitches, no detection — while looking perfectly healthy.
The cause was almost embarrassingly small. Three service definitions were missing one line — the line that says which user the service runs as. They ran as root instead, and root's home directory happened to contain just enough state for the services to start cleanly and just little enough for them to find no work to do. They weren't crashing. They were succeeding at nothing, on schedule, with green status lights.
I found it only because an unrelated mistake forced me to walk the whole system by hand. That's the part that still bothers me: no alarm was ever going to fire, because every metric I monitored was a proxy for "the process ran," and the process did run. What I wasn't monitoring was identity — who the machine believed it was when it did the work.
Uptime is not output. A process can run forever and deliver nothing, and the more automated your operation, the easier that failure hides inside good-looking telemetry.