A counter in process memory is not a guard: 131 restarts proved it

A recent analysis highlights a common pitfall in system monitoring: using in-process memory counters to track failures. The author details a scenario where a daemon process was restarted 131 times due to a stale-heartbeat rule, causing its internal failure counter to reset to zero repeatedly. Consequently, a critical escalation guard never triggered because the process was constantly being replaced, effectively erasing its memory of previous failures. The article argues that counters are unreliable for monitoring processes prone to frequent restarts. Instead, it proposes using persistent, timestamp-based markers in external files to track liveness. By shifting from counting events to recording the last successful execution time, developers can ensure that monitoring remains robust against process amnesia. The piece concludes by categorizing common failure shapes—such as unreachable counters, stalls, and degenerate outputs—and emphasizes that effective monitoring should be structurally integrated rather than relying on per-call-site discipline.