Oncall Nihilism
Why Your Pager is a Design Failure
2:17am. The pager goes off.
An alarm fires: SQS message delay > 5 minutes.
A batch job somewhere in the system just spiked the queue from 100k messages per minute to 1 million. Your workers are auto scaling, but it takes 10–15 minutes to catch up.
Nothing is broken. The processing rate is exactly what it was five minutes ago—stable, efficient, and maxed out. Nothing needs fixing. But someone once decided that a five-minute delay is worthy of waking another human being.
You acknowledge the alarm. You watch the queue drain. After all, you cannot make the auto-scaling go any faster, and messages will remain delayed until the ingestion rate catches up to the new queue size. You are there simply to witness the inevitable. Ten minutes later, the system heals itself.
You go back to sleep — or try to — and spend the next 45 minutes staring at the ceiling while the rest of tomorrow quietly collapses.
After enough nights like this, a certain mindset begins to form.
You start acknowledging alarms with less urgency. You observe the system more than you intervene in it. You learn to distinguish between problems that require action and problems that simply require time. Usually, the system corrects itself.
Eventually you arrive at a quiet realisation:
Many pages exist not because the system is failing, but because someone once mistook temporary discomfort for catastrophe.
This is what I call a nihilist on-call.
Not laziness. Not negligence. Just a gradual understanding that many pages do not correspond to real failures, and that intervention often changes little. You watch the system. You wait. And most of the time, it fixes itself.
The Heat Death of the Service
Spend enough time on-call and another thought begins to surface. Most services do not fail catastrophically. They decay.
Dependencies drift. Dashboards fall out of date. Runbooks turn into archaeological artifacts documenting systems that no longer exist. This is operational entropy, and given enough time, every sufficiently complex service approaches its own version of heat death.
This is where the nihilism takes root.
When you realize the service is in a state of slow-motion decay, the urgency of the pager starts to feel like a lie. You arrive at the nihilistic realization that you aren’t “saving” the system; you are merely performing an act of penance for a design you didn’t choose. Many alerts imply that the system is moments away from collapse, but the reality is a slow, gray fade into obsolescence.
But heat death is only inevitable if you accept the role of the bystander.
The sense of futility is a protective layer of scar tissue, but it is also a choice. We treat entropy as a law of nature, but in software, entropy is a choice of priority. Most pages are not preventing the heat death of the service because they focus on the symptoms of the decay rather than the decay itself. To move past this futility, we have to stop treating the pager as a death knell and start using it as a diagnostic tool for restoration.
Signal Dilution
There is another consequence of this dynamic that only becomes obvious at scale. When a system produces enough meaningless pages, the meaningful ones begin to dissolve into the noise.
The pager still rings. The alarm still says SEV-2. But psychologically it carries far less weight than it should. Not because engineers have become careless, but because the system itself has trained them to treat alerts with skepticism.
Alarm inflation is almost inevitable in large organizations. Every team adds alerts intended to protect their own services. Very few alarms are ever removed. Self-healing systems page simply because a metric briefly crossed an arbitrary threshold. Over time the system begins to generate a constant background hum of operational noise.
This is the real danger of alarm fatigue. Reliability is rarely destroyed by a single catastrophic failure. Much more often it is eroded gradually by a loss of trust in the signals meant to protect the system. The fastest way to make engineers ignore an alarm is to page them repeatedly for events that resolve themselves.
The Myth of the Pager
In the myth of Sisyphus, a man is condemned to push a boulder up a hill forever, only for it to roll back down each time he reaches the top.
On-call sometimes feels similar. The pager rings. You acknowledge the alarm. You click through dashboards. The system stabilizes. And somewhere in the background, another alert is already preparing to wake you tomorrow night.
But unlike Sisyphus, engineers are not actually condemned to this cycle. Most on-call pain is not inevitable. It is designed.
To cure the futility of the on-call, we should refuse absurd labor. We should stop engineering better ways to push the boulder and start questioning why it exists at all.
Practical Rules for On-Call
No Human Intervention, No Page.
If an alert fires and the resolution is simply “wait for it to clear” or “restart the service,” the rock has rolled back to the bottom.
The Rule: If a human doesn’t need to make a unique, creative decision to fix it, a computer should be doing it. Do not wake a human for a task a script could do.
Page on Symptoms, Not Causes.
We should let the system decay in silence if that decay doesn’t hurt the user.
The Rule: High CPU is a cause, not a symptom. If your system is still serving requests and users aren’t experiencing errors or noticeable latency, the pager should stay silent. Only wake someone up when the system is actually broken.
Delete the “Flappy” Alerts.
Every recurring alert that you “acknowledge and ignore” is a high-priority bug in your monitoring system.
The Rule: If an alert fires three times in a shift and requires no action, delete it. Don’t “tune” it. Kill it. If the system doesn’t break when the alert is gone, it was never an alert. It was noise.
Protect the Sleep of Others.
High-signal hygiene is a collective pact.
The Rule: Before you create an alarm, ask yourself: “Am I willing to wake up my best friend at 3:00 AM for this?” If the answer is no, it shouldn’t be pageable.
Conclusion
We should refuse to treat the pager as a tool of penance. The struggle itself toward the heights is enough to fill a man’s heart, but only if the heights actually exist.
If the system is going to reach its heat death anyway, we might as well get some sleep.