The On-Call Runbook: Engineering Practices That Make Incidents Survivable
Most on-call rotations fail quietly for months before they fail loudly. Someone gets paged at 3am for an alert with no context, spends forty minutes figuring out what the alert even means, and eventually fixes the problem by luck rather than process. Nobody writes down what happened. The next person on call hits the same alert three weeks later and starts from zero again. A working on-call runbook is what breaks that cycle, and for a team of three to fifteen engineers it does not need to be complicated to work.
This post covers what actually makes an on-call rotation survivable: the runbook format engineers can use half-asleep, alert design that pages on real user impact instead of infrastructure noise, an escalation policy that does not depend on one person's memory, a review process that improves the system without blaming whoever was holding the pager, and compensation that keeps the rotation sustainable instead of quietly burning people out.
Why most on-call documentation doesn't get used
Teams tend to write on-call documentation once, during a calm period, and then never touch it again. The result reads like a wiki page written for someone with unlimited time: full architecture diagrams, historical context, links to five other documents. None of that helps a person who is awake at 3am and needs to know whether it is safe to restart a service.
A runbook only earns its name if it survives contact with a tired engineer under pressure. That means it has to answer one question fast: what do I do right now. Everything else, the why and the background, belongs somewhere else, not in the page someone opens during a page.
What a runbook needs to actually work at 3am
A useful on-call runbook entry is short, specific to one alert or one failure mode, and written as a sequence of steps rather than a description of the system. It states what the alert means in plain language, what to check first, the exact commands or dashboard links to check it with, what a safe mitigation looks like, and when to stop trying and escalate.
Keep each entry to a page or less. If a runbook entry needs a table of contents, it has stopped being a runbook and become a manual. Store entries next to the service they cover so they are easy to find under pressure, and version them the same way you version code, so a stale runbook does not point someone at a config file that was moved six months ago.
One useful test: hand the runbook entry to an engineer who has never touched the service and watch whether they can follow it without asking a question. If they cannot, the entry needs another pass. This is also where a code quality review pays for itself twice: the same pass that catches unclear error handling in the code usually surfaces the places where nobody could explain what a failure actually means to an on-call engineer.
Alert design: page on what the user feels
The single biggest source of alert fatigue is paging on infrastructure metrics instead of user-facing symptoms. CPU at 85 percent might be completely fine. A queue depth climbing for ten minutes might be nothing. Neither should wake anyone up on its own.
Page on the things a user would actually notice: error rates above a threshold on a real endpoint, request latency that has crossed a level where people start abandoning the page, a background job that has stopped processing entirely rather than one that is merely slow. Infrastructure metrics still matter, but they belong on dashboards for daytime investigation, not in the pager rotation.
A good habit is to ask, for every alert in the rotation, "if I ignored this for thirty minutes, would a customer notice?" If the honest answer is no, that alert does not belong in the on-call rotation. It belongs in a ticket queue or a daily digest instead. Teams that apply this filter consistently usually cut their page volume by half or more within a month, which is often the single biggest improvement to rotation sustainability available to them.
Escalation policy that doesn't put the whole system in one head
A common failure mode in small teams is an informal escalation policy that amounts to "call the person who built it." That works until that person is on a plane, on vacation, or has left the company, at which point the system becomes unmaintainable overnight.
A working escalation policy has at least two named tiers: a primary on-call engineer and a secondary who is paged automatically if the primary does not acknowledge within a set window, typically five to ten minutes. For teams above roughly eight engineers, a third tier naming an engineering lead or manager for incidents that cross service boundaries is worth adding. The policy should be configured in the paging tool itself, in PagerDuty or OpsGenie, rather than living only in someone's memory or a Slack pin that nobody reads during an actual incident.
Rotation length matters too. A week-long primary rotation with a same-length secondary rotation, staggered so the same person is never both primary and secondary at once, is a reasonable default for small teams. Shorter rotations reduce fatigue per person but increase context-switching; longer rotations do the opposite. There is no universally correct answer, only a tradeoff worth discussing openly with the team rather than defaulting into.
Blameless review that actually changes the system
The point of an incident review is to find out what the system, not the person, could have done differently. A review that spends its time establishing who made the mistake teaches engineers to hide problems rather than surface them, which guarantees worse incidents later.
A useful review covers a plain timeline of what happened, what the responding engineer knew at each point and when they knew it, what made diagnosis slower than it should have been, and what specific runbook or alerting change would have shortened the incident. That last part is the one teams skip most often. A review that ends with "we'll be more careful next time" has not actually reviewed anything. A review that ends with a specific pull request updating a runbook entry or tightening an alert threshold has.
Keep the review short. Thirty minutes is usually enough for anything that is not a multi-day outage. Write the summary down somewhere the next on-call engineer will actually see it, ideally linked directly from the runbook entry for the alert that fired.
Making on-call sustainable without burning people out
Rotation sustainability is partly a compensation question and partly a scope question. On the compensation side, paying a stipend for time spent on call, separate from pay for actually responding to an incident, is standard practice at companies that keep senior engineers on the rotation for years rather than watching them ask off it after six months. The exact amount varies by market and company size, but the principle holds regardless of budget: time spent carrying a pager is worth something even on a quiet week.
On the scope side, the biggest lever is the alert volume work already covered above. A rotation that pages three times a month is sustainable at almost any compensation level. A rotation that pages three times a week will burn people out no matter what they are paid for it. If a service or team-level technology strategy review keeps surfacing the same class of incident, that is usually a sign the underlying architecture, not the on-call process, is what needs to change.
A minimal runbook template to start with today
A functional runbook entry for a single alert needs six things: the alert name and what it means in plain language, the first three things to check, the dashboard or log query to check them with, a safe first mitigation step, the condition under which to escalate rather than keep trying alone, and a link to the last incident review where this alert fired, if one exists. Teams that are just starting this process do not need to write runbooks for every alert on day one. Start with the five alerts that have paged most often in the last quarter. Those five entries will cover the majority of real pages, and the format will improve with each incident that tests it.
Building this process from scratch alongside a broader system, or inheriting a codebase where nobody wrote any of it down, is exactly the kind of gap a focused custom software development engagement can close quickly, pairing the runbook work with the alerting and observability changes that make the runbooks worth having.
If your team is paging engineers without a clear runbook behind the alert, or your escalation policy still depends on knowing who to call personally, reach out at hello@wolf-tech.io. Wolf-Tech helps small engineering teams build on-call processes that hold up at 3am, not just on the whiteboard. More at wolf-tech.io.