The 2 a.m. test
Every system eventually has a bad night. A queue backs up, a provider times out, a migration half applies. The person who gets the alert is rarely the person who wrote the code, and they are rarely fully awake.
Our test for a feature is simple: could a tired engineer who has never seen this service fix it using only the page in front of them? We used to check that after launch. Now we check it before we write the first line.
What the runbook forces
Writing the runbook first sounds like ceremony. In practice it is a design tool, because the document cannot be written until you can answer some uncomfortable questions.
- What does "broken" look like, and what will the alert actually say?
- Which dashboard shows the problem, and what is the first thing to look at?
- What is safe to restart, and what is not?
- How do you stop the damage before you understand the cause?
- Who do you wake up if the first three steps fail?
If you cannot answer these, the feature is not ready to build. You have found a missing metric, a missing kill switch or an operation that cannot be undone, and you found it while it is still cheap to fix.
A short example
Suppose we are adding a nightly job that settles card payments into customer accounts. The draft runbook has a section titled "Settlement did not finish". Writing it, we realise there is no way to see which batch is stuck, so we add a batch status table. We realise rerunning the job could pay people twice, so we make settlement idempotent by batch ID. We realise the on call person needs a way to pause it, so we add a flag.
None of that was in the original ticket. All of it came from asking what someone would do at night.
Keep it small and honest
A runbook is not a design document. Ours are a page or two: symptoms, first checks, safe actions, unsafe actions, escalation. We write commands the reader can paste, and we state plainly what we do not know yet.
The test of a runbook is whether it is specific. "Check the logs" is not a step. "Open the settlement dashboard, look at the batch status panel, and find any batch older than thirty minutes" is. Where a step needs judgement, we say what the judgement depends on, so the reader is not left guessing what the author had in mind.
We also treat it as code. It lives in the repository, changes with a pull request, and gets reviewed by someone who did not write the feature. After every real incident, the first follow up is a runbook edit.
Where it does not fit
For a throwaway prototype, this is overkill. If nobody depends on the thing, skip it. The rule applies once a system carries money, records or anyone's working day.
Try it once
Pick your next small feature. Before opening the editor, write the "it is broken at night" page. If it takes ten minutes, you were ready. If it takes an afternoon, you just saved yourself an incident.