Manual monitoring fails because every hand-written monitor is a guess about the system, frozen on the day someone wrote it. The system keeps changing and the monitors don't, so coverage decays. Postmortems don't reverse that. They fix the last incident, and when they do touch monitoring, they add a check for the failure that already happened.
Open almost any postmortem and skip to the end. The action items are careful and specific: a safer rollout, a validation step, a retry limit, sometimes one new alert for the exact thing that broke. Each item is reasonable. Together they describe the last incident in fine detail and say very little about the next one.
What's wrong with manual monitoring?
No single monitor is the problem. The method is. A person decides what to watch, writes a rule and moves on. The rule stays the same while the system it watches changes every week. Over months the gap between what the rules describe and what is actually running grows, and nobody owns closing it.
Most teams feel this as two complaints that seem to contradict each other: too many alerts, and incidents nobody was alerted about. They are the same problem. A pile of rules written for past failures is noisy about the old and silent about the new.
What do postmortems actually fix?
The incident that happened. We read 15 well-known public postmortems and counted their action items, about 90 in all. Most went to safer rollouts, validation, isolation and capacity. Only 5 of the 15 had any item that added or fixed monitoring, about 6% of all items. Postmortems prepare for the last incident.
That surprised us. We expected "add more alerts" everywhere. The real pattern is quieter. When monitoring does show up, it is a check for the exact failure that just happened. After a 2019 database failure, Stripe "implemented additional monitoring to alert us when nodes stop reporting replication lag." That is a good fix. It also describes one failure, after it happened.
What a postmortem rarely asks is the coverage question: why wasn't this already watched, and what else is in the same state right now? That question is about the system as it is today. A postmortem, by design, is about one event in the past.
About the sample: Cloudflare (2019, 2023, 2025), GitHub (2018), Slack (2021), AWS S3 (2017), Google Cloud (2025), Stripe (2019), Datadog (2023), Sentry (2015), CrowdStrike (2024), Honeycomb (2023), LaunchDarkly (2025), CircleCI (2021) and Heroku (2025). An action item counted as monitoring-related only if it added, fixed or tuned monitoring, alerting or detection. Fifteen is a small sample, and counts depend on how you split prose commitments into items.
Every monitor is a guess about the past
A threshold records what someone believed normal looked like, and which failures they pictured, on the day they wrote it. The Google SRE book says pages "should be about a novel problem or an event that hasn't been seen before." A rule can only describe a problem someone has already seen or imagined. The pages that matter most are the hardest ones to write in advance.
So even a perfect set of rules, written by your best engineer, is a snapshot. It is right about the system it was written for. The question is how long that system lasts.
Monitors decay as systems grow
Monitors are written from memory: what the author knew about the system and which parts they thought mattered. New services, endpoints, queues and dependencies ship every week. Each one starts life unwatched unless someone remembers to add it. Meanwhile, old checks keep watching code paths that no longer carry traffic.
Epic Games learned this in April 2021, when an internal certificate expired and set off an outage that spread to its launcher and store. The certificate's zone had never been covered: "The DNS zones for this internal service-to-service communication were not actively monitored." Their certificate monitoring keyed off whole DNS namespaces, and this zone was outside them. The line that stays with us: "With our existing monitoring systems, we believed that we were more protected than we actually were."
Cloudflare described the same drift in resilience, not monitoring, after its November 2023 control-plane outage: "some services, especially for some newer products, had not yet been added to the high availability cluster." New things ship faster than the standards meant to cover them.
Decay isn't anyone's mistake. It is what happens when coverage depends on someone remembering.
They only catch what someone predicted
In April 2022, Atlassian deleted 883 customer sites by mistake. Nothing alerted. In its post-incident review: "Our internal monitoring did not detect an issue because the sites were deleted via a standard workflow." The first customer ticket arrived about eight minutes in.
Every system did what it was told. No rule described "a valid workflow deleting the wrong sites," because nobody had predicted it. This is the shape of many of the worst incidents. Error rates are flat, latency is fine, the job succeeded. What's wrong is that it did the wrong thing.
They fail quietly themselves
Monitors are software too, and they break without telling anyone. In January 2017, GitLab lost production data and turned to its backups. The backups had been failing for some time, and the failure emails were rejected by DMARC on the way to the team. From the postmortem: "we were never aware of the backups failing, until it was too late."
In July 2026, Buttondown's tracking domains were down for about six hours. "We found out about this issue from user reports." There was a monitor for exactly this case. It "was misconfigured and didn't alert us."
A missing alert looks the same as a healthy system. Nobody gets paged when the pager breaks. That is why heartbeat checks exist, and why a hand-kept set of monitors needs monitoring of its own, which is more hand-kept work.
The one-dial problem: why teams trade recall for quiet
Alerting has two scores. The Google SRE Workbook defines precision as "the proportion of events detected that were significant" and recall as "the proportion of significant events detected." You want both high. With hand-written rules you get one dial to do it: how sensitive the rules are.
Turn the dial up and recall rises, along with the noise. Turn it down and things get quiet, and things get missed. Each postmortem that adds a check nudges recall up for one failure. The pile gets louder. Then a cleanup comes, driven by a tired on-call rotation, and the cheapest way to buy quiet is to loosen thresholds and mute the chattiest rules. Recall falls, and nothing announces it.
Reasonable-looking rules hide gaps like this. One of the Workbook's own example setups has "low recall: A 35x burn rate never alerts, but consumes all of the 30-day error budget in 20.5 hours." And noise hides signal even when the signal is there. In Mailchimp's February 2019 Mandrill outage, the first sign of trouble arrived, but "unrelated exceptions coming in from other systems drowned it out in our centralized reporting." Before the outage, the team had spotted the risk and "added a ticket to the backlog to set up additional monitoring." Alerting for it was only added once the outage was underway.
Isn't the fix better alerts?
Better alerts help, for a while. SLO burn-rate alerts, anomaly detection and good runbooks all raise the quality of each rule. They don't change who writes the rules or how often anyone revisits them. Hand-written coverage grows with incidents and headcount. The system grows with every deploy. Those two curves don't meet.
Each postmortem alert is cheap insurance. The pile isn't. And anomaly detection, useful as it is, often ends up as one more dial to mute: production is full of unusual things that aren't broken, so the alerts get noisy and get turned down.
What should replace manual monitors?
Not a better list. A different way of keeping the list. Whatever does the job, a team, a tool or both, it has to meet four principles:
- Coverage follows the system as it is now. New services, endpoints, jobs and dependencies get watched when they ship, not after they fail. Checks on things that are gone get retired.
- Catching is separate from paging. Look widely for anything that changed (high recall), but only interrupt a person for what holds up (high precision). With one dial you trade the two. With two steps you don't have to.
- A finding is proven before anyone is paged. What changed, when, the evidence, who it affects and whether it is still happening. This is the answer to "AI tools are confidently wrong": nothing should page on a hunch, from a person's rule or a model's guess.
- Postmortems ask about coverage. Not "which alert do we add?" but "why wasn't this already covered, and what else looks like this?"
Does relying less on hand-written alerts mean missing more? Only if catching and paging are the same step. Split them, and you can look harder while paging less.
Is your monitoring decaying? A check you can run this week
You don't need a new tool to find out. Five questions, answerable from data you already have:
- New components: list what shipped in the last 90 days: services, endpoints, queues, scheduled jobs, third-party dependencies. How many have no check at all?
- Freshness: do your critical jobs and data pipelines have heartbeat or freshness checks, so that silence itself raises an alert?
- Recall: for each of last quarter's incidents, who found it first: an alert, someone looking at a dashboard, or a customer?
- Precision: what share of last month's pages needed someone to act?
- Postmortem balance: across the last year of postmortems, count monitoring items added and monitoring items retired.
If customers found any of last quarter's incidents first, coverage has drifted. If almost nothing was ever retired, the pile is only growing.
For how this differs from AIOps and AI SRE tools, see AI SRE vs AIOps vs proactive detection. Mills, from Sawmills, is built on these principles.
{{TIPS}}
FAQs on manual monitoring
Why do postmortem action items rarely fix coverage?
A postmortem reviews one incident, so its action items fix that incident: safer rollouts, validation, retry limits, sometimes a check for the exact failure. In 15 public postmortems we read, only 5 had any monitoring item. Few ask why the problem wasn't already covered, or what else is in the same state.
What is monitoring decay?
The growing gap between what your monitors watch and what is actually running. New services and dependencies ship unwatched, old checks keep watching things that no longer matter, and thresholds drift out of date. It happens because coverage depends on people remembering to update it.
What do precision and recall mean for alerts?
Precision is the share of alerts that point to something significant. Recall is the share of significant events that raise an alert. Low precision means noise and alert fatigue. Low recall means incidents nobody was told about. With hand-written thresholds, raising one usually lowers the other.
Why not just alert on everything?
Because recall bought with noise doesn't last. When most pages need no action, people stop trusting them, then loosen or mute the rules, and recall falls again. Watching everything is fine. Paging on everything isn't. Catching and paging should be separate steps.
What should earn a page?
A problem that is real, affects users or will soon, and needs a person to act now. Before it pages, a finding should show what changed, the evidence, who it affects and whether it is still happening. Anything less belongs in a ticket or a daily summary.
What is a deadman (heartbeat) check?
A check that alerts when an expected signal stops arriving, such as a job that should report every hour or a pipeline that should keep writing rows. Threshold alerts can't see silence. A heartbeat check turns missing data into an alert, which also catches monitors that quietly broke.
