There is a moment in most operations teams that nobody writes down. Someone mutes the monitoring channel — just for the afternoon, just to get something finished. It never gets unmuted.
From that point the alerting system is decorative. It still fires, still costs money, still shows green dashboards to whoever asks. It just no longer changes anyone’s behaviour.
This is worth taking seriously rather than treating as a discipline failure, because the muting was rational. If ninety per cent of what arrives is noise, ignoring the channel is the correct local decision. The problem is upstream.
Measure it before you fix it
Almost everyone reaches for automation at this point. Before you do, spend four weeks measuring, because the numbers determine which fix is worth anything. Four things are worth counting:
Alerts per incident. How many separate notifications does one underlying problem generate? A database saturating can easily produce forty alerts across the application, the connection pool, the load balancer and every downstream service. Forty notifications, one thing wrong. If this ratio is above about five, correlation will pay for itself before anything else you could do.
Actionability rate. What share of alerts led to a human doing something? Not investigating — doing. Anything under about a third means the thresholds are describing the system rather than describing a problem.
Repeat share. How many incidents this month were recognisably the same as an incident from a previous month? The disk fills, the pod thrashes, the certificate expires, the queue backs up. This is the number that determines whether automated remediation is worth building, and in most estates it is far higher than the team expects.
Out-of-hours volume. How many pages landed between 10pm and 7am, and how many of those were in the repeat category? This is the retention number. Engineers rarely resign over the work; they resign over the third consecutive week of being woken by something they already fixed and documented.
What the numbers tell you to do
If alerts per incident is high but actionability is reasonable, you do not have an alerting problem, you have a correlation problem. Grouping related alerts into one incident with one owner fixes it, and it needs no trust and no automation authority at all — nothing acts, things are merely gathered up. This is usually the first thing a team feels, because the noise reduction is immediate.
If actionability is low, the thresholds are wrong. Alerting on CPU at eighty per cent describes a machine. Alerting on the error budget for a user-facing objective describes a problem. Moving alerts onto service level objectives typically deletes more noise than any tool will, and it is free.
If repeat share is high, you have a runbook problem that looks like a staffing problem. Those incidents already have documented fixes; the fix is simply being executed by a person at an inconvenient hour. That is the class of work worth automating, and it is the only class we would propose automating early.
If out-of-hours volume is high but repeat share is low, do not automate. Every incident is novel, which means judgement is required, which means a human is required. Fix the architecture instead.
Why measuring first matters
Automation applied to a noisy system automates the noise. If forty alerts arrive for one problem and you point an agent at them, you now have an agent responding forty times. The correlation has to come first, and it has to come first regardless of whether you ever automate anything.
This is also why we run agents in shadow mode — observing, correlating, drafting diagnoses, acting on nothing — for four weeks before anyone considers giving them authority. At the end of it you have real numbers for your own estate rather than a vendor’s averages: what your alerting cost you in noise, which incidents were repeats, and how many hours of last month were recoverable.
Quite often that report is enough on its own. A team that discovers eighty per cent of its pages come from six recurring causes usually does not need an agent first. It needs to fix six things.
The measurement above is exactly what a shadow-mode trial produces. It changes nothing and touches nothing.