Most infrastructure has monitoring installed. Far fewer environments have monitoring that's actually working — tuned, meaningful, and acted on. There's a real difference between collecting data and running a monitoring program, and that difference is usually the reason an outage is discovered by a user complaint rather than an alert that should have fired first. This article covers what actually needs to be monitored, why over-alerting undermines the entire program, and what mature infrastructure monitoring looks like in practice.
What actually needs to be monitored
Infrastructure monitoring spans three layers, and each one has a different set of signals that matter.
Compute. CPU, memory, and disk utilization on servers and hosts; the health of critical services and processes — is the service actually running, not just is the server powered on. A server that's up but has a stopped critical service is functionally down for whatever depends on that service, even though a basic ping check would report it healthy.
Network. Interface utilization on switches and routers, packet loss, latency, and the reachability of critical network devices. Network problems are often gradual before they're total — rising latency or intermittent packet loss on a link is frequently an early warning of a failure that hasn't happened yet, which is exactly the kind of signal worth catching before it becomes an outage.
Storage. Capacity thresholds — how close a volume is to full, which causes cascading failures across everything depending on it, not just a warning message; I/O performance, since degraded storage performance often shows up first as "everything feels slow" rather than an obvious failure; and replication and backup job status, since a backup or replication job that's been silently failing for weeks provides no actual protection despite appearing to be part of the environment's safety net.
Alert fatigue is the failure mode that matters most
The instinct when standing up monitoring is often to collect and alert on everything possible — more visibility feels safer. In practice, this is the single most common way monitoring programs fail. A monitoring system that generates hundreds of low-value alerts — transient CPU spikes, routine reboots, thresholds set to vendor defaults that don't match the actual environment — trains the people receiving those alerts to skim past them, filter them into a folder nobody checks, or simply stop reading them closely. Once that happens, the one alert that actually mattered is sitting in the same inbox as the noise, indistinguishable from everything else that turned out to be nothing.
A monitoring system nobody trusts is not providing protection
The value of monitoring isn't the volume of data collected — it's whether the right person takes the right action when something that matters actually happens. A high-noise alerting environment can look comprehensive on paper while providing essentially no real protection, because the signal that matters is buried and, functionally, unmonitored.
The goal of a monitoring program is meaningful signal, not maximum data collection. That means being deliberate about what generates an alert versus what's simply logged for later reference, and being willing to leave data uncollected — or collected but silent — if it doesn't actually change what anyone does.
What a mature monitoring program looks like
Tuned thresholds specific to the environment. Vendor default thresholds are a starting point, not a destination. A server that normally runs at 70% memory utilization shouldn't alert at a generic 80% threshold set for a different kind of workload — thresholds need to reflect what's actually normal for that specific system, which requires someone to actually look at baseline behavior and adjust.
Clear escalation paths. Different severities should reach different people through different channels — a capacity warning that can wait until business hours shouldn't page someone at 2 a.m., and a service-down alert on a critical system shouldn't sit in a queue waiting for someone to notice it during their next check-in. Escalation paths that don't distinguish severity end up either under-reacting to real emergencies or over-reacting to routine warnings, and often both at once.
A periodic review cadence. Environments change — systems get decommissioned, workloads shift, new services get added — and monitoring configurations that aren't revisited drift out of sync with reality. A recurring review to retire alerts for systems that no longer exist, re-tune thresholds that generate consistent noise, and confirm that what's being monitored still matches what's actually running is what keeps a monitoring program relevant instead of becoming an accumulating pile of stale configuration.
Common mistakes
- Monitoring installed once and never tuned. Alert thresholds set at initial deployment and never revisited as the environment changes become progressively less accurate, generating both false alarms and missed detections over time.
- Alerting on everything by default. Treating every available metric as alert-worthy produces the volume of noise that causes alert fatigue in the first place — most metrics are worth logging, far fewer are worth interrupting someone for.
- No escalation logic. Routing every alert, regardless of severity, to the same channel and the same people means real emergencies compete with routine notices for attention, and nothing gets prioritized correctly.
- Not monitoring backup and replication job status. A backup job silently failing for an extended period provides zero actual protection, but without status monitoring, that failure is often only discovered when a restore is attempted and fails — see Backup Testing Best Practices for why this specific blind spot is so common.
FAQ
Is more monitoring always better? No — more monitoring is better only when the additional data leads to different, better decisions. Monitoring that isn't tied to a meaningful threshold or a clear response just adds noise, which actively works against the goal of catching what matters.
How often should alert thresholds be reviewed? There's no universal number, but a recurring cadence — often quarterly, or tied to significant infrastructure changes — is far better than the common default of "never after initial setup." The right frequency depends on how quickly the environment changes.
Should every alert page someone immediately? No. Alerts should be tiered by actual urgency — a capacity warning with weeks of runway left doesn't need the same response speed as a service outage on a production system. Treating every alert as equally urgent is one of the fastest paths to alert fatigue.
Related reading
- Enterprise Infrastructure Fundamentals
- Network Architecture Fundamentals
- Backup Testing Best Practices
- Monitoring & Alerting Strategy — the operational process and escalation discipline behind what to monitor here
- Download: Infrastructure Monitoring Checklist