Ask most IT teams what actually caused their last major outage, and the honest answer is rarely a hardware failure or an external attack. It's usually a change — a configuration update, a patch, a deployment — that went wrong because nobody reviewed it, nobody tested it, or nobody had a plan for undoing it if it failed. Change management is the discipline that exists specifically to prevent that. This article covers why uncontrolled change is such a common cause of outages, how a proportionate change management process is actually structured, and why matching process rigor to actual risk is what makes the whole thing sustainable in practice.
Why uncontrolled change is a leading cause of preventable outages
A change made to a production system without review, without testing, and without a rollback plan is one of the most common root causes of unplanned downtime — more common, in many environments, than hardware failure or an external attack. This isn't a controversial claim; it's a pattern that shows up repeatedly across IT environments of every size, because the failure mode is structural rather than incidental.
The reason is straightforward: every change to a production system carries some risk of unintended consequences, and that risk compounds when nobody outside the person making the change has looked at it, when it hasn't been tested in a non-production environment first, and when there's no defined way to reverse it if something goes wrong. Without review, mistakes that a second set of eyes would have caught go straight to production. Without testing, problems are discovered by users rather than by the team making the change. Without a rollback plan, a failed change turns into an extended outage instead of a quick reversal.
None of this means every change is dangerous — most changes go fine. But across a large enough number of uncontrolled changes, the ones that don't go fine are what produce most preventable outages, and they're preventable specifically because a defined process would have caught the problem before it reached production.
How a proportionate change management process works
The instinctive response to this risk is to require review and approval for every change, no matter how small. In practice, that instinct backfires — treating a routine, low-risk change with the same rigor as a high-risk one makes the process slow and burdensome without making the organization meaningfully safer, and it's the single biggest reason change management processes get bypassed. The alternative is a proportionate, tiered model that matches the level of scrutiny to the level of risk.
A common and effective structure uses three tiers:
- Standard changes are pre-approved and low-risk — changes that are well understood, have been performed successfully many times before, and follow a known, tested procedure. A standard change doesn't require individual review each time it's made, because the risk has already been assessed and accepted in advance. A routine patch deployment following an established, tested process is a typical example.
- Normal changes are changes that carry enough risk or novelty to require review by a change advisory function before implementation. This is where a proposed change — what it affects, how it will be tested, and what the rollback plan is — gets evaluated by someone other than the person making it, before it happens rather than after.
- Emergency changes are changes made under genuine urgency, where the normal review process can't happen before the change is needed — typically to resolve an active incident. Emergency changes are expedited, but they're still reviewed, just retroactively: after the fact, to confirm the change was appropriate and to capture what should be done differently next time.
Why "proportionate" is the part that matters
The three-tier structure only works if the tiers are actually calibrated to real risk, and this is where change management programs most often fail. An overly bureaucratic process that requires the same approval rigor for a routine password reset as for a production database migration doesn't make the password reset safer — it just makes the process slow enough that people start finding ways around it.
A bypassed process is worse than no process
When a change management process is disproportionately burdensome for routine work, staff under time pressure will route around it — making the change directly and documenting it after the fact, if at all. At that point, the organization has the operational overhead of a formal process without any of its actual risk-reduction benefit, which is a worse position than simply not having a formal process to begin with.
Right-sizing the process to actual risk — a fast, low-friction path for standard changes, real scrutiny for normal changes, and a retroactive safety net for emergencies — is what keeps the process both effective and sustainable. A process people actually follow, because it isn't disproportionate to the risk of the change being made, catches far more real problems than a stricter process that gets quietly bypassed in practice.
Common mistakes
- A one-size-fits-all approval process applied to every change regardless of risk. This is the most common failure mode — it's disproportionate to the vast majority of actual changes, and it gets routinely bypassed as a result, which defeats the purpose of having a process at all.
- No defined rollback plan as part of the review. A change reviewed for what it's supposed to do, without an equally clear plan for undoing it if it doesn't work, still leaves the organization exposed to an extended outage if the change fails.
- Treating "emergency" as a way to skip review entirely rather than defer it. Emergency changes should still be reviewed — just after the fact — not treated as exempt from any oversight at all.
- Letting the standard-change list go stale. A standard change category that isn't periodically reassessed can end up pre-approving changes that no longer carry the same low risk they once did, as systems and dependencies evolve around them.
FAQ
Who should be part of the change advisory function reviewing normal changes? It varies by organization size, but effective review generally includes someone with technical knowledge of the affected system and someone with visibility into other changes and dependencies that might be affected — the goal is a second perspective the person making the change doesn't have alone.
Does a small IT team really need a formal three-tier change process? The formality can scale down, but the underlying discipline — knowing which changes are routine enough to proceed without review and which need a second look before they happen — matters at any size. Even a two-person IT team benefits from agreeing in advance on what counts as routine versus what needs discussion first.
How do we decide what qualifies as a standard, pre-approved change? A change qualifies as standard when it's been performed successfully many times using a known, tested procedure, and its risk is well understood and consistently low. Anything novel, anything affecting a system with limited testing history, or anything with a meaningful blast radius belongs in the normal-change tier instead.