Root Cause Analysis

1🔍 Root Cause Analysis & Symptom vs. Root Cause

Consider: the application stopped responding, an engineer restarted the server, the application started working again. The service is restored — but why did the server stop responding in the first place? Possible causes include a memory leak, database exhaustion, a full disk, a certificate failure, thread exhaustion, a software defect, a network failure, or a dependency outage. This is where root cause analysis becomes important, and it means repeatedly separating the symptom from the cause:

A strong investigation keeps asking "why?" at every layer instead of stopping at the first explanation that makes the alert go away.

2❓ The Five Whys & Timeline-Based RCA

One simple RCA technique is the Five Whys — keep asking "why" until you hit something you can actually fix:

Why could users not authenticate?
  Because the authentication server was unavailable.
Why was it unavailable?
  Because the application crashed.
Why did the application crash?
  Because memory usage reached 100%.
Why did memory usage reach 100%?
  Because a process continuously allocated memory.
Why did the process continuously allocate memory?
  Because of a memory leak introduced in the latest software release.

Root cause: Software memory leak.

For major incidents, a timeline often reveals cause and response effectiveness better than a five-step chain alone:

08:00  New application version deployed
08:20  Memory usage begins increasing
09:15  First authentication errors
09:22  Monitoring alert generated
09:27  Service desk receives user reports
09:30  Incident declared
09:48  Application restarted
09:52  Service restored
11:30  Engineering identifies memory leak

3⚡ Root Cause vs. Trigger & Contributing Factors

These aren't always the same thing. A backup process starting is a trigger; if it then fills the disk because of insufficient capacity combined with uncontrolled temporary file growth, that combination is the underlying root cause — the backup itself didn't necessarily contain a defect, it just exposed a weakness that was already there.

Major incidents often have multiple contributing factors, and good RCA resists the temptation to oversimplify:

Root Cause: Database connection leak.

Contributing Factors:
- Monitoring threshold too high
- No automated restart
- Insufficient capacity
- Alert not routed to on-call engineer
- Deployment testing did not include sustained load

4🔧 Corrective Actions, Workaround vs. Fix & Known Errors

An RCA should produce actual actions, usually spanning several time horizons:

Immediate:  Restart affected service.
Short Term: Increase database connection pool.
Medium Term: Deploy software patch.
Long Term:  Add connection utilisation monitoring.
Process:    Add load testing before future releases.

A workaround reduces impact without removing the root cause — restarting the service every 24 hours might prevent the leak from causing outages, while the permanent fix is actually upgrading to the version that corrects the leak. Tickets should clearly state which one you delivered: "Workaround implemented, permanent fix pending" is a very different statement from "resolved."

Some organisations maintain a known error database so the next engineer doesn't have to re-run the same investigation: "Known Error: Authentication service may exhaust memory after approximately seven days under high load. Workaround: restart service. Permanent Fix: upgrade to release 8.x.x." Repeated resolutions are exactly the kind of thing that should graduate into a Knowledge Base article (Module 17 covers writing these well) — if support gets 100 tickets asking how to clear the Windows DNS cache, that's a KB article, not 100 separate investigations.

5📝 Good Ticket Titles & the Description Template

A useful description template to reach for every time:

ISSUE            Describe the problem.
BUSINESS IMPACT  Who is affected? How many users? Production or test?
START TIME       When did the problem begin?
ENVIRONMENT      Production / UAT / Development, app version, OS, cloud platform
EXPECTED RESULT  What should happen?
ACTUAL RESULT    What actually happens?
ERROR            Exact error message.
REPRODUCTION     Steps required to reproduce.
CHANGES          Any recent changes?
TROUBLESHOOTING  What has already been tested?
LOGS             Relevant timestamps and correlation IDs.