Module 16 Lesson 2 of 6 🕑 ~50 min

> cat module-16-2-systematic-techniques.md

Systematic Investigation Techniques

With the initial picture from Lesson 1 in hand, these are the actual moves: work through layers methodically, narrow the search geometrically instead of linearly, and never let more than one variable change at a time.

1 Think in Layers

A simplified layer model: User → Device → OS → Application → Network → DNS → Firewall/Proxy/VPN → Load Balancer → Web Server → Application Server → Authentication System → Database → External Services. When troubleshooting, work out which layers are actually functioning.

Layered troubleshooting test travelling from device through network, DNS, and application layers with an animated marker showing where it stops passing Device / IP Gateway DNS TCP 443 / TLS PASS PASS PASS FAILS HERE

IP, gateway and DNS all pass — the failure is isolated to TLS negotiation before the investigation ever needs to touch the application itself.

Testing https://portal.company.com: does the device have an IP address? Can it reach its default gateway? Can DNS resolve the hostname? Can the destination IP be reached? Can TCP 443 be reached? Does TLS negotiation succeed? Does the HTTP server respond? Does authentication work? Does the application backend respond? Every successful test eliminates a whole category of potential causes — that's the entire value of thinking in layers.

2 Top-Down vs Bottom-Up

Working bottom-up (networking terminology): Layer 1 Physical (power, cable, Wi-Fi signal, network interface, switch port); Layer 2 Data Link (MAC address, VLAN, switch config, ARP, port security); Layer 3 Network (IP address, subnet, gateway, routing); Layer 4 Transport (TCP, UDP, ports, firewalls); Layer 7 Application (HTTP, DNS, SMTP, LDAP, Kerberos, APIs, application behaviour). Extremely useful for network-related incidents specifically.

Working top-down starts at the application instead: Application failing → HTTP error → Authentication failure → Identity provider → Network connectivity → DNS. Experienced engineers pick whichever direction produces useful evidence fastest — there's no universally "correct" starting point, just the one that matches where the initial symptom actually points.

3 Divide and Conquer

Instead of checking everything sequentially from one end, test somewhere in the middle. A user can't reach an application — test whether the application server responds directly. If yes, the problem sits between User → Network → Proxy → Load Balancer; if no, it sits within Application → Database → Backend services. One well-placed test can eliminate half the architecture at once, which is exactly why this technique can dramatically cut investigation time compared with working end to end.

4 Develop a Hypothesis

Troubleshooting is essentially hypothesis testing. Observation: "application works using IP address but not hostname." Hypothesis: "DNS resolution is failing." Test:

Resolve-DnsName portal.company.com
# or
dig portal.company.com

If DNS resolution fails, the hypothesis gets stronger. If it resolves correctly, that hypothesis is disproven and you move to the next one — the value here is having a hypothesis explicit enough that a single test can actually confirm or kill it, rather than a vague hunch you can never quite settle.

5 One Variable at a Time & Evidence Before Action

🔮 Think it through

An engineer restarts the service, changes DNS, disables the firewall, resets the password, and reboots the server — all before the next test. The system starts working. What's actually wrong with this outcome?

Reveal the problem

Nobody knows which change actually fixed it — and worse, some of those changes (disabling the firewall, resetting the password) may now be sitting in production, unexplained, for the next engineer to trip over. Good troubleshooting changes one controlled variable at a time whenever possible, specifically so cause and effect stay traceable.

Before restarting a system, collect evidence — a restart can destroy memory state, temporary logs, active connections, process state, network sessions, authentication sessions, and application state, all at once. Before restarting, capture CPU/memory/disk usage, open connections, application and system logs, thread dumps, process lists, network captures, and the exact error messages. During cybersecurity incidents specifically, restarting systems without collecting evidence first can be genuinely damaging — Module 12, Lesson 1's evidence-preservation content covers exactly why.

Lesson Outcome

You should now be able to work through a problem layer by layer and interpret what each pass/fail result eliminates, choose bottom-up or top-down based on where the symptom points, use divide and conquer to skip past half an architecture in one test, form a hypothesis specific enough that one test can confirm or disprove it, and explain why changing multiple variables at once destroys your ability to know what actually fixed anything. Lesson 3 gives you the tools that actually run these tests — on Windows, Linux, and modern observability platforms.