Module 14 Lesson 6 of 6 🕑 ~65 min

> cat module-14-6-investigation-method-capstone.md

Investigation Method, Labs & Capstone

Everything from Lessons 1–5 converges into one repeatable method. This final lesson gives you the method, the mistakes to avoid, a set of realistic investigation scenarios, and a capstone that has you build a complete incident timeline from raw multi-system evidence.

1 Common Mistakes & the 8-Step Method

Watch for these recurring mistakes: searching only for ERROR (important events are often INFO/WARN/SUCCESS); ignoring timestamps (correlation becomes unreliable without them); assuming every system shares a timezone (always verify, per Lesson 2); checking only one server when the system is load-balanced (the user's request may have hit APP03 while you're staring at APP01); assuming the visible error is the root cause (a "login failed" message can hide an LDAP timeout, database failure, DNS failure, certificate problem, or API failure underneath); searching only by username instead of also using timestamp/session ID/request ID/source IP/device ID; ignoring events that preceded the failure by seconds or minutes; and changing or restarting things before collecting evidence, which can destroy exactly what you needed.

Use this process consistently:

  1. Define the problem. E.g. "User Alice cannot authenticate to the customer portal."
  2. Establish the time. Convert the reported time into the timezone the servers actually use (Lesson 2).
  3. Identify relevant systems. Browser, WAF, load balancer, web server, authentication server, LDAP, database, MFA provider — whatever the architecture actually involves.
  4. Identify searchable values. Email address, source IP, device ID, approximate time, request ID, session ID.
  5. Follow the transaction. Start where the request entered the architecture and follow it through each component in order.
  6. Identify the first failure. Don't stop at the final, most visible error message.
  7. Confirm with another log source. Strong investigations rest on more than one piece of evidence.
  8. Document the timeline. Turn scattered log lines into one coherent, chronological story.

2 Evidence vs Assumption, Writing Findings & Escalating

Good analysts keep evidence and assumption clearly separate. Evidence: "Firewall log shows connection from 10.1.5.20 → 10.1.8.30:443 was allowed at 12:01:22." Assumption: "The application therefore received the request." The second statement hasn't been proven — another system could still have blocked or dropped that connection somewhere between the firewall and the application (exactly Module 13, Lesson 5's layered-architecture point).

Write findings professionally. Avoid "the server broke." Instead:

At 14:32:17 UTC, APP03 attempted to connect to the authentication service at 10.20.5.15:443. The connection timed out after approximately 30 seconds. Firewall logs show that traffic from APP03 to 10.20.5.15:443 was permitted. No corresponding request appears in the authentication application's access logs. Further investigation of the network path and load balancer is recommended.

This distinguishes facts, observations, conclusions, and next steps — exactly the discipline Module 12's executive-summary lesson also taught, now applied to a technical audience instead of an executive one.

When escalating to another team, provide the problem description, exact timestamp and timezone, affected user/device, affected environment, relevant log snippets, correlation/request/session IDs, what's already been tested, and expected vs actual behaviour. Don't send a 5GB log archive with "please investigate" — make the evidence easy to find, or the escalation just becomes someone else's identical starting point.

3 Redacting Logs & the SOC Investigation Mindset

Before sharing logs externally, inspect them for sensitive information and redact passwords, access/refresh tokens, API keys, customer data, personal information, private URLs, internal IPs where appropriate, and cryptographic material. Authorization: Bearer eyJhbGciOi... should normally become Authorization: Bearer [REDACTED] before it leaves your hands (Lesson 5's sensitive-data discussion, and Module 13, Lesson 5's secret-management rules, both apply here).

SOC analysts (Module 11) repeatedly ask the same structured questions of every piece of evidence: Who (which user, machine, service account)? What happened? When (exact timestamp, timezone)? Where (source system, destination, IP)? How (authentication method, protocol, process, API)? And what happened next (successful login, privilege escalation, new process, network connection, data access)? This is the same investigative shape every lesson in this module has been building toward, phrased as one repeatable checklist.

Investigation Scenarios

🦡 Hands-on investigations

🔮 Scenario 1 – SSH Brute Force

Aug 30 02:14:11 linux01 sshd[14211]: Failed password for root from 198.51.100.24 port 49133 ssh2
Aug 30 02:14:12 linux01 sshd[14217]: Failed password for root from 198.51.100.24 port 49139 ssh2
Aug 30 02:14:13 linux01 sshd[14220]: Failed password for admin from 198.51.100.24 port 49145 ssh2
Aug 30 02:14:14 linux01 sshd[14222]: Failed password for ubuntu from 198.51.100.24 port 49152 ssh2

What service is under attack, from where, and what technique does this suggest?

Reveal the analysis

SSH, from 198.51.100.24, repeatedly guessing common usernames — consistent with SSH brute-force/credential guessing. Next: did any login succeed? Is SSH exposed to the internet? What owns that IP/network range? Was key-based auth or MFA enabled? Were any commands executed after a successful login?

🔮 Scenario 2 – Windows Account Compromise

09:41:11 Event ID 4625  User: admin  Source IP: 203.0.113.77
09:41:13 Event ID 4625  User: admin  Source IP: 203.0.113.77
09:41:15 Event ID 4625  User: admin  Source IP: 203.0.113.77
09:41:19 Event ID 4624  User: admin  Source IP: 203.0.113.77
09:42:02 Event ID 4720  New account: support_admin

What happened, and how suspicious is this?

Reveal the analysis

Several failed logons, then a success, then a brand-new account created less than a minute later — treat this as suspicious until proven otherwise. Investigate logon type, the destination computer, process activity, group membership changes, EDR telemetry, VPN logs, and whether this matches known, approved administrator activity.

🔮 Scenario 3 – Password Spraying vs Brute Force

02:10 alice LOGIN_FAILED src=198.51.100.55
02:10 bob LOGIN_FAILED src=198.51.100.55
02:11 charles LOGIN_FAILED src=198.51.100.55
02:11 david LOGIN_FAILED src=198.51.100.55

Compare this to five failed attempts, all for alice, from the same IP. What's the difference?

Reveal the answer

Password spraying: a small number of passwords against many accounts, staying under per-account lockout thresholds. Brute force: many passwords against one account. Same source, opposite shape — and the shape is the whole signal (Lesson 3, Section 1).

🔮 Scenario 4 – Suspicious PowerShell

User: CORP\jsmith
Process: powershell.exe
Parent: WINWORD.EXE
Command: powershell.exe -EncodedCommand ...

What questions does this raise, and is PowerShell itself the problem?

Reveal the analysis

Why did Word launch PowerShell at all? Was a document opened immediately beforehand? What does the encoded command actually do? Did PowerShell create network connections or files? Did another process start afterward? The WINWORD.EXE → powershell.exe relationship is more important than PowerShell's mere presence — PowerShell is a completely legitimate admin tool (Module 11, Lesson 2); context is what determines whether this specific instance is suspicious.

🔮 Scenario 5 – Impossible Timeline

Application server: 12:05:18 Login succeeded. Authentication server: 12:05:24 Authentication request received. How can authentication succeed six seconds before the request was even received?

Reveal the explanation

Almost certainly clock drift, a timezone mismatch, different NTP sources, or plain incorrect system time on one of the two servers (Lesson 2). Before building an elaborate theory about what happened, always verify time synchronisation first — a surprising number of "impossible" timelines turn out to be nothing more than a clock problem.

Capstone – Full Incident Investigation

🦡 Assessed exercise

Architecture: Internet → CDN → WAF → Load Balancer → Web Servers → API Gateway → Authentication Service → Active Directory → MFA Provider → Database. Scenario: at 08:15 UTC, the SOC detected suspicious activity involving a finance employee. You're given logs from the firewall, VPN, Windows, Active Directory, DNS, proxy, EDR, and a cloud application.

🔮 Investigate before revealing

Determine: how the attacker entered, which account was compromised, which endpoint was involved, what processes ran, which systems were contacted, whether privileges were escalated, and whether data may have been accessed. Produce a final incident timeline.

Reveal an example timeline
08:02:11 User received phishing email
08:04:27 User accessed suspicious website through corporate proxy
08:05:14 WINWORD.EXE launched PowerShell
08:05:17 PowerShell contacted suspicious domain
08:07:43 Credentials for CORP\jsmith used against SERVER03
08:07:44 Windows Event ID 4624 confirms successful network login
08:09:11 Administrator account "svc_backup2" created
08:09:29 svc_backup2 added to Administrators group
08:14:07 Large number of finance files accessed

This is exactly what an individual log entry becoming an investigation looks like — no single line above proves the whole story, but read in sequence they reconstruct a complete phishing-to-data-access incident, the same skill Module 12's ransomware case study built at a larger scale.

For every investigation, fill in a consistent record: incident, affected system, affected user, start time and timezone, source IP, destination, session ID, request ID, relevant log files, first observed failure, root cause, supporting evidence, and recommended action. This discipline is what separates a structured investigator from someone guessing — and when asked "which logs would you check, in what order?" for an architecture like the one above, a sensible answer works outward from where the failure was reported: WAF/access log, load balancer, web application, API gateway, authentication service, directory authentication, MFA service, then database — adjusting the order based on what each earlier log actually reveals.

Module 14 Outcome

Employers rarely expect a junior IT or cybersecurity professional to memorise every log format. What they care about is whether you can identify the correct log source, find the relevant time period, search large logs efficiently, recognise important fields, correlate events, understand timestamps, follow session IDs and request IDs, distinguish symptoms from root causes, document evidence clearly, and escalate with genuinely useful information. These skills transfer between products — a Splunk engineer, a Windows administrator, an IAM engineer and a SOC analyst use different tools, but the investigative thinking underneath is remarkably similar.

The single most important lesson of this module:

Do not simply search for errors. Reconstruct what happened.

Think in terms of WHO, WHAT, WHEN, WHERE, HOW — then ask what happened immediately before this, what happened immediately afterwards, and what other system should have recorded the same transaction. A good investigator can start with "the user cannot log in" and end with:

14:31:58 User submits credentials
14:31:58 Web server receives request
14:31:59 Authentication service receives request
14:31:59 Authentication service contacts LDAP
14:32:29 LDAP connection times out
14:32:29 Authentication service returns backend error
14:32:29 Web application converts error into HTTP 500
14:32:30 User sees "Login failed"

That's the difference between simply reading logs and actually investigating an incident — one of the most valuable, transferable skills you can build for an IT or cybersecurity career, and one that every module from here forward will keep drawing on.