Module 13 Lesson 6 of 6 🕑 ~65 min

> cat module-13-6-practical-troubleshooting.md

Practical Troubleshooting & Capstone

Everything from Lessons 1–5 converges into one repeatable method. This final lesson gives you the method itself, the mistakes to avoid, a set of diagnostic labs, and a capstone that forces you to trace one failing request through an entire enterprise stack.

1 The 9-Step API Troubleshooting Method

When an API fails, investigate systematically — never immediately blame "the API." Break the transaction into layers:

CLIENT → DNS → NETWORK → TLS → PROXY/WAF
   → API GATEWAY → AUTHENTICATION → AUTHORIZATION
   → APPLICATION → DATABASE → DEPENDENCY
  1. Confirm the exact endpoint. What exact URL is being called — hostname, scheme, port, path, API version, query parameters, environment? A surprising number of incidents are simply UAT credentials being sent to a production API, or vice versa.
  2. Confirm DNS. nslookup api.example.com or dig api.example.com — does it resolve to the expected address?
  3. Confirm network connectivity. Can the destination and port actually be reached (typically TCP 443)? Check firewall, security group, network ACL, proxy, routing, VPN, or private endpoint configuration.
  4. Check TLS. Expired certificate, hostname mismatch, missing CA, unsupported TLS version, incorrect SNI, or a client-certificate problem. An error occurring before HTTP communication even begins is not a 401/403 problem — it never got that far.
  5. Inspect the actual HTTP request. Capture method, URL, headers, body, timestamp. Check whether the request genuinely matches the documentation — don't rely on what the developer says the application sends; look at what it really sends.
  6. Inspect the response. Record status, headers, body, timestamp, correlation ID. The response body is frequently far more useful than the status code alone — {"error": "invalid_token", "error_description": "Access token expired"} tells you far more than a bare 401.
  7. Find the server logs. Using the correlation ID, timestamp, username, client ID, source IP, or transaction ID, locate the matching server-side event — it might reveal something like "JWT validation failed: audience mismatch," instantly redirecting the investigation.
  8. Reproduce independently. Use Postman, curl or PowerShell to separate an "API problem" from an "application problem" (Lesson 4).
  9. Compare working and failing requests. One of the strongest troubleshooting techniques there is — capture one that works and one that fails, then compare every field: URL, method, Content-Type, client ID, scope, source IP. The difference usually exposes the root cause directly.

2 Common Beginner Mistakes

Assuming Base64 is encryption
It is not (Lesson 3).
Assuming every bearer token is a JWT
It is not (Lesson 3).
Assuming decoding a JWT validates it
It does not (Lesson 3).
Assuming 401 and 403 mean the same thing
They do not (Lesson 2).
Ignoring Content-Type
It can completely change how a server interprets an otherwise-correct request.
Testing destructive calls against production
Never casually test POST/PUT/PATCH/DELETE against production systems.
Sharing complete access tokens in tickets
Tokens are credentials — redact them (Lesson 5).
Assuming 500 automatically means "server bug"
A backend dependency, database, configuration problem, or unexpected client input may be involved instead.
Only looking at client logs
Enterprise requests often cross five or ten systems — the client's view is one slice of the picture.
Looking only at the HTTP status
Always inspect the response body, headers, correlation ID, and logs as well.

3 Career Relevance

API troubleshooting skill transfers directly into almost every technical role: as a Support Engineer you'll investigate 401/403/404/500/502/504 daily; as an IAM Engineer you'll work with OAuth, OIDC, SCIM, JWT, SAML, bearer tokens and provisioning APIs; as a SOC Analyst you'll pull alerts/events/users/devices/threat data through APIs; as a Security Engineer you'll automate blocking an IP, disabling an account, isolating an endpoint, or revoking a token through APIs; as a Cloud Engineer nearly every action you take has an API behind it; as a DevOps Engineer your automation platforms depend on APIs entirely; and even as a Penetration Tester, modern web and mobile testing frequently means API security testing.

Diagnostic Labs

🦡 Hands-on labs

Start simple, hands-on: call GET /users/1 against https://jsonplaceholder.typicode.com in a browser, then in Postman, and inspect the method, status code, Content-Type and JSON body each time. Try the same request with a nonsensical user ID and compare the response to a valid one. Then work through the diagnostic scenarios below.

🔮 Lab – Troubleshoot a 401

GET /api/v1/users
Host: api.company.com

401 Unauthorized

What would you check, in order?

Reveal the resolution

Decoding the token reveals "aud": "https://test-api.company.com", but the production API is https://api.company.com. Root cause: the token was issued for the test API, not production — exactly the audience-mismatch check from Lesson 2's 401 checklist.

🔮 Lab – 401 vs 403

Scenario A: GET /admin/users → 401 Unauthorized, no Authorization header present.
Scenario B: GET /admin/users → 403 Forbidden, token scope is profile.read but admin.users.read is required.

Explain the difference between these two.

Reveal the explanation

A: authentication is missing entirely — nothing to verify. B: authentication succeeded (the token is real and valid), but authorization is insufficient — the token simply doesn't carry the required scope. Same status-code family question, completely different fix.

🔮 Lab – Content-Type Problem

Documentation says POST /oauth/token expects Content-Type: application/x-www-form-urlencoded with body grant_type=client_credentials&client_id=student&client_secret=secret. A developer instead sends Content-Type: application/json with the equivalent JSON object. What should you change before investigating the credentials themselves?

Reveal the answer

Fix the Content-Type and body encoding first. The credential values are correct — the encoding doesn't match what the endpoint expects, so nothing about the credentials themselves needs investigating yet.

🔮 Lab – Enterprise Incident

Architecture: Mobile App → Cloud WAF → API Gateway → Authentication Service → Database. Customers report 403 Forbidden when authenticating. Authentication-service logs show nothing at all for the affected requests. What does that tell you?

Reveal the resolution

The request may never have reached the authentication application — investigate upstream (WAF, gateway, load balancer) instead of downstream. WAF logs reveal a security rule updated at 10:00 UTC, with failures starting at 10:02 UTC: the WAF is mistakenly flagging part of the authentication request as malicious. Lesson: never assume HTTP 403 = application permissions — exactly the layered-diagram point from Lesson 5.

🔮 Lab – Timeout Investigation

Architecture: Client → API Gateway → Application → Database. Timeouts: Client 60s, Gateway 30s, Application 120s, Database 180s. The database query takes 42 seconds. Users receive 504 Gateway Timeout. Which timeout is actually firing?

Reveal the answer

The API Gateway's 30-second timeout. The database eventually completes successfully at 42 seconds, but the gateway already gave up and terminated the upstream request at 30 — the exact cross-layer timeout trap from Lesson 5.

🔮 Lab – Working vs Broken Request

Working: POST /oauth/token, Content-Type: application/x-www-form-urlencoded, body grant_type=client_credentials&client_id=app01&client_secret=secret → 200 OK.
Failing: Same URL, same credentials, Content-Type: application/json, equivalent JSON body → 400 Bad Request.

The credentials, endpoint and method are all identical. Find the difference.

Reveal the answer

The request encoding is the only difference — this is the core troubleshooting technique from Step 9 above in miniature: compare the complete working request against the complete failing request, field by field, and the difference nearly always jumps out once you're looking at both side by side.

Mini Project – API Incident Report

🦡 Applied project

An HR platform provisions employees into a SaaS application: HR Platform → (OAuth client credentials) → Identity Platform → (SCIM) → SaaS Application. At 09:15 UTC, all provisioning stops. Logs show POST /Users → HTTP 401. The OAuth token request itself still succeeds, and the token looks like:

{
  "iss": "https://identity.company.com",
  "aud": "https://saas-api.company.com",
  "scope": "users.read users.write",
  "iat": 1788076800,
  "exp": 1788080400
}

🔮 Investigate before revealing

The token request succeeds but every provisioning call still gets a 401. What would you check?

Reveal the root cause

The SaaS vendor changed the expected API audience from https://saas-api.company.com to https://api.saas-company.com after a platform migration — the token is valid and well-formed, but its aud claim no longer matches what the SaaS API actually expects post-migration, so every call fails authentication even though nothing on the HR/Identity side changed at all.

Produce a full report: incident summary, architecture diagram, observed error, evidence, root cause, impact, resolution, and prevention recommendations (integration regression testing, vendor change notifications, monitoring for a spike in 401 responses, UAT testing before production migrations, configuration management, and an API dependency inventory). This connects directly back to Module 12 (Incident Response), Module 11 (SOC monitoring), and Module 10 (IAM).

Final Capstone Challenge

🦡 Assessed exercise

Application error: "Unable to retrieve customer profile. HTTP 403." Architecture: Mobile App → Internet → WAF → API Gateway → Customer API → Database. The application's actual request:

GET /api/v2/customers/81483 HTTP/1.1
Host: api.company.com
Authorization: Bearer eyJ...
Accept: application/json
X-Correlation-ID: d73c3e14

The token decodes to {"sub": "8394", "aud": "customer-api", "scope": "customer.read", "exp": 1788092034} — correct audience, correct scope, not expired. The Customer API's own logs show: "No request found for correlation ID d73c3e14."

🔮 Investigate before revealing

The token looks entirely correct. Where should the investigation move next, and why?

Reveal the resolution

Since the Customer API has no record of the request at all, it apparently never arrived — investigate upstream (API Gateway, then WAF), not the token. Gateway logs show nothing either. WAF logs reveal: Request blocked, Rule: GeoRestriction, Country: XX, Correlation: d73c3e14. Root cause: the WAF blocked the client's source region before the request ever reached authentication or authorization. The key lesson: this 403 did not originate from the API at all — exactly the layered-architecture trap Lesson 5 introduced, now solved end to end using every step of this lesson's 9-step method.

Module 13 Outcome

An API request is never simply Application → API. Real enterprise communication looks more like:

CLIENT↓ DNS · FIREWALL/PROXY · TLS · WAF↓ LOAD BALANCER · API GATEWAY↓ AUTHENTICATION · AUTHORIZATION↓ APPLICATION · MICROSERVICE↓ DATABASE

A skilled IT professional learns to determine where in that chain a request actually failed. The most important skills from this module were never about memorising every HTTP status code — they're about learning to ask: what endpoint am I calling? Which HTTP method? What headers and authentication am I presenting? What does the body contain? What status and response body did I actually get back? Did the request even reach the application? What do the logs show? Can I reproduce it independently with Postman or curl? What differs between the working request and the failing one?

Once you can answer those questions confidently, APIs stop looking mysterious. They become just another system that can be methodically troubleshot — the same investigative discipline this entire course has been building, module after module, now applied to the layer that quietly holds the rest of the enterprise together.