Log Management, SIEM & Observability

1📋 Centralised Logging & SIEM

Large companies don't expect engineers to manually log into hundreds of servers. Logs get forwarded to central platforms — Splunk, Elastic/Elasticsearch/Kibana, Microsoft Sentinel, IBM QRadar, Google Security Operations, Datadog, Sumo Logic, Graylog, Grafana Loki, OpenSearch, ArcSight — letting analysts search millions or billions of events from one place (Module 11, Lesson 1 covers SIEM technologies in full).

A SIEM (Security Information and Event Management) collects and analyses security-relevant events from Windows, Linux, firewalls, VPN, cloud, Active Directory, EDR, email, proxy and DNS all at once, then correlates them:

IF 10 failed logins occur
AND a successful login follows
AND the source IP is unusual
THEN generate suspicious authentication alert

This is the exact same correlation-rule pattern Module 11's Lesson 1 introduced — a SIEM is, at its core, an automated version of the manual correlation technique Lesson 1 of this module taught you to do by hand.

2⌛ Legacy Logging & Syslog

Not every organisation uses modern JSON logging. You'll meet plain text, CSV, XML, binary proprietary formats, Windows EVT, syslog, SNMP traps, mainframe logs, database audit tables, and flat files with names like server.log, audit.log, trace.log, debug.txt, messages, stdout.log, or stderr.log. Never assume modern observability tooling exists everywhere — a large bank might run systems built decades ago right alongside Kubernetes clusters and cloud services, and enterprise IT frequently means working across both worlds at once.

Syslog is one of the most widely used logging standards in infrastructure — network devices, Linux servers, firewalls and security appliances commonly send syslog messages, traditionally over UDP 514, though TCP and encrypted TLS-based syslog are both widely used too. A syslog message carries a timestamp, hostname, facility, severity, application, and message.

3💾 Log Rotation, Compression & Retention

Logs can't grow forever, so systems rotate them: application.log, application.log.1, application.log.2, application.log.3.gz. Linux commonly uses logrotate for this, with older archives compressed as .gz. The log you need may no longer be in the active file at all — a genuinely common reason an investigation stalls until someone remembers to check the rotated archives.

You don't have to decompress a .gz file first: zgrep "ERROR" application.log.3.gz searches a gzip-compressed log directly.

Retention policies define how long logs are kept — 7 days, 30 days, 90 days, 1 year, or several years, depending on regulation, security policy, operational needs, storage cost, privacy requirements, and contractual obligations. Banking, government, healthcare, telecommunications and critical infrastructure often carry strict audit and retention requirements (Module 12, Lesson 4 covers the regulatory reporting side of this).

4🔒 Audit vs Diagnostic Logs, and Sensitive Data

Diagnostic logs are designed for troubleshooting (DEBUG database connection opened); audit logs are designed to record security or administrative actions (ADMIN user=john ACTION=DELETE_USER TARGET=alice RESULT=SUCCESS). Audit logs often carry stronger retention and integrity requirements, precisely because they may need to stand up as evidence later.

Logs must never intentionally contain passwords, private cryptographic keys, full credit card numbers, authentication secrets, MFA seeds, session cookies, refresh tokens, or access tokens. Developers sometimes accidentally write sensitive values into DEBUG logs — which can turn the logging infrastructure itself into a security risk, since logs are often far more broadly accessible than the systems they describe.

Enterprise logs also frequently contain names, email addresses, IP addresses, device identifiers, location information, and employee/customer IDs — PII that companies must handle under GDPR, UK GDPR, CCPA, or local privacy legislation. Security investigations constantly balance operational visibility against privacy obligations, not just against convenience.

5🛡️ Production Log Levels, Integrity & Missing Logs

Production systems usually run at INFO/WARN/ERROR rather than DEBUG/TRACE, because verbose logging consumes storage, reduces performance, exposes sensitive information, and produces excessive noise. Support engineers may temporarily request DEBUG logging: enable it, reproduce the issue, collect the logs, then disable it again — never leave verbose logging on indefinitely without understanding the impact.

Log integrity matters because logs are only useful if an attacker can't silently modify them. Protections include centralised collection, restricted permissions, immutable storage, digital signatures, write-once storage, separate security accounts, and cloud retention controls. During serious incidents, investigators must always ask: could the attacker have deleted or modified these logs? (Module 12, Lesson 1's evidence-preservation discussion applies directly here.)

Missing logs are themselves useful evidence. If application logs show a request received, but the authentication system shows no matching request, possible explanations include a network failure, an application routing problem, an incorrect endpoint, a proxy or load balancer problem, logging disabled somewhere, the wrong server being inspected, or a timestamp mismatch. The absence of an expected entry can narrow an investigation just as effectively as a present one.

6⏳ Performance Investigation & Distributed Architectures

Logs aren't only for errors — they reveal performance problems too:

10:01:01 Request received
10:01:01 Database query started
10:01:18 Database query completed
10:01:18 Response returned

That transaction took 17 seconds, almost entirely spent waiting on the database — evidence pointing directly at which component actually needs investigating, without a single error line anywhere in the log.

Modern applications increasingly run as microservices, containers, Kubernetes, serverless functions, API gateways, message queues, cloud databases and third-party services. One transaction might travel Browser → CDN → API Gateway → Frontend Service → Authentication Service → Identity Provider → Risk Engine → Database → Notification Service — which is exactly why distributed tracing (next section) has become increasingly important rather than optional.

7📊 Logs, Metrics, Traces & OpenTelemetry

Modern observability typically rests on three pillars. Logs are individual recorded events (Authentication failed for alice). Metrics are numeric measurements over time (Authentication failures per minute = 437). Traces show how a single request travels through multiple services, with timing per hop:

API Gateway      10ms
Auth Service    120ms
Risk Service    900ms
Database         45ms

Together these give far more visibility than logs alone — a trace instantly shows you the Risk Service is the bottleneck here, something no single log line would reveal on its own. OpenTelemetry is an increasingly common standard for unifying logs, metrics and traces across cloud-native organisations, particularly valuable in distributed architectures where requests cross many services and platforms. You don't need to become an observability engineer at this stage — just understand that modern troubleshooting is evolving beyond traditional text logs, and that logs, metrics and traces are complementary, not competing, tools.

Lesson Outcome

You should now be able to explain what centralised logging and SIEM actually do, recognise legacy logging formats and syslog, explain log rotation/compression/retention, distinguish audit from diagnostic logs and identify what should never appear in either, explain log integrity and why missing logs are informative, use logs for performance investigation, and explain how logs, metrics and traces complement each other in a distributed architecture. Lesson 6 pulls everything from this whole module into one repeatable investigation method, hands-on labs, and a final multi-system capstone.