🔥 Linux Crisis Investigation

⚠️ URGENT: Critical Production Server Failure ⚠️

FROM: Senior Network Administrator

TO: Junior Linux Administrator (YOU)

PRIORITY: CRITICAL - IMMEDIATE ACTION REQUIRED

⚠️ SITUATION STATUS ⚠️

Our main production server has experienced multiple critical failures. Services are down. Network connectivity is compromised. Log files are corrupted. We're losing money every minute.

I'm in an emergency meeting with stakeholders. YOU are our only hope on the ground right now.

📋 YOUR MISSION:

Investigate 5 critical system failures
Piece together clues from cryptic error messages
Identify root causes using your Linux knowledge
Provide detailed incident reports
Recommend solutions to prevent future disasters

⏰ TIME LIMIT: 60 MINUTES

Every minute counts. The executives are breathing down my neck. Don't let me down.

— Chief SysAdmin

🎯 HOW THIS WORKS:

Read error logs, terminal outputs, and system clues
Use your Unit 07 knowledge to diagnose problems
Write detailed explanations (not just answers!)
You MUST explain WHY and HOW, not just WHAT
Hints available but they reveal you're struggling...

⚠️ WARNING: This is not a simple quiz. You WILL get frustrated. You WILL need to think hard. That's the point. Real incidents don't come with multiple choice answers.

● CRITICAL ELAPSED TIME:

00:00

📊 INVESTIGATION STATUS

Solved: 0/5

Score: 0

Hints Used: 0

INCIDENT #1

🗂️ The Vanishing Configuration Files

ROOKIE LEVEL

admin@production-server:~$

admin@server:~$ sudo systemctl restart networking

Failed to restart networking.service: Unit file networking.service not found.

admin@server:~$ ls /etc/net*

ls: cannot access '/etc/net*': No such file or directory

admin@server:~$ cat /etc/hosts

cat: /etc/hosts: No such file or directory

admin@server:~$ ls /etc/ | wc -l

admin@server:~$ df -h

/dev/sda1 50G 48G 0 100% /

/dev/sda2 100G 1.2G 94G 2% /home

💡 CONTEXT: A junior administrator ran some "cleanup scripts" yesterday. They claimed to free up disk space. Now the network service won't start and critical config files are missing from /etc/. Something went horribly wrong...

🔍 INVESTIGATION REPORT: What happened and WHY?

💡 HINT: Look at the disk space usage. The root partition (/) is 100% full. The /etc directory contains only 3 files instead of hundreds. Think about what happens when someone tries to "free up space" without understanding which directories are critical vs. which are safe to clean. What does /etc contain?

INCIDENT #2

⚙️ The Zombie Process Apocalypse

EXPERT LEVEL

admin@production-server:~$

admin@server:~$ top

Tasks: 873 total, 2 running, 768 zombie, 103 sleeping

%Cpu(s): 98.2 us, 1.8 sy

KiB Mem: 16384000 total, 16380000 used, 4000 free

PID USER PR NI VIRT RES S %CPU %MEM TIME+ COMMAND

1234 webapp 20 0 850000 800000 R 87.3 48.8 145:23 python3

1235 webapp 20 0 0 0 Z 0.0 0.0 0:00.00 python3

1236 webapp 20 0 0 0 Z 0.0 0.0 0:00.00 python3

1237 webapp 20 0 0 0 Z 0.0 0.0 0:00.00 python3

... (765 more zombie processes)

admin@server:~$ ps aux | grep defunct | wc -l

768

admin@server:~$ kill -9 1235

admin@server:~$ ps aux | grep 1235

1235 webapp 20 0 0 0 Z 0.0 0.0 0:00.00 python3

[ERROR] WebApp: Failed to spawn child process - Resource temporarily unavailable

[WARN] WebApp: Parent process not reaping children

[ERROR] System: fork() failed - Cannot allocate memory

[INFO] WebApp: Attempting to spawn worker #769...

[ERROR] Kernel: Process table full

💡 CONTEXT: The web application has been spawning child processes all night. Users report the website is incredibly slow. New processes can't start. Someone tried to kill the zombie processes but they won't die. The server is basically frozen.

🔍 INVESTIGATION REPORT: What's causing this and HOW do you fix it?

💡 HINT: Zombie processes are already dead - they're just waiting for their parent to read their exit status. You can't kill what's already dead! Look at PID 1234 - it's the parent process consuming CPU. The application is spawning children but not properly "reaping" them with wait(). You need to address the parent, not the zombies.

INCIDENT #3

📝 The Authentication Attack

EXPERT LEVEL

Dec 04 03:15:22 server sshd[8234]: Accepted publickey for admin from 192.168.1.50

Dec 04 03:47:11 server sshd[9821]: Failed password for root from 45.142.214.198

Dec 04 03:47:14 server sshd[9821]: Failed password for root from 45.142.214.198

Dec 04 03:47:17 server sshd[9823]: Failed password for root from 45.142.214.198

Dec 04 03:47:19 server sshd[9825]: Failed password for admin from 45.142.214.198

Dec 04 03:47:22 server sshd[9827]: Failed password for admin from 45.142.214.198

Dec 04 03:47:25 server sshd[9829]: Failed password for admin from 45.142.214.198

Dec 04 03:47:28 server sshd[9831]: Accepted password for admin from 45.142.214.198

Dec 04 03:48:01 server sudo: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/bin/bash

Dec 04 03:49:15 server useradd[9945]: new user: name=backup_svc, UID=1337, GID=1337

Dec 04 03:49:23 server usermod[9950]: add 'backup_svc' to group 'sudo'

Dec 04 03:50:02 server sshd[9987]: Accepted publickey for backup_svc from 45.142.214.198

Dec 04 04:12:33 server cron[10234]: (backup_svc) CMD (/tmp/.system/monitor.sh)

Dec 04 06:15:11 server kernel: TCP: too many orphaned sockets

Dec 04 06:15:44 server NetworkManager: Device wlan0 disconnected

admin@production-server:~$

admin@server:~$ last | grep 45.142.214.198

admin pts/0 45.142.214.198 Dec 4 03:47 - 04:55 (01:08)

backup_s pts/1 45.142.214.198 Dec 4 03:50 still logged in

admin@server:~$ cat /etc/passwd | grep backup_svc

backup_svc:x:1337:1337::/home/backup_svc:/bin/bash

admin@server:~$ groups backup_svc

backup_svc : backup_svc sudo

admin@server:~$ ls -la /tmp/.system/

total 12

-rwxr-xr-x 1 backup_svc backup_svc 8192 Dec 4 03:49 monitor.sh

💡 CONTEXT: Network security alarms went off at 3:47 AM. The CEO's assistant called you in a panic saying someone unauthorized accessed the admin account. By the time you check, there's a new user account you don't recognize with sudo privileges. Something bad happened here...

🔍 INCIDENT REPORT: Reconstruct the attack timeline and identify vulnerabilities

💡 HINT: This is a brute force attack followed by persistence mechanism. The attacker tried multiple passwords on 'root' and 'admin' accounts from IP 45.142.214.198 until one succeeded. After gaining access, they created a backdoor user account (backup_svc) with sudo privileges and set up a cron job to maintain access. The weak admin password was the vulnerability. Look at which log file this is from - what command shows authentication attempts?

INCIDENT #4

📦 The Broken Package Manager

ROOKIE LEVEL

admin@production-server:~$

admin@server:~$ sudo apt install nginx

E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 3456

E: Unable to acquire the dpkg frontend lock, is another process using it?

admin@server:~$ ps aux | grep 3456

root 3456 0.0 0.0 0 0 ? Z Dec03 0:00 [apt-get]

admin@server:~$ sudo apt update

E: Could not get lock /var/lib/apt/lists/lock

admin@server:~$ sudo dpkg --configure -a

dpkg: error: dpkg frontend lock is locked by another process

admin@server:~$ ls -la /var/lib/dpkg/lock*

-rw-r----- 1 root root 0 Dec 3 14:23 /var/lib/dpkg/lock

-rw-r----- 1 root root 0 Dec 3 14:23 /var/lib/dpkg/lock-frontend

💡 CONTEXT: Someone was installing packages yesterday when the server suddenly lost power. Ever since, no one can install, update, or remove any software. The package manager is completely frozen. Production deployments are blocked. This needs to be fixed ASAP.

🔍 INVESTIGATION REPORT: What's wrong and how do you safely fix it?

💡 HINT: Lock files prevent multiple package managers from running simultaneously, which could corrupt the package database. Process 3456 is a zombie (defunct) from an interrupted apt-get operation. Since it's dead, the locks weren't released properly. You need to: 1) Verify no other package managers are running, 2) Kill or wait for the parent of the zombie, 3) Safely remove the lock files, 4) Run 'sudo dpkg --configure -a' to fix any half-installed packages.

INCIDENT #5

🌐 The DNS Mystery

NIGHTMARE MODE

admin@production-server:~$

admin@server:~$ ping google.com

ping: google.com: Temporary failure in name resolution

admin@server:~$ ping 8.8.8.8

64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=12.3 ms

64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=11.8 ms

admin@server:~$ cat /etc/resolv.conf

# This file is managed by systemd-resolved

nameserver 127.0.0.53

admin@server:~$ systemctl status systemd-resolved

● systemd-resolved.service - Network Name Resolution

Loaded: loaded (/lib/systemd/system/systemd-resolved.service)

Active: failed (Result: exit-code) since Wed 2024-12-04 03:15:22

Process: 1823 ExecStart=/lib/systemd/systemd-resolved (code=exited, status=1/FAILURE)

admin@server:~$ journalctl -u systemd-resolved -n 20

Dec 04 03:15:22 systemd-resolved[1823]: Failed to read configuration file: Permission denied

Dec 04 03:15:22 systemd-resolved[1823]: /etc/systemd/resolved.conf: Permission denied

admin@server:~$ ls -l /etc/systemd/resolved.conf

-rw------- 1 root root 892 Dec 3 22:45 /etc/systemd/resolved.conf

admin@server:~$ ls -l /etc/systemd/resolved.conf.d/

total 4

-rw-r--r-- 1 admin admin 156 Dec 3 22:47 custom-dns.conf

admin@server:~$ cat /etc/systemd/resolved.conf.d/custom-dns.conf

[Resolve]

DNS=192.168.1.1

FallbackDNS=8.8.8.8 8.8.4.4

admin@server:~$ ping 192.168.1.1

From 192.168.1.50: icmp_seq=1 Destination Host Unreachable

💡 CONTEXT: Users started complaining they can't access ANY websites by name - only by IP address. Email won't send. The web application can't connect to external APIs. Everything that relies on domain names is broken. You notice someone made "DNS improvements" yesterday evening. This is a multi-layered problem...

🔍 CRITICAL INVESTIGATION: Multiple DNS failures - identify ALL issues

💡 MAJOR HINT: There are TWO problems here: Problem 1: systemd-resolved service is crashed because /etc/systemd/resolved.conf has wrong permissions (should be 644 not 600). The service can't read its own config file! Problem 2: Even if you fix that, the custom DNS server 192.168.1.1 is unreachable (probably the router is down or changed IP). You need to either fix network connectivity to that DNS server OR modify custom-dns.conf to use working DNS servers like 8.8.8.8 directly. The fix requires: 1) chmod 644 /etc/systemd/resolved.conf, 2) Fix or remove the unreachable DNS config, 3) systemctl restart systemd-resolved, 4) Test with ping google.com

🎉 INVESTIGATION COMPLETE 🎉

All critical incidents have been documented.

Senior Administrator Review:

FINAL SCORE: 0 points

Incidents Resolved: 0/5

Time Taken: 00:00

Hints Used: 0

🔥 SYSTEM CRISIS INVESTIGATION 🔥

⚠️ URGENT: Critical Production Server Failure ⚠️

🔥 CRISIS INVESTIGATION IN PROGRESS 🔥

🎉 INVESTIGATION COMPLETE 🎉