Linux

Linux Gotchas

Permission Traps

chmod 777 fixes nothing, breaks everything — find the actual owner/group issue
Setuid on scripts is ignored for security — only works on binaries
chown -R follows symlinks outside target directory — use --no-dereference
Default umask 022 makes files world-readable — set 077 for sensitive systems
ACLs override traditional permissions silently — check with getfacl

Process Gotchas

kill sends SIGTERM by default, not SIGKILL — process can ignore it
nohup doesn't work if process already running — use disown instead
Background job with & still dies on terminal close without disown or nohup
Zombie processes can't be killed — parent must call wait() or be killed
kill -9 skips cleanup handlers — data loss possible, use SIGTERM first

Filesystem Traps

Deleting open file doesn't free space until process closes it — check lsof +L1
rm -rf /path / with accidental space = disaster — use rm -rf /path/ trailing slash
Inodes exhausted while disk shows space free — many small files problem
Symlink loops cause infinite recursion — find -L follows them
/tmp cleared on reboot — don't store persistent data there

Disk Space Mysteries

Deleted files held open by processes — lsof +L1 shows them, restart process to free
Reserved blocks (5% default) only for root — tune2fs -m 1 to reduce
Journal eating space — journalctl --vacuum-size=500M
Docker overlay eating space — docker system prune -a
Snapshots consuming space — check LVM, ZFS, or cloud provider snapshots

Networking

localhost and 127.0.0.1 may resolve differently — check /etc/hosts
Firewall rules flushed on reboot unless saved — iptables-save or use firewalld/ufw persistence
netstat deprecated — use ss instead
Port below 1024 requires root — use setcap for capability instead
TCP TIME_WAIT exhaustion under load — tune net.ipv4.tcp_tw_reuse

SSH Traps

Wrong permissions on ~/.ssh = silent auth failure — 700 for dir, 600 for keys
Agent forwarding exposes your keys to remote admins — avoid on untrusted servers
Known hosts hash doesn't match after server rebuild — remove old entry with ssh-keygen -R
SSH config Host blocks: first match wins — put specific hosts before wildcards
Connection timeout on idle — add ServerAliveInterval 60 to config

Systemd

systemctl enable doesn't start service — also need start
restart vs reload: restart drops connections, reload doesn't (if supported)
Journal logs lost on reboot by default — set Storage=persistent in journald.conf
Failed service doesn't retry by default — add Restart=on-failure to unit
Dependency on network: After=network.target isn't enough — use network-online.target

Cron Pitfalls

Cron has minimal PATH — use absolute paths or set PATH in crontab
Output goes to mail by default — redirect to file or /dev/null
Cron uses system timezone, not user's — set TZ in crontab if needed
Crontab lost if edited incorrectly — crontab -l > backup before editing
@reboot runs on daemon restart too, not just system reboot

Memory and OOM

OOM killer picks "best" victim, often not the offender — check dmesg for kills
Swap thrashing worse than OOM — monitor with vmstat
Memory usage in free includes cache — "available" is what matters
Process memory in /proc/[pid]/status — VmRSS is actual usage
cgroups limit respected before system OOM — containers die first

Commands That Lie

df shows filesystem capacity, not physical disk — check underlying device
du doesn't count sparse files correctly — file appears smaller than disk usage
ps aux memory percentage can exceed 100% (shared memory counted multiple times)
uptime load average includes uninterruptible I/O wait — not just CPU
top CPU percentage is per-core — 400% means 4 cores maxed