System Administration Rules

User Management

• Create service accounts with --system flag — no home directory, no login shell

• sudo with specific commands, not blanket ALL — principle of least privilege

• Lock accounts instead of deleting: usermod -L — preserves audit trail and file ownership

• SSH keys in ~/.ssh/authorized_keys with restrictive permissions — 600 for file, 700 for directory

• visudo to edit sudoers — catches syntax errors before saving, prevents lockout

Process Management

• systemctl for services, not service — systemd is standard on modern distros

• journalctl -u service -f for live logs — more powerful than tail on log files

• nice and ionice for background tasks — don't compete with production workloads

• Kill signals: SIGTERM (15) first, SIGKILL (9) last resort — SIGKILL doesn't allow cleanup

• nohup or screen/tmux for long-running commands — SSH disconnect kills regular processes

File Systems and Storage

• df -h for disk usage, du -sh * to find culprits — check before disk fills completely

• lsof +D /path finds processes using a directory — needed before unmounting

• ncdu for interactive disk usage — faster than repeated du commands

• Mount options matter: noexec, nosuid for security on data partitions

• Resize filesystems with care: grow is safe, shrink risks data loss — always backup first

Logs and Monitoring

• logrotate prevents disk fill — configure size limits and retention

• Centralize logs to external system — local logs lost if server dies

• /var/log/auth.log or /var/log/secure for login attempts — watch for brute force

• dmesg for kernel messages — hardware errors, OOM kills appear here

• Monitor inode usage, not just disk space — many small files exhaust inodes

Permissions and Security

• chmod 600 for secrets, 640 for configs, 644 for public — world-writable is almost never correct

• Sticky bit on shared directories (chmod +t) — users can only delete their own files

• setfacl for complex permissions — when traditional owner/group/other isn't enough

• chattr +i makes files immutable — even root can't modify without removing flag

• SELinux/AppArmor in enforcing mode — permissive logs but doesn't protect

Package Management

• apt update before apt upgrade — upgrade without update uses stale package lists

• Unattended security updates: unattended-upgrades — critical patches shouldn't wait

• Pin package versions in production — unexpected upgrades cause unexpected outages

• Remove unused packages: apt autoremove — reduces attack surface and disk usage

• Know your package manager: apt/yum/dnf/pacman — commands differ, concepts similar

Backups

• Test restores regularly — backups that can't restore are worthless

• Include package lists and configs, not just data — recreating environment is painful

• Offsite backups mandatory — local backups don't survive disk failure or ransomware

• Backup before any risky change — "I'll just quickly edit" famous last words

• Document restore procedure — 3am disaster is wrong time to figure it out

Performance

• top/htop for live view, vmstat for trends — understand baseline before diagnosing

• iotop for disk I/O bottlenecks — slow disk often blamed on CPU

• Load average: 1.0 per core is healthy — consistently higher means queuing

• Swap usage isn't inherently bad — but consistent swapping indicates memory shortage

• sar for historical data — retroactively diagnose what happened during incident

Networking Basics

• ss -tulpn shows listening ports — netstat is deprecated

• ip addr and ip route replace ifconfig and route — learn the new tools

• Check both host firewall and cloud security groups — traffic blocked at either level fails

• /etc/hosts for local overrides — quick testing without DNS changes

• curl -v shows full connection details — headers, timing, TLS handshake

Common Mistakes

• Running services as root — one exploit owns the system

• No monitoring until something breaks — reactive is expensive

• Editing config without backup — cp file file.bak takes two seconds

• Rebooting to "fix" issues — masks the problem, it'll return

• Ignoring disk space warnings — 100% full causes cascading failures

• Forgetting timezone configuration — logs from different servers don't correlate