Skip to main content

​
Monitoring

The server runs Prometheus Node Exporter for system metrics, along with hardware monitoring daemons.
ServicePurpose
node_exporterCPU, memory, disk, network metrics (Prometheus)
smartdS.M.A.R.T. disk health monitoring
lm-sensorsCPU/motherboard temperature and fan sensors
All three are enabled and started on boot. Node Exporter exposes metrics on localhost:9100 — not exposed to the public internet. Useful admin commands: 
# Disk health
smartctl -a /dev/sda

# Temperature and sensors
sensors

# Bandwidth stats
vnstat

# System overview
glances

​
Backups

Backups run via Borgmatic on a systemd timer.

​
What’s backed up

PathContents
/homeAll user home directories
/etcSystem configuration
/var/spool/cronCron jobs
Cache directories, temporary files, and node_modules/ are excluded automatically.

​
Retention policy

FrequencyKeep
Daily7 days
Weekly4 weeks
Monthly6 months

​
Borgmatic commands

# Run backup manually
borgmatic

# List archives
borgmatic list

# Check backup integrity
borgmatic check

# Extract a file from backup
borgmatic extract --archive latest --path /home/username/important-file.txt

​
Security

​
SSH

SettingValue
Ports22 (primary), 2222 (secondary)
AuthenticationKey-only (passwords disabled)
Max auth attempts3
Login grace period30s
Allowed groupspubnix, root, sudo
Port 2222 is useful for users behind firewalls that block 22: 
ssh -p 2222 your-username@atl.sh

​
Fail2ban

Brute-force protection on SSH.
SettingValue
Ban time1 hour (3600s)
Detection window10 minutes (600s)
Max failures5
# View current bans
fail2ban-client status sshd

# Unban an IP
fail2ban-client set sshd unbanip 1.2.3.4

​
UFW Firewall

Open TCP ports:
PortService
22, 2222SSH
70Gopher
79Finger
80HTTP
443HTTPS
1965Gemini
21FTP control
40000–40100FTP passive data
ufw status verbose

​
File Integrity (AIDE)

AIDE checks system file integrity daily at 05:00 UTC and emails a report to root. It monitors system binaries, configuration files, and kernel settings. 
# Run a manual check
aide --check

# Update the database after intentional changes
aide --update
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

​
Audit Logging

auditd records security-relevant events with 40+ rules covering:
  • Identity files (/etc/passwd, /etc/shadow, /etc/sudoers)
  • SSH configuration changes
  • Privilege escalation (sudo, su)
  • Suspicious tools (wget, curl, nc, socat)
  • Scripting interpreters run by users (Python, Perl, Ruby, Vim)
  • System calls: ptrace, memfd_create, execve, chmod, chown
# View recent audit events
ausearch -ts recent

# View sudo usage
ausearch -k priv_esc

# View file permission changes
ausearch -k perm_mod

# Generate a report
aureport --summary

​
Automatic Updates

Unattended security upgrades are enabled via unattended-upgrades. Security patches are applied automatically without requiring manual intervention. 
# View upgrade history
cat /var/log/unattended-upgrades/unattended-upgrades.log

​
User Management

​
Creating a user

just create-user username 'ssh-ed25519 AAAA...' prod
This runs ansible/playbooks/create-user.yml which:
  1. Creates the system account in the users group
  2. Installs their SSH public key
  3. Copies /etc/skel/ to their home directory (including public_html/, public_gemini/, public_gopher/, .plan, .project, .tmux.conf, etc.)

​
Removing a user

just remove-user username prod

​
Log Locations

LogPath
Nginx access/error/var/log/nginx/
Molly Brown (Gemini)/var/log/molly-brown/
Fail2ban/var/log/fail2ban.log
Audit log/var/log/audit/audit.log
System journaljournalctl (1 GB cap)
Borgmaticjournalctl -u borgmatic
# Follow nginx access log
tail -f /var/log/nginx/access.log

# View all journal since last boot
journalctl -b

# View logs for a specific service
journalctl -u sshd -f