Documentation Index
Fetch the complete documentation index at: https://docs.allthingslinux.org/llms.txt
Use this file to discover all available pages before exploring further.
This page is your starting point for diagnosing problems across the atl.chat stack. Issues are organised by service so you can jump directly to the relevant section.
For IRC-specific troubleshooting with deeper detail, see IRC Troubleshooting.
Quick stack health check
Run these commands first to get a fast overview of the entire stack:
# Show all container statuses and health
just status
# Check health of every atl-* container in one pass
for c in atl-irc-server atl-irc-services atl-xmpp-server atl-xmpp-nginx atl-bridge atl-thelounge atl-irc-webpanel; do
status=$(docker inspect --format='{{.State.Health.Status}}' "$c" 2>/dev/null || echo "not found")
printf "%-22s %s\n" "$c" "$status"
done
# View recent logs across all services
docker compose logs --tail=20
unhealthy or not found, jump to that service’s section below.
General diagnostic commands
These commands apply to any service in the stack:
# View logs for a specific container (last 50 lines)
docker compose logs --tail=50 <container-name>
# Follow logs in real time
docker compose logs --follow <container-name>
# Filter logs by time range (last hour)
docker compose logs --since=1h <container-name>
# Search logs for errors
docker compose logs <container-name> 2>&1 | grep -i error
# Check container resource usage
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}" | grep atl-
# Inspect a container's full state
docker inspect <container-name> | jq '.[0].State'
# Check disk usage of data directories
du -sh data/*/
UnrealIRCd (IRC server)
Container: atl-irc-server · Ports: 6697 (TLS), 8000 (WebSocket), 8600 (RPC)
Diagnostic commands
# Container health and status
docker inspect --format='{{.State.Health.Status}}' atl-irc-server
# Recent logs
docker compose logs --tail=50 atl-irc-server
# Test TLS connectivity
echo "QUIT" | openssl s_client -connect localhost:6697 -servername irc.localhost 2>/dev/null | head -5
# Test WebSocket port
curl -s -o /dev/null -w "%{http_code}" http://localhost:8000
# Test JSON-RPC endpoint
curl -s http://localhost:8600/api/rpc.info | jq .
# Check if ports are listening on the host
ss -tlnp | grep -E "(6697|8000|8600)"
# Shell into the container
docker compose exec atl-irc-server bash
Common issues
The most common cause is a configuration syntax error.
# Check the exit code
docker compose ps -a | grep atl-irc-server
# View startup logs for the error message
docker compose logs atl-irc-server | head -40
error or fatal in the output. Fix the config template in apps/unrealircd/config/, then regenerate and restart:
scripts/prepare-config.sh
just dev
2. TLS certificate errors
Clients report certificate warnings or TLS handshake failures.
# Check certificate expiry
openssl x509 -in data/certs/live/irc.localhost/fullchain.pem -noout -dates 2>/dev/null || \
echo "Certificate file not found"
# Verify the certificate matches the expected domain
openssl x509 -in data/certs/live/irc.localhost/fullchain.pem -noout -subject 2>/dev/null
# Check cert files are readable inside the container
docker compose exec atl-irc-server ls -la /home/unrealircd/unrealircd/certs/live/
just init generates self-signed certificates. If certs are missing or expired, re-run just init. For production, see SSL/TLS.
3. Port already in use
Another process is binding to 6697, 8000, or 8600.
# Find what's using the port
ss -tlnp | grep 6697
.env:
# Override the port in .env
IRC_TLS_PORT=6698
just dev
4. Unsubstituted environment variables in config
The generated config contains literal ${VARIABLE} strings instead of values.
# Check for unsubstituted variables
grep -n '${' apps/unrealircd/config/unrealircd.conf | head -20
.env exists and contains all required variables, then regenerate:
scripts/prepare-config.sh
just irc reload
5. Cannot connect from external clients
The server is running but clients outside localhost cannot reach it.
# Check the bind address — 127.0.0.1 means localhost only
grep ATL_CHAT_IP .env
# Check firewall rules
sudo ufw status 2>/dev/null || sudo iptables -L -n 2>/dev/null | head -20
ATL_CHAT_IP=0.0.0.0 in .env and restart. Ensure your firewall allows the relevant ports.
6. ObsidianIRC / Firefox: “Can’t establish connection” to wss://127.0.0.1:8000
In dev, ObsidianIRC connects to the IRC WebSocket over TLS with a self-signed certificate. Firefox blocks WebSocket connections to untrusted certs without prompting. Add a certificate exception first: visit https://127.0.0.1:8000/ → Advanced → Accept the Risk and Continue, then reload ObsidianIRC at http://localhost:8090.
Atheme (IRC services)
Container: atl-irc-services · Shares network with atl-irc-server (port 8081 for HTTP API)
Diagnostic commands
# Container health
docker inspect --format='{{.State.Health.Status}}' atl-irc-services
# Recent logs
docker compose logs --tail=50 atl-irc-services
# Check Atheme process is running
docker compose exec atl-irc-server pgrep -f atheme-services
# Check Atheme HTTP API responds
curl -sf http://localhost:8081/ && echo "Atheme HTTP OK" || echo "Atheme HTTP not responding"
# Check Atheme database file
ls -la data/atheme/data/services.db
# Verify Atheme is linked to UnrealIRCd (from an IRC client as oper)
# /LINKS — should show services listed
Common issues
1. Atheme won’t start — waiting for UnrealIRCd
Atheme depends on atl-irc-server being healthy before it starts.
# Check if UnrealIRCd is healthy
docker inspect --format='{{.State.Health.Status}}' atl-irc-server
docker compose logs atl-irc-services | head -30
data/atheme/data/ exists and is writable. Re-run just init if directories are missing.
3. NickServ/ChanServ not responding
Atheme is running but service bots do not respond to commands.
# Check if Atheme is linked to UnrealIRCd
docker compose logs atl-irc-services 2>&1 | grep -i "link\|connect"
docker compose restart atl-irc-services
4. Database corruption
Atheme refuses to load services.db on startup.
docker compose logs atl-irc-services 2>&1 | grep -iE "database|corrupt|error"
.bak files in data/atheme/data/. Restore from the most recent backup:
docker compose stop atl-irc-services
cp data/atheme/data/services.db.bak data/atheme/data/services.db
docker compose start atl-irc-services
5. Permission denied on data directory
Resolution: fix ownership to match your host user:
sudo chown -R "$(id -u):$(id -g)" data/atheme/
Prosody (XMPP server)
Container: atl-xmpp-server · Ports: 5222 (C2S), 5280 (HTTP/BOSH), 5281 (HTTPS via nginx)
Diagnostic commands
# Container health
docker inspect --format='{{.State.Health.Status}}' atl-xmpp-server
# Recent logs
docker compose logs --tail=50 atl-xmpp-server
# HTTP status endpoint
curl -sf http://localhost:5280/status && echo " — Prosody OK"
# Test XMPP C2S port
nc -zv localhost 5222
# Test HTTPS proxy (nginx)
curl -sk https://localhost:5281/health
# Check nginx proxy health
docker inspect --format='{{.State.Health.Status}}' atl-xmpp-nginx
# Shell into the container
docker compose exec atl-xmpp-server bash
Common issues
1. Prosody fails to start
docker compose logs atl-xmpp-server | head -40
data/certs/ contains valid certificates for your XMPP domain. Re-run just init for dev certs. Check for config errors in apps/prosody/config/prosody.cfg.lua.
2. XMPP clients cannot authenticate
Users get “authentication failed” errors.
# Check auth-related log entries
docker compose logs atl-xmpp-server 2>&1 | grep -iE "auth.*fail|sasl|credential"
3. BOSH/WebSocket endpoint not responding
Web clients cannot connect via http://localhost:5280/http-bind.
curl -sf http://localhost:5280/http-bind && echo "BOSH OK" || echo "BOSH not responding"
bosh and websocket modules are enabled in the Prosody config. Check that port 5280 is mapped correctly in .env.
4. HTTPS proxy (nginx) returning 502
The nginx sidecar at port 5281 returns a bad gateway error.
docker compose logs --tail=30 atl-xmpp-nginx
atl-xmpp-server:5280. Ensure Prosody is running and healthy before checking nginx. Restart both if needed:
docker compose restart atl-xmpp-server atl-xmpp-nginx
5. Certificate mismatch for XMPP domain
Clients report the certificate does not match the expected domain.
# Check which domain the cert covers
openssl s_client -connect localhost:5222 -starttls xmpp </dev/null 2>/dev/null | \
openssl x509 -noout -subject -dates
XMPP_DOMAIN in .env matches the domain in your TLS certificate. For dev, re-run just init to regenerate certs matching the configured domain.
Bridge (Discord↔IRC↔XMPP)
Container: atl-bridge · No exposed ports (connects outbound to IRC, XMPP, and Discord)
Diagnostic commands
# Container health
docker inspect --format='{{.State.Health.Status}}' atl-bridge
# Recent logs
docker compose logs --tail=50 atl-bridge
# Check bridge process is running
docker exec atl-bridge pgrep -f "bridge.__main__" && echo "Bridge running"
# Search for connection errors
docker compose logs atl-bridge 2>&1 | grep -iE "error|disconnect|reconnect|refused"
# Check bridge config is mounted
docker exec atl-bridge cat /app/config.yaml | head -10
Common issues
1. Bridge won’t start — waiting for dependencies
The bridge depends on both atl-irc-server and atl-xmpp-server being healthy.
docker inspect --format='{{.State.Health.Status}}' atl-irc-server
docker inspect --format='{{.State.Health.Status}}' atl-xmpp-server
2. Discord connection fails
docker compose logs atl-bridge 2>&1 | grep -i discord
BRIDGE_DISCORD_TOKEN is set correctly in .env. The token must be a valid Discord bot token with the required intents enabled.
3. IRC connection refused
The bridge cannot connect to UnrealIRCd.
docker compose logs atl-bridge 2>&1 | grep -iE "irc.*refused|irc.*error|irc.*timeout"
BRIDGE_IRC_NICK and BRIDGE_IRC_OPER_PASSWORD are set in .env and match the UnrealIRCd oper block configuration.
4. XMPP component connection fails
docker compose logs atl-bridge 2>&1 | grep -iE "xmpp.*component|xmpp.*error"
.env match the Prosody component configuration:
BRIDGE_XMPP_COMPONENT_JIDBRIDGE_XMPP_COMPONENT_SECRETBRIDGE_XMPP_COMPONENT_SERVERBRIDGE_XMPP_COMPONENT_PORT
5. Messages not relaying between protocols
The bridge is running but messages do not appear on the other side.
# Check for relay errors
docker compose logs atl-bridge 2>&1 | grep -iE "relay|forward|send.*fail"
# Verify config.yaml channel mappings
docker exec atl-bridge cat /app/config.yaml
apps/bridge/config.yaml are correct. Ensure the bridge bot has the necessary permissions in each channel (IRC oper, Discord channel access, XMPP MUC membership).
The Lounge (web IRC client)
Container: atl-thelounge · Port: 9000
Diagnostic commands
# Check container is running (no built-in health check)
docker compose ps atl-thelounge
# Recent logs
docker compose logs --tail=50 atl-thelounge
# Test HTTP connectivity
curl -sf http://localhost:9000/ -o /dev/null && echo "The Lounge OK" || echo "The Lounge not responding"
# List registered users
just lounge list
# Check data directory
ls -la data/thelounge/
Common issues
1. Cannot access the web interface
Browser shows connection refused at http://localhost:9000.
docker compose ps atl-thelounge
docker compose logs --tail=20 atl-thelounge
data/thelounge/ directory does not exist or has wrong permissions.
# Ensure directory exists with correct ownership
mkdir -p data/thelounge
sudo chown -R "$(id -u):$(id -g)" data/thelounge/
docker compose restart atl-thelounge
2. Login page appears but no users exist
The Lounge runs in private mode — you must create a user account first.
Follow the prompts to set a username and password.
3. Cannot connect to IRC server from The Lounge
You can log in but The Lounge cannot reach UnrealIRCd.
# Check if UnrealIRCd is healthy
docker inspect --format='{{.State.Health.Status}}' atl-irc-server
# Check The Lounge logs for connection errors
docker compose logs atl-thelounge 2>&1 | grep -iE "error|refused|timeout"
atl-irc-server (the container hostname).
4. Plugin installation fails
The janitor or giphy plugins fail to install on startup.
docker compose logs atl-thelounge 2>&1 | grep -i plugin
data/thelounge/ volume. Failures are non-fatal — The Lounge still starts without the plugins.
5. Chat history missing after restart
The Lounge stores history in data/thelounge/. If the volume mount is misconfigured, data is lost on restart.
# Verify the volume mount
docker inspect atl-thelounge | jq '.[0].Mounts'
data/thelounge/ is bind-mounted to /var/opt/thelounge in the container. Check that the directory was not accidentally deleted. See Backups for restore procedures.
WebPanel (UnrealIRCd admin)
Container: atl-irc-webpanel · Port: 8080
Diagnostic commands
# Check container is running (no built-in health check)
docker compose ps atl-irc-webpanel
# Recent logs
docker compose logs --tail=30 atl-irc-webpanel
# Test HTTP connectivity
curl -sf http://localhost:8080/ -o /dev/null && echo "WebPanel OK" || echo "WebPanel not responding"
# Check if the RPC connection to UnrealIRCd works
docker compose exec atl-irc-server ls -la /home/unrealircd/unrealircd/data/rpc.socket
Common issues
1. WebPanel returns 502 or blank page
docker compose logs --tail=30 atl-irc-webpanel
docker inspect --format='{{.State.Health.Status}}' atl-irc-server
docker compose restart atl-irc-webpanel
2. Cannot log in to WebPanel
Authentication fails even with correct credentials.
Resolution: WebPanel authenticates against UnrealIRCd’s RPC interface. Verify the RPC credentials in .env match the UnrealIRCd configuration. Check that the RPC user block is configured in apps/unrealircd/config/.
3. WebPanel shows stale data
The panel displays outdated server information.
Resolution: WebPanel queries UnrealIRCd’s RPC in real time. If data appears stale, the RPC connection may have dropped. Restart the WebPanel:
docker compose restart atl-irc-webpanel
4. Port conflict on 8080
Another service (commonly a local web server or proxy) is using port 8080.
Resolution: change the WebPanel port in .env:
Then restart: just dev
5. WebPanel data directory permission errors
ls -la data/irc/webpanel-data/
sudo chown -R "$(id -u):$(id -g)" data/irc/webpanel-data/
docker compose restart atl-irc-webpanel
Cross-service issues
These problems affect multiple services or the stack as a whole.
Docker Compose fails to start
# Validate compose configuration
docker compose config --quiet && echo "Config OK" || echo "Config has errors"
# Check for missing .env file
test -f .env && echo ".env exists" || echo ".env missing — copy from .env.example"
.env exists. The CLOUDFLARE_DNS_API_TOKEN warning is safe to ignore in dev.
All services unhealthy after host reboot
After a host reboot, Docker may not be running.
# Check if Docker daemon is running
docker info >/dev/null 2>&1 && echo "Docker OK" || echo "Docker not running"
sudo dockerd &>/tmp/dockerd.log &
just dev
Network connectivity between containers
Services cannot reach each other over the Docker network.
# Check the atl-chat network exists
docker network ls | grep atl-chat
# Inspect which containers are on the network
docker network inspect atl-chat | jq '.[0].Containers | to_entries[] | {name: .value.Name, ip: .value.IPv4Address}'
Data directory missing or empty
Services fail because data/ subdirectories do not exist.
Resolution: re-run the init script to create all required directories:
Config regeneration after .env changes
After editing .env, services still use old values.
Resolution: regenerate configs from templates and restart:
scripts/prepare-config.sh
just down
just dev
Recovery quick reference
| Scenario | Command |
|---|
| Restart a single service | docker compose restart <container-name> |
| Restart the full stack | just down && just dev |
| Regenerate configs | scripts/prepare-config.sh |
| Recreate data directories | just init |
| Rebuild all containers | just build |
| Reset to clean state | just down && rm -rf data/ && just init && just dev |
| View real-time logs | docker compose logs --follow <container-name> |
| Check all container health | just status |
Warning: Resetting to clean state (rm -rf data/) destroys all persistent data including user registrations, chat history, and certificates. Back up first — see Backups.
Related pages