79 lines
3.4 KiB
Markdown
79 lines
3.4 KiB
Markdown
# Backups - Technology Context
|
|
|
|
## Overview
|
|
Automated backup system for nobara-desktop using restic with systemd timers, backing up to TrueNAS NAS over local network.
|
|
|
|
## Restic Backup (nobara-desktop)
|
|
|
|
### Configuration
|
|
| Setting | Value |
|
|
|---------|-------|
|
|
| Tool | restic 0.18.0 |
|
|
| Source | /home (~181 GiB) |
|
|
| Destination | /mnt/truenas/cals-files/Backups/nobara-desktop/restic-repo |
|
|
| Schedule | Daily at 3:00 AM (systemd timer, +-30min randomized) |
|
|
| Encryption | None (trusted LAN) |
|
|
| Retention | 7 daily, 4 weekly, 6 monthly, 1 yearly |
|
|
| Alerting | Discord webhook (Homelab Alerts channel) on failure |
|
|
|
|
### Key Files
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| /home/cal/.local/bin/restic-backup.sh | Backup script |
|
|
| /home/cal/.config/restic/excludes.txt | Exclusion patterns |
|
|
| /etc/systemd/system/restic-backup.service | systemd service unit |
|
|
| /etc/systemd/system/restic-backup.timer | systemd timer unit |
|
|
|
|
### Features
|
|
- **Stale lock cleanup**: Runs `restic unlock --remove-all` before each backup to prevent lock accumulation
|
|
- **Discord failure alerts**: Red embed sent to Homelab Alerts on any failure (mount, repo access, backup, retention)
|
|
- **ERR trap**: Catches unexpected script exits and alerts
|
|
- **Cache cleanup**: `--cleanup-cache` flag prevents cache bloat
|
|
- **Low system impact**: Nice=19, IOSchedulingClass=idle
|
|
- **Missed backup recovery**: Persistent=true catches backups missed while system was off
|
|
- **Mount dependency**: RequiresMountsFor ensures TrueNAS is mounted
|
|
|
|
### Quick Commands
|
|
```bash
|
|
# Check timer status
|
|
systemctl list-timers restic-backup.timer
|
|
|
|
# View recent logs
|
|
journalctl -u restic-backup.service -e
|
|
|
|
# List snapshots
|
|
restic -r /mnt/truenas/cals-files/Backups/nobara-desktop/restic-repo --insecure-no-password snapshots
|
|
|
|
# Check repo size
|
|
restic -r /mnt/truenas/cals-files/Backups/nobara-desktop/restic-repo --insecure-no-password stats --mode raw-data
|
|
|
|
# Manual backup
|
|
sudo systemctl start restic-backup.service
|
|
|
|
# Remove stale locks (should not be needed - script handles this automatically)
|
|
restic -r /mnt/truenas/cals-files/Backups/nobara-desktop/restic-repo --insecure-no-password unlock
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Backup service failing with exit code 11
|
|
**Cause**: Stale lock in the restic repository preventing `forget --prune` from running.
|
|
**Fix**: `restic -r <repo> --insecure-no-password unlock --remove-all`
|
|
**Prevention**: The backup script now clears stale locks automatically before each run (added 2026-02-19).
|
|
|
|
### Snapshots accumulating beyond retention policy
|
|
**Cause**: If `forget --prune` fails (e.g., stale lock), snapshots pile up but deduplication keeps disk usage manageable.
|
|
**Detection**: Check snapshot count with `restic snapshots` - should be ~18 max with current retention.
|
|
**Impact**: Disk usage grows slowly due to dedup, but metadata overhead increases.
|
|
|
|
### TrueNAS mount not available
|
|
**Cause**: NAS offline, network issue, or mount dropped.
|
|
**Detection**: Script checks `mountpoint -q /mnt/truenas/cals-files` and alerts on failure.
|
|
**Fix**: Check NAS status, remount with appropriate fstab entry.
|
|
|
|
## Architecture Notes
|
|
- Restic was chosen over borg (simpler CLI, good dedup) and rsync (no dedup)
|
|
- systemd timers chosen over cron for Persistent=true, RequiresMountsFor, and Nice/IOScheduling support
|
|
- No encryption because backups stay on trusted local TrueNAS - acceptable tradeoff for this environment
|
|
- Deduplication ratio is excellent (~65:1 across snapshots of similar data)
|