4b7eca8a46
All checks were successful
Reindex Knowledge Base / reindex (push) Successful in 3s
Adds title, description, type, domain, and tags frontmatter to every doc for improved KB semantic search. The description field is prepended to every search chunk, and domain/type/tags enable filtered queries. Type values: context, guide, runbook, reference, troubleshooting Domain values match directory structure (networking, docker, etc.) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
87 lines
3.7 KiB
Markdown
87 lines
3.7 KiB
Markdown
---
|
|
title: "Backups Overview"
|
|
description: "Restic backup system for nobara-desktop with systemd timers, backing up /home to TrueNAS NAS. Covers configuration, retention policy, Discord failure alerts, and troubleshooting stale locks."
|
|
type: context
|
|
domain: backups
|
|
tags: [restic, backup, systemd, truenas, discord, retention, nobara-desktop]
|
|
---
|
|
|
|
# Backups - Technology Context
|
|
|
|
## Overview
|
|
Automated backup system for nobara-desktop using restic with systemd timers, backing up to TrueNAS NAS over local network.
|
|
|
|
## Restic Backup (nobara-desktop)
|
|
|
|
### Configuration
|
|
| Setting | Value |
|
|
|---------|-------|
|
|
| Tool | restic 0.18.0 |
|
|
| Source | /home (~181 GiB) |
|
|
| Destination | /mnt/truenas/cals-files/Backups/nobara-desktop/restic-repo |
|
|
| Schedule | Daily at 3:00 AM (systemd timer, +-30min randomized) |
|
|
| Encryption | None (trusted LAN) |
|
|
| Retention | 7 daily, 4 weekly, 6 monthly, 1 yearly |
|
|
| Alerting | Discord webhook (Homelab Alerts channel) on failure |
|
|
|
|
### Key Files
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| /home/cal/.local/bin/restic-backup.sh | Backup script |
|
|
| /home/cal/.config/restic/excludes.txt | Exclusion patterns |
|
|
| /etc/systemd/system/restic-backup.service | systemd service unit |
|
|
| /etc/systemd/system/restic-backup.timer | systemd timer unit |
|
|
|
|
### Features
|
|
- **Stale lock cleanup**: Runs `restic unlock --remove-all` before each backup to prevent lock accumulation
|
|
- **Discord failure alerts**: Red embed sent to Homelab Alerts on any failure (mount, repo access, backup, retention)
|
|
- **ERR trap**: Catches unexpected script exits and alerts
|
|
- **Cache cleanup**: `--cleanup-cache` flag prevents cache bloat
|
|
- **Low system impact**: Nice=19, IOSchedulingClass=idle
|
|
- **Missed backup recovery**: Persistent=true catches backups missed while system was off
|
|
- **Mount dependency**: RequiresMountsFor ensures TrueNAS is mounted
|
|
|
|
### Quick Commands
|
|
```bash
|
|
# Check timer status
|
|
systemctl list-timers restic-backup.timer
|
|
|
|
# View recent logs
|
|
journalctl -u restic-backup.service -e
|
|
|
|
# List snapshots
|
|
restic -r /mnt/truenas/cals-files/Backups/nobara-desktop/restic-repo --insecure-no-password snapshots
|
|
|
|
# Check repo size
|
|
restic -r /mnt/truenas/cals-files/Backups/nobara-desktop/restic-repo --insecure-no-password stats --mode raw-data
|
|
|
|
# Manual backup
|
|
sudo systemctl start restic-backup.service
|
|
|
|
# Remove stale locks (should not be needed - script handles this automatically)
|
|
restic -r /mnt/truenas/cals-files/Backups/nobara-desktop/restic-repo --insecure-no-password unlock
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Backup service failing with exit code 11
|
|
**Cause**: Stale lock in the restic repository preventing `forget --prune` from running.
|
|
**Fix**: `restic -r <repo> --insecure-no-password unlock --remove-all`
|
|
**Prevention**: The backup script now clears stale locks automatically before each run (added 2026-02-19).
|
|
|
|
### Snapshots accumulating beyond retention policy
|
|
**Cause**: If `forget --prune` fails (e.g., stale lock), snapshots pile up but deduplication keeps disk usage manageable.
|
|
**Detection**: Check snapshot count with `restic snapshots` - should be ~18 max with current retention.
|
|
**Impact**: Disk usage grows slowly due to dedup, but metadata overhead increases.
|
|
|
|
### TrueNAS mount not available
|
|
**Cause**: NAS offline, network issue, or mount dropped.
|
|
**Detection**: Script checks `mountpoint -q /mnt/truenas/cals-files` and alerts on failure.
|
|
**Fix**: Check NAS status, remount with appropriate fstab entry.
|
|
|
|
## Architecture Notes
|
|
- Restic was chosen over borg (simpler CLI, good dedup) and rsync (no dedup)
|
|
- systemd timers chosen over cron for Persistent=true, RequiresMountsFor, and Nice/IOScheduling support
|
|
- No encryption because backups stay on trusted local TrueNAS - acceptable tradeoff for this environment
|
|
- Deduplication ratio is excellent (~65:1 across snapshots of similar data)
|