Auto-merge docs-only PRs / auto-merge-docs (pull_request) Successful in 2s

Details

chore: decommission VM 105 (docker-vpn) — repo cleanup (#20 )

VM 105 was already destroyed on Proxmox. This removes stale references:
- Delete server-configs/proxmox/qemu/105.conf
- Comment out docker-vpn entries in example SSH config and server inventory
- Move VM 105 from Stopped/Investigate to Removed in upgrade plan
- Check off decommission task in wave2 migration results

Closes #20

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-02 23:57:55 -05:00

9.9 KiB

Raw Blame History

title

description

type

domain

SSH Home Lab Setup - Complete Implementation

Overview

Complete working implementation for setting up secure SSH key-based authentication across a home lab environment with emergency backup keys and NAS storage.

Server Inventory

Home Network (10.10.0.0/24)

servers:
  database_apis:      10.10.0.42  # Database and API services
  discord_bots:       10.10.0.33  # Discord bot hosting
  home_docker:        10.10.0.124 # Main Docker container host
  pihole:             10.10.0.16  # Pi-hole DNS and ad blocking
  sba_pd_bots:        10.10.0.88  # SBa and PD bot services
  tdarr:              10.10.0.43  # Media transcoding
  # vpn_docker:       10.10.0.121 # DECOMMISSIONED — VM 105 destroyed, migrated to arr-stack LXC 221

Cloud Servers

remote_servers:
  akamai_nano:        172.237.147.99  # Akamai cloud instance
  vultr_host:         45.76.25.231    # Vultr cloud host

SSH Key Generation Script

#!/bin/bash
# SSH Key Setup Script for Home Lab
# Creates primary + emergency keys with NAS backup

set -e

echo "🔐 Setting up SSH keys for Home Lab with Emergency Backup..."

# Create directories
mkdir -p ~/.ssh
chmod 700 ~/.ssh

# Create NAS backup directory
BACKUP_DIR="/mnt/NV2/ssh-keys/backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"
chmod 700 "$BACKUP_DIR"
echo "📁 NAS backup directory created: $BACKUP_DIR"

# Generate primary keys
if [ ! -f ~/.ssh/homelab_rsa ]; then
    ssh-keygen -t rsa -b 4096 -C "homelab-$(whoami)@$(hostname)" -f ~/.ssh/homelab_rsa -N ""
    chmod 600 ~/.ssh/homelab_rsa
    chmod 644 ~/.ssh/homelab_rsa.pub
    echo "✅ Home lab SSH key generated"
fi

if [ ! -f ~/.ssh/cloud_servers_rsa ]; then
    ssh-keygen -t rsa -b 4096 -C "cloud-$(whoami)@$(hostname)" -f ~/.ssh/cloud_servers_rsa -N ""
    chmod 600 ~/.ssh/cloud_servers_rsa
    chmod 644 ~/.ssh/cloud_servers_rsa.pub
    echo "✅ Cloud servers SSH key generated"
fi

# Generate emergency keys
if [ ! -f ~/.ssh/emergency_homelab_rsa ]; then
    ssh-keygen -t rsa -b 4096 -C "emergency-homelab-$(whoami)@$(hostname)" -f ~/.ssh/emergency_homelab_rsa -N ""
    chmod 600 ~/.ssh/emergency_homelab_rsa
    chmod 644 ~/.ssh/emergency_homelab_rsa.pub
    echo "✅ Emergency home lab key generated"
fi

if [ ! -f ~/.ssh/emergency_cloud_rsa ]; then
    ssh-keygen -t rsa -b 4096 -C "emergency-cloud-$(whoami)@$(hostname)" -f ~/.ssh/emergency_cloud_rsa -N ""
    chmod 600 ~/.ssh/emergency_cloud_rsa
    chmod 644 ~/.ssh/emergency_cloud_rsa.pub
    echo "✅ Emergency cloud key generated"
fi

# Backup all keys to NAS
echo "💾 Backing up keys to NAS..."
cp ~/.ssh/*_rsa* "$BACKUP_DIR/" 2>/dev/null || true
cp ~/.ssh/config "$BACKUP_DIR/" 2>/dev/null || true

# Create recovery documentation
cat > "$BACKUP_DIR/RECOVERY_INSTRUCTIONS.md" << EOF
# SSH Key Recovery Instructions
Generated: $(date)
Location: $BACKUP_DIR

## Emergency Recovery Steps

### If you lose access to your primary keys:

1. **Copy emergency keys back:**
   \`\`\`bash
   cp $BACKUP_DIR/emergency_*_rsa* ~/.ssh/
   chmod 600 ~/.ssh/emergency_*_rsa
   chmod 644 ~/.ssh/emergency_*_rsa.pub
   \`\`\`

2. **Test emergency access:**
   \`\`\`bash
   ssh -i ~/.ssh/emergency_homelab_rsa cal@10.10.0.16
   ssh -i ~/.ssh/emergency_cloud_rsa root@172.237.147.99
   \`\`\`

3. **If emergency keys weren't deployed, use console access:**
   - Home servers: Physical console access
   - Cloud servers: Provider web console
   - Add emergency public key to ~/.ssh/authorized_keys
EOF

ls -la "$BACKUP_DIR" >> "$BACKUP_DIR/RECOVERY_INSTRUCTIONS.md"
echo "✅ Keys backed up to: $BACKUP_DIR"

echo "🎉 SSH keys and emergency backups created successfully!"

SSH Configuration

# SSH Configuration for Home Lab (~/.ssh/config)
# Global defaults
Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3
    StrictHostKeyChecking ask
    VerifyHostKeyDNS yes
    ForwardAgent no
    ForwardX11 no
    AddKeysToAgent yes

# Home Network Servers
Host strat-database pd-database sba-database strat-db pd-db sba-db
    HostName 10.10.0.42
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host discord-bots
    HostName 10.10.0.33
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host docker-home docker-main
    HostName 10.10.0.124
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host pihole dns
    HostName 10.10.0.16
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host sba-bots pd-bots
    HostName 10.10.0.88
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host tdarr media
    HostName 10.10.0.43
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

# DECOMMISSIONED: docker-vpn (10.10.0.121) - VM 105 destroyed, migrated to arr-stack LXC 221
# Host docker-vpn
#     HostName 10.10.0.121
#     User cal
#     Port 22
#     IdentityFile ~/.ssh/homelab_rsa

# Remote Cloud Servers
Host akamai-nano akamai
    HostName 172.237.147.99
    User root
    Port 22
    IdentityFile ~/.ssh/cloud_servers_rsa

Host vultr-host vultr
    HostName 45.76.25.231
    User root
    Port 22
    IdentityFile ~/.ssh/cloud_servers_rsa

# Home network wildcard for easy access
Host 10.10.0.*
    User cal
    IdentityFile ~/.ssh/homelab_rsa
    StrictHostKeyChecking accept-new

Key Deployment Commands

Primary Keys

# Home lab servers
for ip in 42 33 124 16 88 43 121; do
    ssh-copy-id -i ~/.ssh/homelab_rsa.pub cal@10.10.0.$ip
done

# Cloud servers
ssh-copy-id -i ~/.ssh/cloud_servers_rsa.pub root@172.237.147.99
ssh-copy-id -i ~/.ssh/cloud_servers_rsa.pub root@45.76.25.231

Emergency Keys

# Home lab servers (use -f to force)
for ip in 42 33 124 16 88 43 121; do
    ssh-copy-id -f -i ~/.ssh/emergency_homelab_rsa.pub cal@10.10.0.$ip
done

# Cloud servers
ssh-copy-id -f -i ~/.ssh/emergency_cloud_rsa.pub root@172.237.147.99
ssh-copy-id -f -i ~/.ssh/emergency_cloud_rsa.pub root@45.76.25.231

Maintenance Script

#!/bin/bash
# SSH Key Maintenance and Backup Script
# Location: ~/bin/ssh_key_maintenance.sh
# Schedule: 0 2 1 * * (monthly at 2 AM)

echo "🔧 SSH Key Maintenance and Backup"

# Check NAS availability
if [ ! -d "/mnt/NV2" ]; then
    echo "❌ ERROR: NAS not mounted at /mnt/NV2"
    exit 1
fi

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_ROOT="/mnt/NV2/ssh-keys"
BACKUP_DIR="$BACKUP_ROOT/maintenance-$TIMESTAMP"

mkdir -p "$BACKUP_DIR"
chmod 700 "$BACKUP_DIR"

# Backup current state
cp ~/.ssh/*_rsa* "$BACKUP_DIR/" 2>/dev/null || true
cp ~/.ssh/config "$BACKUP_DIR/" 2>/dev/null || true
cp ~/.ssh/known_hosts "$BACKUP_DIR/" 2>/dev/null || true

# Check key ages and health
echo "🔍 Key Age Analysis:"
for key in ~/.ssh/*_rsa; do
    if [ -f "$key" ]; then
        age_days=$(( ($(date +%s) - $(stat -c %Y "$key")) / 86400 ))
        basename_key=$(basename "$key")
        
        if [ $age_days -gt 365 ]; then
            echo "⚠️  $basename_key: $age_days days old - ROTATION RECOMMENDED"
        elif [ $age_days -gt 180 ]; then
            echo "⚡ $basename_key: $age_days days old - consider rotation"
        else
            echo "✅ $basename_key: $age_days days old - OK"
        fi
    fi
done

# Clean up old backups (keep last 10)
cd "$BACKUP_ROOT"
ls -dt backup-* maintenance-* 2>/dev/null | tail -n +11 | while read old_backup; do
    if [ -d "$old_backup" ]; then
        echo "🗑️  Removing old backup: $old_backup"
        rm -rf "$old_backup"
    fi
done

echo "✅ Maintenance backup completed: $BACKUP_DIR"

Testing Procedures

Test Primary Access

# Test aliases work
ssh strat-database 'echo "Primary access: $(hostname)"'
ssh pihole 'echo "Primary access: $(hostname)"'
ssh akamai 'echo "Primary access: $(hostname)"'

Test Emergency Access

# Test emergency keys work
ssh -i ~/.ssh/emergency_homelab_rsa cal@10.10.0.16 'echo "Emergency access: $(hostname)"'
ssh -i ~/.ssh/emergency_cloud_rsa root@172.237.147.99 'echo "Emergency access: $(hostname)"'

Cron Schedule Setup

# Open crontab editor
crontab -e

# Add monthly maintenance (1st of month at 2 AM)
0 2 1 * * /home/cal/bin/ssh_key_maintenance.sh

Security Hardening

After key deployment, disable password authentication on servers:

# On each server, edit /etc/ssh/sshd_config:
PasswordAuthentication no
PubkeyAuthentication yes
PermitRootLogin no  # Create non-root users first on cloud servers
AllowUsers cal      # Restrict SSH access

# Restart SSH service
sudo systemctl restart sshd

Mobile Device SSH Access

Termius Setup (iPhone/iPad)

Connect to local development machine using existing homelab keys:

Connection Details:
  Host: 10.0.0.206 (or current eno1 IP)
  Username: cal
  Port: 22
  Key: homelab_rsa (same key used for server access)

Prerequisites Checklist

Before mobile SSH access works, ensure:

SSH service is running:

sudo systemctl start sshd
sudo systemctl enable sshd

Firewall allows SSH:

sudo firewall-cmd --list-services  # Should include 'ssh'
# If not present, SSH was already allowed in this setup

Authorized keys configured:

# homelab_rsa.pub should be in ~/.ssh/authorized_keys
cat ~/.ssh/homelab_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Key Transfer Process

Copy private key: cp ~/.ssh/homelab_rsa ./homelab_rsa_for_mobile
Transfer securely to mobile device (AirDrop, secure file share)
Import into Termius app
Clean up: rm homelab_rsa_for_mobile

Network Discovery

Find local machine IP: ip addr show eno1 | grep 'inet ' | awk '{print $2}' | cut -d'/' -f1

Patterns: patterns/networking/ssh-key-management.md
Troubleshooting: reference/networking/ssh-troubleshooting.md

9.9 KiB Raw Blame History