# SSH Home Lab Setup - Complete Implementation

## Overview

Complete working implementation for setting up secure SSH key-based authentication across a home lab environment with emergency backup keys and NAS storage.

## Server Inventory

### Home Network (10.10.0.0/24)
```yaml
servers:
  database_apis:      10.10.0.42  # Database and API services
  discord_bots:       10.10.0.33  # Discord bot hosting
  home_docker:        10.10.0.124 # Main Docker container host
  pihole:             10.10.0.16  # Pi-hole DNS and ad blocking
  sba_pd_bots:        10.10.0.88  # SBa and PD bot services
  tdarr:              10.10.0.43  # Media transcoding
  vpn_docker:         10.10.0.121 # VPN and Docker services
```

### Cloud Servers
```yaml
remote_servers:
  akamai_nano:        172.237.147.99  # Akamai cloud instance
  vultr_host:         45.76.25.231    # Vultr cloud host
```

## SSH Key Generation Script

```bash
#!/bin/bash
# SSH Key Setup Script for Home Lab
# Creates primary + emergency keys with NAS backup

set -e

echo "🔐 Setting up SSH keys for Home Lab with Emergency Backup..."

# Create directories
mkdir -p ~/.ssh
chmod 700 ~/.ssh

# Create NAS backup directory
BACKUP_DIR="/mnt/NV2/ssh-keys/backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"
chmod 700 "$BACKUP_DIR"
echo "📁 NAS backup directory created: $BACKUP_DIR"

# Generate primary keys
if [ ! -f ~/.ssh/homelab_rsa ]; then
    ssh-keygen -t rsa -b 4096 -C "homelab-$(whoami)@$(hostname)" -f ~/.ssh/homelab_rsa -N ""
    chmod 600 ~/.ssh/homelab_rsa
    chmod 644 ~/.ssh/homelab_rsa.pub
    echo "✅ Home lab SSH key generated"
fi

if [ ! -f ~/.ssh/cloud_servers_rsa ]; then
    ssh-keygen -t rsa -b 4096 -C "cloud-$(whoami)@$(hostname)" -f ~/.ssh/cloud_servers_rsa -N ""
    chmod 600 ~/.ssh/cloud_servers_rsa
    chmod 644 ~/.ssh/cloud_servers_rsa.pub
    echo "✅ Cloud servers SSH key generated"
fi

# Generate emergency keys
if [ ! -f ~/.ssh/emergency_homelab_rsa ]; then
    ssh-keygen -t rsa -b 4096 -C "emergency-homelab-$(whoami)@$(hostname)" -f ~/.ssh/emergency_homelab_rsa -N ""
    chmod 600 ~/.ssh/emergency_homelab_rsa
    chmod 644 ~/.ssh/emergency_homelab_rsa.pub
    echo "✅ Emergency home lab key generated"
fi

if [ ! -f ~/.ssh/emergency_cloud_rsa ]; then
    ssh-keygen -t rsa -b 4096 -C "emergency-cloud-$(whoami)@$(hostname)" -f ~/.ssh/emergency_cloud_rsa -N ""
    chmod 600 ~/.ssh/emergency_cloud_rsa
    chmod 644 ~/.ssh/emergency_cloud_rsa.pub
    echo "✅ Emergency cloud key generated"
fi

# Backup all keys to NAS
echo "💾 Backing up keys to NAS..."
cp ~/.ssh/*_rsa* "$BACKUP_DIR/" 2>/dev/null || true
cp ~/.ssh/config "$BACKUP_DIR/" 2>/dev/null || true

# Create recovery documentation
cat > "$BACKUP_DIR/RECOVERY_INSTRUCTIONS.md" << EOF
# SSH Key Recovery Instructions
Generated: $(date)
Location: $BACKUP_DIR

## Emergency Recovery Steps

### If you lose access to your primary keys:

1. **Copy emergency keys back:**
   \`\`\`bash
   cp $BACKUP_DIR/emergency_*_rsa* ~/.ssh/
   chmod 600 ~/.ssh/emergency_*_rsa
   chmod 644 ~/.ssh/emergency_*_rsa.pub
   \`\`\`

2. **Test emergency access:**
   \`\`\`bash
   ssh -i ~/.ssh/emergency_homelab_rsa cal@10.10.0.16
   ssh -i ~/.ssh/emergency_cloud_rsa root@172.237.147.99
   \`\`\`

3. **If emergency keys weren't deployed, use console access:**
   - Home servers: Physical console access
   - Cloud servers: Provider web console
   - Add emergency public key to ~/.ssh/authorized_keys
EOF

ls -la "$BACKUP_DIR" >> "$BACKUP_DIR/RECOVERY_INSTRUCTIONS.md"
echo "✅ Keys backed up to: $BACKUP_DIR"

echo "🎉 SSH keys and emergency backups created successfully!"
```

## SSH Configuration

```bash
# SSH Configuration for Home Lab (~/.ssh/config)
# Global defaults
Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3
    StrictHostKeyChecking ask
    VerifyHostKeyDNS yes
    ForwardAgent no
    ForwardX11 no
    AddKeysToAgent yes

# Home Network Servers
Host strat-database pd-database sba-database strat-db pd-db sba-db
    HostName 10.10.0.42
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host discord-bots
    HostName 10.10.0.33
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host docker-home docker-main
    HostName 10.10.0.124
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host pihole dns
    HostName 10.10.0.16
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host sba-bots pd-bots
    HostName 10.10.0.88
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host tdarr media
    HostName 10.10.0.43
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

Host docker-vpn
    HostName 10.10.0.121
    User cal
    Port 22
    IdentityFile ~/.ssh/homelab_rsa

# Remote Cloud Servers
Host akamai-nano akamai
    HostName 172.237.147.99
    User root
    Port 22
    IdentityFile ~/.ssh/cloud_servers_rsa

Host vultr-host vultr
    HostName 45.76.25.231
    User root
    Port 22
    IdentityFile ~/.ssh/cloud_servers_rsa

# Home network wildcard for easy access
Host 10.10.0.*
    User cal
    IdentityFile ~/.ssh/homelab_rsa
    StrictHostKeyChecking accept-new
```

## Key Deployment Commands

### Primary Keys
```bash
# Home lab servers
for ip in 42 33 124 16 88 43 121; do
    ssh-copy-id -i ~/.ssh/homelab_rsa.pub cal@10.10.0.$ip
done

# Cloud servers
ssh-copy-id -i ~/.ssh/cloud_servers_rsa.pub root@172.237.147.99
ssh-copy-id -i ~/.ssh/cloud_servers_rsa.pub root@45.76.25.231
```

### Emergency Keys
```bash
# Home lab servers (use -f to force)
for ip in 42 33 124 16 88 43 121; do
    ssh-copy-id -f -i ~/.ssh/emergency_homelab_rsa.pub cal@10.10.0.$ip
done

# Cloud servers
ssh-copy-id -f -i ~/.ssh/emergency_cloud_rsa.pub root@172.237.147.99
ssh-copy-id -f -i ~/.ssh/emergency_cloud_rsa.pub root@45.76.25.231
```

## Maintenance Script

```bash
#!/bin/bash
# SSH Key Maintenance and Backup Script
# Location: ~/bin/ssh_key_maintenance.sh
# Schedule: 0 2 1 * * (monthly at 2 AM)

echo "🔧 SSH Key Maintenance and Backup"

# Check NAS availability
if [ ! -d "/mnt/NV2" ]; then
    echo "❌ ERROR: NAS not mounted at /mnt/NV2"
    exit 1
fi

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_ROOT="/mnt/NV2/ssh-keys"
BACKUP_DIR="$BACKUP_ROOT/maintenance-$TIMESTAMP"

mkdir -p "$BACKUP_DIR"
chmod 700 "$BACKUP_DIR"

# Backup current state
cp ~/.ssh/*_rsa* "$BACKUP_DIR/" 2>/dev/null || true
cp ~/.ssh/config "$BACKUP_DIR/" 2>/dev/null || true
cp ~/.ssh/known_hosts "$BACKUP_DIR/" 2>/dev/null || true

# Check key ages and health
echo "🔍 Key Age Analysis:"
for key in ~/.ssh/*_rsa; do
    if [ -f "$key" ]; then
        age_days=$(( ($(date +%s) - $(stat -c %Y "$key")) / 86400 ))
        basename_key=$(basename "$key")
        
        if [ $age_days -gt 365 ]; then
            echo "⚠️  $basename_key: $age_days days old - ROTATION RECOMMENDED"
        elif [ $age_days -gt 180 ]; then
            echo "⚡ $basename_key: $age_days days old - consider rotation"
        else
            echo "✅ $basename_key: $age_days days old - OK"
        fi
    fi
done

# Clean up old backups (keep last 10)
cd "$BACKUP_ROOT"
ls -dt backup-* maintenance-* 2>/dev/null | tail -n +11 | while read old_backup; do
    if [ -d "$old_backup" ]; then
        echo "🗑️  Removing old backup: $old_backup"
        rm -rf "$old_backup"
    fi
done

echo "✅ Maintenance backup completed: $BACKUP_DIR"
```

## Testing Procedures

### Test Primary Access
```bash
# Test aliases work
ssh strat-database 'echo "Primary access: $(hostname)"'
ssh pihole 'echo "Primary access: $(hostname)"'
ssh akamai 'echo "Primary access: $(hostname)"'
```

### Test Emergency Access
```bash
# Test emergency keys work
ssh -i ~/.ssh/emergency_homelab_rsa cal@10.10.0.16 'echo "Emergency access: $(hostname)"'
ssh -i ~/.ssh/emergency_cloud_rsa root@172.237.147.99 'echo "Emergency access: $(hostname)"'
```

## Cron Schedule Setup

```bash
# Open crontab editor
crontab -e

# Add monthly maintenance (1st of month at 2 AM)
0 2 1 * * /home/cal/bin/ssh_key_maintenance.sh
```

## Security Hardening

After key deployment, disable password authentication on servers:

```bash
# On each server, edit /etc/ssh/sshd_config:
PasswordAuthentication no
PubkeyAuthentication yes
PermitRootLogin no  # Create non-root users first on cloud servers
AllowUsers cal      # Restrict SSH access

# Restart SSH service
sudo systemctl restart sshd
```

## Mobile Device SSH Access

### Termius Setup (iPhone/iPad)
Connect to local development machine using existing homelab keys:

```yaml
Connection Details:
  Host: 10.0.0.206 (or current eno1 IP)
  Username: cal
  Port: 22
  Key: homelab_rsa (same key used for server access)
```

### Prerequisites Checklist
Before mobile SSH access works, ensure:

1. **SSH service is running:**
   ```bash
   sudo systemctl start sshd
   sudo systemctl enable sshd
   ```

2. **Firewall allows SSH:**
   ```bash
   sudo firewall-cmd --list-services  # Should include 'ssh'
   # If not present, SSH was already allowed in this setup
   ```

3. **Authorized keys configured:**
   ```bash
   # homelab_rsa.pub should be in ~/.ssh/authorized_keys
   cat ~/.ssh/homelab_rsa.pub >> ~/.ssh/authorized_keys
   chmod 600 ~/.ssh/authorized_keys
   ```

### Key Transfer Process
1. Copy private key: `cp ~/.ssh/homelab_rsa ./homelab_rsa_for_mobile`
2. Transfer securely to mobile device (AirDrop, secure file share)
3. Import into Termius app
4. Clean up: `rm homelab_rsa_for_mobile`

### Network Discovery
Find local machine IP: `ip addr show eno1 | grep 'inet ' | awk '{print $2}' | cut -d'/' -f1`

## Related Documentation

- Patterns: `patterns/networking/ssh-key-management.md`
- Troubleshooting: `reference/networking/ssh-troubleshooting.md`