From 704bad1547983d46a344258f82007db99b8c627b Mon Sep 17 00:00:00 2001 From: Cal Corum Date: Fri, 8 Aug 2025 21:02:46 -0500 Subject: [PATCH] CLAUDE: Add comprehensive SSH key management documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add SSH key management patterns with dual-key strategy and NAS backup architecture - Add complete SSH home lab setup implementation with scripts and configurations - Add SSH troubleshooting reference with common issues and emergency procedures - Update CLAUDE.md with SSH keyword triggers for automatic context loading - Add .gitignore to exclude temporary files 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- .gitignore | 1 + CLAUDE.md | 8 +- examples/networking/ssh-homelab-setup.md | 324 ++++++++++++++++++++ patterns/networking/ssh-key-management.md | 100 ++++++ reference/networking/ssh-troubleshooting.md | 258 ++++++++++++++++ 5 files changed, 690 insertions(+), 1 deletion(-) create mode 100644 .gitignore create mode 100644 examples/networking/ssh-homelab-setup.md create mode 100644 patterns/networking/ssh-key-management.md create mode 100644 reference/networking/ssh-troubleshooting.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..bfc8577 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.claude/tmp/ \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 62ceba1..e2f3a1b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,7 +5,7 @@ - When new code is written, include checking for imports as part of the code review. We should never get NameErrors during testing. - Always ask clarifying questions for complex configurations or multi-step tasks. - Do what has been asked; nothing more, nothing less. -- If creating a temporary file will help achieve your goal, please create the file in the /tmp directory and clean up when you're done. +- If creating a temporary file will help achieve your goal, please create the file in the .claude/tmp/ directory and clean up when you're done. - Prefer editing an existing file to creating a new one. - Following a complex task or series of tasks, prompt the user to save any key learnings from the session. @@ -101,6 +101,12 @@ When user mentions specific terms, automatically load relevant docs: - Load: `patterns/networking/` - Load: `examples/networking/` +**SSH Keywords** +- "ssh", "key", "authentication", "authorized_keys", "ssh-copy-id" + - Load: `patterns/networking/ssh-key-management.md` + - Load: `examples/networking/ssh-homelab-setup.md` + - Load: `reference/networking/ssh-troubleshooting.md` + **VM Keywords** - "virtual machine", "vm", "proxmox", "kvm", "hypervisor", "guest" - Load: `patterns/vm-management/` diff --git a/examples/networking/ssh-homelab-setup.md b/examples/networking/ssh-homelab-setup.md new file mode 100644 index 0000000..4b76c38 --- /dev/null +++ b/examples/networking/ssh-homelab-setup.md @@ -0,0 +1,324 @@ +# SSH Home Lab Setup - Complete Implementation + +## Overview + +Complete working implementation for setting up secure SSH key-based authentication across a home lab environment with emergency backup keys and NAS storage. + +## Server Inventory + +### Home Network (10.10.0.0/24) +```yaml +servers: + database_apis: 10.10.0.42 # Database and API services + discord_bots: 10.10.0.33 # Discord bot hosting + home_docker: 10.10.0.124 # Main Docker container host + pihole: 10.10.0.16 # Pi-hole DNS and ad blocking + sba_pd_bots: 10.10.0.88 # SBa and PD bot services + tdarr: 10.10.0.43 # Media transcoding + vpn_docker: 10.10.0.121 # VPN and Docker services +``` + +### Cloud Servers +```yaml +remote_servers: + akamai_nano: 172.237.147.99 # Akamai cloud instance + vultr_host: 45.76.25.231 # Vultr cloud host +``` + +## SSH Key Generation Script + +```bash +#!/bin/bash +# SSH Key Setup Script for Home Lab +# Creates primary + emergency keys with NAS backup + +set -e + +echo "🔐 Setting up SSH keys for Home Lab with Emergency Backup..." + +# Create directories +mkdir -p ~/.ssh +chmod 700 ~/.ssh + +# Create NAS backup directory +BACKUP_DIR="/mnt/NV2/ssh-keys/backup-$(date +%Y%m%d-%H%M%S)" +mkdir -p "$BACKUP_DIR" +chmod 700 "$BACKUP_DIR" +echo "📁 NAS backup directory created: $BACKUP_DIR" + +# Generate primary keys +if [ ! -f ~/.ssh/homelab_rsa ]; then + ssh-keygen -t rsa -b 4096 -C "homelab-$(whoami)@$(hostname)" -f ~/.ssh/homelab_rsa -N "" + chmod 600 ~/.ssh/homelab_rsa + chmod 644 ~/.ssh/homelab_rsa.pub + echo "✅ Home lab SSH key generated" +fi + +if [ ! -f ~/.ssh/cloud_servers_rsa ]; then + ssh-keygen -t rsa -b 4096 -C "cloud-$(whoami)@$(hostname)" -f ~/.ssh/cloud_servers_rsa -N "" + chmod 600 ~/.ssh/cloud_servers_rsa + chmod 644 ~/.ssh/cloud_servers_rsa.pub + echo "✅ Cloud servers SSH key generated" +fi + +# Generate emergency keys +if [ ! -f ~/.ssh/emergency_homelab_rsa ]; then + ssh-keygen -t rsa -b 4096 -C "emergency-homelab-$(whoami)@$(hostname)" -f ~/.ssh/emergency_homelab_rsa -N "" + chmod 600 ~/.ssh/emergency_homelab_rsa + chmod 644 ~/.ssh/emergency_homelab_rsa.pub + echo "✅ Emergency home lab key generated" +fi + +if [ ! -f ~/.ssh/emergency_cloud_rsa ]; then + ssh-keygen -t rsa -b 4096 -C "emergency-cloud-$(whoami)@$(hostname)" -f ~/.ssh/emergency_cloud_rsa -N "" + chmod 600 ~/.ssh/emergency_cloud_rsa + chmod 644 ~/.ssh/emergency_cloud_rsa.pub + echo "✅ Emergency cloud key generated" +fi + +# Backup all keys to NAS +echo "💾 Backing up keys to NAS..." +cp ~/.ssh/*_rsa* "$BACKUP_DIR/" 2>/dev/null || true +cp ~/.ssh/config "$BACKUP_DIR/" 2>/dev/null || true + +# Create recovery documentation +cat > "$BACKUP_DIR/RECOVERY_INSTRUCTIONS.md" << EOF +# SSH Key Recovery Instructions +Generated: $(date) +Location: $BACKUP_DIR + +## Emergency Recovery Steps + +### If you lose access to your primary keys: + +1. **Copy emergency keys back:** + \`\`\`bash + cp $BACKUP_DIR/emergency_*_rsa* ~/.ssh/ + chmod 600 ~/.ssh/emergency_*_rsa + chmod 644 ~/.ssh/emergency_*_rsa.pub + \`\`\` + +2. **Test emergency access:** + \`\`\`bash + ssh -i ~/.ssh/emergency_homelab_rsa cal@10.10.0.16 + ssh -i ~/.ssh/emergency_cloud_rsa root@172.237.147.99 + \`\`\` + +3. **If emergency keys weren't deployed, use console access:** + - Home servers: Physical console access + - Cloud servers: Provider web console + - Add emergency public key to ~/.ssh/authorized_keys +EOF + +ls -la "$BACKUP_DIR" >> "$BACKUP_DIR/RECOVERY_INSTRUCTIONS.md" +echo "✅ Keys backed up to: $BACKUP_DIR" + +echo "🎉 SSH keys and emergency backups created successfully!" +``` + +## SSH Configuration + +```bash +# SSH Configuration for Home Lab (~/.ssh/config) +# Global defaults +Host * + ServerAliveInterval 60 + ServerAliveCountMax 3 + StrictHostKeyChecking ask + VerifyHostKeyDNS yes + ForwardAgent no + ForwardX11 no + AddKeysToAgent yes + +# Home Network Servers +Host strat-database pd-database sba-database strat-db pd-db sba-db + HostName 10.10.0.42 + User cal + Port 22 + IdentityFile ~/.ssh/homelab_rsa + +Host discord-bots + HostName 10.10.0.33 + User cal + Port 22 + IdentityFile ~/.ssh/homelab_rsa + +Host docker-home docker-main + HostName 10.10.0.124 + User cal + Port 22 + IdentityFile ~/.ssh/homelab_rsa + +Host pihole dns + HostName 10.10.0.16 + User cal + Port 22 + IdentityFile ~/.ssh/homelab_rsa + +Host sba-bots pd-bots + HostName 10.10.0.88 + User cal + Port 22 + IdentityFile ~/.ssh/homelab_rsa + +Host tdarr media + HostName 10.10.0.43 + User cal + Port 22 + IdentityFile ~/.ssh/homelab_rsa + +Host docker-vpn + HostName 10.10.0.121 + User cal + Port 22 + IdentityFile ~/.ssh/homelab_rsa + +# Remote Cloud Servers +Host akamai-nano akamai + HostName 172.237.147.99 + User root + Port 22 + IdentityFile ~/.ssh/cloud_servers_rsa + +Host vultr-host vultr + HostName 45.76.25.231 + User root + Port 22 + IdentityFile ~/.ssh/cloud_servers_rsa + +# Home network wildcard for easy access +Host 10.10.0.* + User cal + IdentityFile ~/.ssh/homelab_rsa + StrictHostKeyChecking accept-new +``` + +## Key Deployment Commands + +### Primary Keys +```bash +# Home lab servers +for ip in 42 33 124 16 88 43 121; do + ssh-copy-id -i ~/.ssh/homelab_rsa.pub cal@10.10.0.$ip +done + +# Cloud servers +ssh-copy-id -i ~/.ssh/cloud_servers_rsa.pub root@172.237.147.99 +ssh-copy-id -i ~/.ssh/cloud_servers_rsa.pub root@45.76.25.231 +``` + +### Emergency Keys +```bash +# Home lab servers (use -f to force) +for ip in 42 33 124 16 88 43 121; do + ssh-copy-id -f -i ~/.ssh/emergency_homelab_rsa.pub cal@10.10.0.$ip +done + +# Cloud servers +ssh-copy-id -f -i ~/.ssh/emergency_cloud_rsa.pub root@172.237.147.99 +ssh-copy-id -f -i ~/.ssh/emergency_cloud_rsa.pub root@45.76.25.231 +``` + +## Maintenance Script + +```bash +#!/bin/bash +# SSH Key Maintenance and Backup Script +# Location: ~/bin/ssh_key_maintenance.sh +# Schedule: 0 2 1 * * (monthly at 2 AM) + +echo "🔧 SSH Key Maintenance and Backup" + +# Check NAS availability +if [ ! -d "/mnt/NV2" ]; then + echo "❌ ERROR: NAS not mounted at /mnt/NV2" + exit 1 +fi + +TIMESTAMP=$(date +%Y%m%d-%H%M%S) +BACKUP_ROOT="/mnt/NV2/ssh-keys" +BACKUP_DIR="$BACKUP_ROOT/maintenance-$TIMESTAMP" + +mkdir -p "$BACKUP_DIR" +chmod 700 "$BACKUP_DIR" + +# Backup current state +cp ~/.ssh/*_rsa* "$BACKUP_DIR/" 2>/dev/null || true +cp ~/.ssh/config "$BACKUP_DIR/" 2>/dev/null || true +cp ~/.ssh/known_hosts "$BACKUP_DIR/" 2>/dev/null || true + +# Check key ages and health +echo "🔍 Key Age Analysis:" +for key in ~/.ssh/*_rsa; do + if [ -f "$key" ]; then + age_days=$(( ($(date +%s) - $(stat -c %Y "$key")) / 86400 )) + basename_key=$(basename "$key") + + if [ $age_days -gt 365 ]; then + echo "⚠️ $basename_key: $age_days days old - ROTATION RECOMMENDED" + elif [ $age_days -gt 180 ]; then + echo "⚡ $basename_key: $age_days days old - consider rotation" + else + echo "✅ $basename_key: $age_days days old - OK" + fi + fi +done + +# Clean up old backups (keep last 10) +cd "$BACKUP_ROOT" +ls -dt backup-* maintenance-* 2>/dev/null | tail -n +11 | while read old_backup; do + if [ -d "$old_backup" ]; then + echo "🗑️ Removing old backup: $old_backup" + rm -rf "$old_backup" + fi +done + +echo "✅ Maintenance backup completed: $BACKUP_DIR" +``` + +## Testing Procedures + +### Test Primary Access +```bash +# Test aliases work +ssh strat-database 'echo "Primary access: $(hostname)"' +ssh pihole 'echo "Primary access: $(hostname)"' +ssh akamai 'echo "Primary access: $(hostname)"' +``` + +### Test Emergency Access +```bash +# Test emergency keys work +ssh -i ~/.ssh/emergency_homelab_rsa cal@10.10.0.16 'echo "Emergency access: $(hostname)"' +ssh -i ~/.ssh/emergency_cloud_rsa root@172.237.147.99 'echo "Emergency access: $(hostname)"' +``` + +## Cron Schedule Setup + +```bash +# Open crontab editor +crontab -e + +# Add monthly maintenance (1st of month at 2 AM) +0 2 1 * * /home/cal/bin/ssh_key_maintenance.sh +``` + +## Security Hardening + +After key deployment, disable password authentication on servers: + +```bash +# On each server, edit /etc/ssh/sshd_config: +PasswordAuthentication no +PubkeyAuthentication yes +PermitRootLogin no # Create non-root users first on cloud servers +AllowUsers cal # Restrict SSH access + +# Restart SSH service +sudo systemctl restart sshd +``` + +## Related Documentation + +- Patterns: `patterns/networking/ssh-key-management.md` +- Troubleshooting: `reference/networking/ssh-troubleshooting.md` \ No newline at end of file diff --git a/patterns/networking/ssh-key-management.md b/patterns/networking/ssh-key-management.md new file mode 100644 index 0000000..77b397b --- /dev/null +++ b/patterns/networking/ssh-key-management.md @@ -0,0 +1,100 @@ +# SSH Key Management for Home Labs + +## Overview + +This document outlines best practices for managing SSH keys in home lab environments, focusing on security, reliability, and maintainability. + +## Core Principles + +### Dual-Key Strategy +- **Primary keys**: Daily use authentication +- **Emergency keys**: Backup access when primary keys fail +- **Separate key pairs**: Home network vs cloud servers +- **Multiple authorized keys**: Each server accepts both primary and emergency + +### Key Lifecycle Management +- **Generation**: 4096-bit RSA keys with descriptive comments +- **Distribution**: Automated deployment with `ssh-copy-id` +- **Backup**: Centralized storage on NAS with recovery documentation +- **Rotation**: Annual for primary keys, bi-annual for emergency keys +- **Monitoring**: Monthly health checks and access verification + +## Architecture Patterns + +### Network Segmentation +``` +Home Network (10.10.0.0/24) +├── Primary: ~/.ssh/homelab_rsa +├── Emergency: ~/.ssh/emergency_homelab_rsa +└── Wildcard config for easy access + +Cloud Servers (Public IPs) +├── Primary: ~/.ssh/cloud_servers_rsa +├── Emergency: ~/.ssh/emergency_cloud_rsa +└── Individual host configurations +``` + +### Backup Strategy +``` +NAS Storage: /mnt/NV2/ssh-keys/ +├── backup-YYYYMMDD-HHMMSS/ +│ ├── All key pairs (*.rsa, *.rsa.pub) +│ ├── SSH config +│ └── RECOVERY_INSTRUCTIONS.md +└── maintenance-YYYYMMDD-HHMMSS/ + ├── Current state backup + ├── Key health report + └── MAINTENANCE_REPORT.md +``` + +## Security Considerations + +### Authentication Methods +- **Eliminate password authentication** after key deployment +- **Use key-based authentication** exclusively +- **Deploy multiple keys** per server for redundancy +- **Maintain console access** as ultimate fallback + +### Access Control +- **User-specific keys** (avoid root when possible) +- **Service-specific aliases** for organized access +- **Strict host key checking** for unknown servers +- **Accept-new policy** for trusted home network + +### Key Protection +- **Proper file permissions** (600 for private keys) +- **No passphrase** for automation (home lab context) +- **Regular backup verification** +- **Secure storage location** on NAS + +## Maintenance Practices + +### Automated Monitoring +- **Monthly maintenance script** via cron +- **Key health verification** +- **Connection testing** +- **Backup rotation** (keep 10 most recent) +- **Age-based rotation alerts** + +### Recovery Procedures +- **Emergency key deployment** for immediate access +- **NAS backup restoration** for complete recovery +- **Console access documentation** for worst-case scenarios +- **Provider web console** access for cloud servers + +## Implementation Guidelines + +1. **Start with key generation** using standardized naming +2. **Deploy primary keys first** and test thoroughly +3. **Add emergency keys** to all servers +4. **Configure SSH client** with aliases and settings +5. **Implement backup strategy** with NAS storage +6. **Schedule maintenance automation** +7. **Document recovery procedures** +8. **Test emergency access regularly** + +## Related Documentation + +- Implementation: `examples/networking/ssh-homelab-setup.md` +- Troubleshooting: `reference/networking/ssh-troubleshooting.md` +- Security patterns: `patterns/networking/security.md` \ No newline at end of file diff --git a/reference/networking/ssh-troubleshooting.md b/reference/networking/ssh-troubleshooting.md new file mode 100644 index 0000000..ba1f1a1 --- /dev/null +++ b/reference/networking/ssh-troubleshooting.md @@ -0,0 +1,258 @@ +# SSH Troubleshooting Reference + +## Common Configuration Issues + +### UseKeychain Compatibility Error +**Error:** `Bad configuration option: usekeychain` + +**Cause:** `UseKeychain yes` is macOS-specific and not supported on Linux + +**Solution:** Remove or comment out the line from SSH config +```bash +# UseKeychain yes # macOS only - remove on Linux +``` + +### Port Forwarding Conflicts +**Error:** `bind [127.0.0.1]:8080: Address already in use` + +**Cause:** Local port already in use by another service + +**Solutions:** +1. Remove LocalForward line from SSH config +2. Change to different port: `LocalForward 8081 localhost:80` +3. Find conflicting service: `sudo netstat -tulpn | grep :8080` + +### Host Key Verification Loops +**Issue:** Asked to verify host key on every connection + +**Cause:** SSH config discarding host keys with `UserKnownHostsFile /dev/null` + +**Solution:** Change StrictHostKeyChecking policy +```bash +# Instead of: +StrictHostKeyChecking no +UserKnownHostsFile /dev/null + +# Use: +StrictHostKeyChecking accept-new +``` + +## Key Deployment Issues + +### ssh-copy-id False Warnings +**Warning:** `All keys were skipped because they already exist on the remote system` + +**Issue:** Warning appears even when keys aren't actually deployed + +**Solution:** Force deployment with `-f` flag +```bash +ssh-copy-id -f -i ~/.ssh/emergency_homelab_rsa.pub cal@10.10.0.42 +``` + +### Permission Denied After Key Deployment +**Error:** `Permission denied (publickey)` + +**Troubleshooting Steps:** +1. Check key permissions locally: + ```bash + ls -la ~/.ssh/ + # Private keys should be 600, public keys 644 + ``` + +2. Check authorized_keys on remote server: + ```bash + ssh user@server "ls -la ~/.ssh/authorized_keys" + # Should be 600 with correct ownership + ``` + +3. Verify key is actually deployed: + ```bash + ssh user@server "cat ~/.ssh/authorized_keys" + ``` + +4. Test specific key file: + ```bash + ssh -i ~/.ssh/specific_key user@server + ``` + +### Key Authentication Not Working +**Debug connection issues:** +```bash +# Verbose SSH connection for debugging +ssh -v user@server + +# Super verbose for detailed debugging +ssh -vvv user@server + +# Test specific identity file +ssh -i ~/.ssh/homelab_rsa -v cal@10.10.0.42 +``` + +## Server-Side Issues + +### SSH Service Not Running +**Check SSH service status:** +```bash +sudo systemctl status sshd +sudo systemctl start sshd +sudo systemctl enable sshd +``` + +### Firewall Blocking SSH +**Check firewall rules:** +```bash +# Ubuntu/Debian +sudo ufw status +sudo ufw allow ssh + +# CentOS/RHEL +sudo firewall-cmd --list-services +sudo firewall-cmd --add-service=ssh --permanent +sudo firewall-cmd --reload +``` + +### Wrong SSH Port +**Check SSH configuration:** +```bash +sudo grep "^Port" /etc/ssh/sshd_config +# Update SSH client config accordingly +``` + +## Emergency Access Procedures + +### Primary Keys Lost/Corrupted +1. **Use emergency keys:** + ```bash + ssh -i ~/.ssh/emergency_homelab_rsa cal@10.10.0.16 + ``` + +2. **Restore from NAS backup:** + ```bash + cp /mnt/NV2/ssh-keys/backup-*/homelab_rsa* ~/.ssh/ + chmod 600 ~/.ssh/homelab_rsa + chmod 644 ~/.ssh/homelab_rsa.pub + ``` + +3. **Generate new keys if needed:** + ```bash + ssh-keygen -t rsa -b 4096 -f ~/.ssh/new_homelab_rsa + ssh-copy-id -i ~/.ssh/new_homelab_rsa.pub user@server + ``` + +### Complete SSH Access Lost +1. **Physical/console access** (home servers) +2. **Cloud provider web console** (cloud servers) +3. **Recovery mode** if available +4. **Manual authorized_keys editing:** + ```bash + # On the server via console: + echo "your-public-key-here" >> ~/.ssh/authorized_keys + chmod 600 ~/.ssh/authorized_keys + ``` + +## Network Connectivity Issues + +### Connection Timeouts +**Check network connectivity:** +```bash +# Basic connectivity test +ping 10.10.0.42 + +# Check if SSH port is open +telnet 10.10.0.42 22 +# Or using nc +nc -zv 10.10.0.42 22 +``` + +### DNS Resolution Issues +**Bypass DNS with IP addresses:** +```bash +# Instead of hostname +ssh server.local + +# Use IP directly +ssh 10.10.0.42 +``` + +### VPN/Network Routing +**Check routing to server:** +```bash +traceroute 10.10.0.42 +ip route | grep 10.10.0.0 +``` + +## Configuration Validation + +### SSH Config Syntax Check +```bash +# Test SSH config syntax +ssh -F ~/.ssh/config -T git@github.com 2>&1 | head +``` + +### Key Fingerprint Verification +```bash +# Local key fingerprint +ssh-keygen -lf ~/.ssh/homelab_rsa.pub + +# Remote server's authorized keys fingerprints +ssh user@server "ssh-keygen -lf ~/.ssh/authorized_keys" +``` + +### Connection Test Script +```bash +#!/bin/bash +# Test all configured SSH hosts +for host in strat-database pihole docker-home akamai vultr; do + echo "Testing $host..." + if ssh -o ConnectTimeout=5 -o BatchMode=yes "$host" 'echo "OK"' 2>/dev/null; then + echo "✅ $host: Connected successfully" + else + echo "❌ $host: Connection failed" + fi +done +``` + +## Maintenance Commands + +### Clean Up Known Hosts +```bash +# Remove specific host key +ssh-keygen -R 10.10.0.42 + +# Remove hostname and IP +ssh-keygen -R server.local +ssh-keygen -R 10.10.0.42 +``` + +### Key Rotation Process +```bash +# Generate new key +ssh-keygen -t rsa -b 4096 -f ~/.ssh/homelab_rsa_new + +# Deploy new key alongside old one +ssh-copy-id -i ~/.ssh/homelab_rsa_new.pub user@server + +# Test new key works +ssh -i ~/.ssh/homelab_rsa_new user@server + +# Update SSH config to use new key +# Remove old public key from server authorized_keys +# Archive old key pair +``` + +## Server-Specific Troubleshooting + +### Home Lab Servers (10.10.0.x) +- **Physical access available** for recovery +- **Container hosts** may need different user contexts +- **Shared credentials** historically used (security risk) + +### Cloud Servers +- **Provider console access** as fallback +- **Root user** typically used (create non-root users) +- **Different security contexts** than home network + +## Related Documentation + +- Patterns: `patterns/networking/ssh-key-management.md` +- Complete setup: `examples/networking/ssh-homelab-setup.md` \ No newline at end of file