claude-home/networking/scripts/CONTEXT.md

# Networking Scripts - Operational Context

## Script Overview
This directory contains active operational scripts for SSH key management, network security maintenance, and connectivity automation.

## Core Scripts

### SSH Key Maintenance
**Script**: `ssh_key_maintenance.sh`
**Purpose**: Comprehensive SSH key backup, age analysis, and rotation management

**Key Functions**:
- **Automated Backup**: Timestamped backups of SSH keys, config, and known_hosts
- **Age Analysis**: Identifies keys older than 180/365 days and recommends rotation
- **Validity Testing**: Verifies key integrity with ssh-keygen
- **Backup Cleanup**: Maintains only the last 10 backups
- **Maintenance Reports**: Generates comprehensive status reports with connection tests

**Usage**:
```bash
# Run manual maintenance check
./ssh_key_maintenance.sh

# Recommended: Schedule monthly via cron
0 2 1 * * /mnt/NV2/Development/claude-home/networking/scripts/ssh_key_maintenance.sh
```

**Output Locations**:
- **Backup Directory**: `/mnt/NV2/ssh-keys/maintenance-YYYYMMDD-HHMMSS/`
- **Maintenance Report**: `$BACKUP_DIR/MAINTENANCE_REPORT.md`

**Key Age Thresholds**:
- ✅ **< 180 days**: OK
- ⚡ **180-365 days**: Consider rotation
- ⚠️ **> 365 days**: Rotation recommended

**Backup Retention**: Last 10 backups kept, older ones automatically removed

## Operational Patterns

### Automated Maintenance Schedule

**Recommended Frequency**: Monthly (1st of month at 2 AM)

**Cron Configuration**:
```bash
# SSH key maintenance - Monthly backup and analysis
0 2 1 * * /mnt/NV2/Development/claude-home/networking/scripts/ssh_key_maintenance.sh >> /var/log/ssh-key-maintenance.log 2>&1
```

**Manual Execution**:
- Before major infrastructure changes
- After key rotation
- When troubleshooting SSH connectivity issues
- Quarterly security audits

### Key Rotation Workflow

**When Rotation Recommended**:
1. Run maintenance script to create pre-rotation backup
2. Generate new SSH key pair:
   ```bash
   ssh-keygen -t rsa -b 4096 -f ~/.ssh/new_key_rsa -C "description"
   ```
3. Deploy new public key to servers
4. Test connectivity with new key
5. Update SSH config to use new key
6. Archive old key (don't delete immediately)
7. Run maintenance script again to verify
8. Remove old public keys from servers after grace period (30 days)

### Backup Strategy

**Backup Contents**:
- All `*_rsa` and `*_rsa.pub` files from ~/.ssh/
- SSH config file (`~/.ssh/config`)
- Known hosts file (`~/.ssh/known_hosts`)

**Storage Location**: `/mnt/NV2/ssh-keys/`
- Persistent NAS storage
- Protected with 700 permissions
- Timestamped directories for point-in-time recovery

**Retention Policy**:
- Keep last 10 backups (approximately 10 months with monthly schedule)
- Older backups automatically pruned
- Manual backups preserved if named differently

## Configuration Dependencies

### Required Directories
- **NAS Mount**: `/mnt/NV2` must be mounted and accessible
- **Backup Root**: `/mnt/NV2/ssh-keys/` (created automatically)
- **SSH Directory**: `~/.ssh/` with appropriate permissions (700)

### SSH Key Types Supported
- **RSA keys**: `*_rsa` and `*_rsa.pub`
- **Config files**: `~/.ssh/config`
- **Known hosts**: `~/.ssh/known_hosts`

**Note**: Script currently focuses on RSA keys. For Ed25519 or other key types, script modification needed.

### Server Connection Tests

**Maintenance Report Includes**:
- Primary server connectivity tests (database-apis, pihole, akamai)
- Emergency key connectivity tests (homelab, cloud)
- Suggested commands for manual verification

**Test Targets** (from maintenance report):
```bash
# Primary Keys
ssh -o ConnectTimeout=5 database-apis 'echo "DB APIs: OK"'
ssh -o ConnectTimeout=5 pihole 'echo "PiHole: OK"'
ssh -o ConnectTimeout=5 akamai 'echo "Akamai: OK"'

# Emergency Keys
ssh -i ~/.ssh/emergency_homelab_rsa -o ConnectTimeout=5 cal@10.10.0.16 'echo "Emergency Home: OK"'
ssh -i ~/.ssh/emergency_cloud_rsa -o ConnectTimeout=5 root@172.237.147.99 'echo "Emergency Cloud: OK"'
```

## Troubleshooting Context

### Common Issues

**1. NAS Not Mounted**
- **Symptom**: Script exits with "NAS not mounted at /mnt/NV2"
- **Solution**: Verify NAS mount before running script
  ```bash
  mount | grep /mnt/NV2
  ls -la /mnt/NV2/
  ```

**2. Permission Denied on Backup**
- **Symptom**: Cannot create backup directory
- **Solution**: Check permissions on `/mnt/NV2/ssh-keys/`
  ```bash
  mkdir -p /mnt/NV2/ssh-keys/
  chmod 700 /mnt/NV2/ssh-keys/
  ```

**3. Key Corruption Detected**
- **Symptom**: Script reports "CORRUPTED or unreadable"
- **Solution**: Restore from latest backup and regenerate if necessary
  ```bash
  # Restore from backup
  cp /mnt/NV2/ssh-keys/maintenance-LATEST/key_rsa ~/.ssh/
  chmod 600 ~/.ssh/key_rsa

  # Verify
  ssh-keygen -l -f ~/.ssh/key_rsa
  ```

**4. Backup Cleanup Fails**
- **Symptom**: Old backups not being removed
- **Solution**: Manual cleanup or check directory permissions
  ```bash
  cd /mnt/NV2/ssh-keys/
  ls -dt maintenance-* | tail -n +11
  # Manually remove old backups if needed
  ```

### Diagnostic Commands

```bash
# Check script execution
bash -x ./ssh_key_maintenance.sh

# Verify NAS mount
df -h | grep NV2

# List all backups
ls -lah /mnt/NV2/ssh-keys/

# Check key permissions
ls -la ~/.ssh/*_rsa*

# Test key validity
ssh-keygen -l -f ~/.ssh/key_rsa

# View latest maintenance report
cat /mnt/NV2/ssh-keys/maintenance-*/MAINTENANCE_REPORT.md | tail -1
```

## Integration Points

### External Dependencies
- **NAS Storage**: `/mnt/NV2` must be mounted
- **SSH Tools**: `ssh-keygen`, `ssh` commands
- **cron**: For automated scheduling (optional)
- **bash**: Script requires bash shell

### File System Dependencies
- **Backup Storage**: `/mnt/NV2/ssh-keys/` (persistent NAS)
- **SSH Directory**: `~/.ssh/` (local user directory)
- **Log Files**: Optional log redirection for cron jobs

### Network Dependencies
- **NAS Access**: Network connectivity to NAS storage
- **SSH Servers**: For connection testing in maintenance report
- **DNS Resolution**: For hostname-based SSH connections

## Security Considerations

### Backup Security
- Backup directories set to 700 permissions (owner-only access)
- Private keys never transmitted over network during backup
- Backups stored on trusted NAS with access controls

### Key Rotation Best Practices
- **Annual rotation**: Recommended for primary keys
- **Bi-annual rotation**: Emergency/high-privilege keys
- **Grace period**: Keep old keys active for 30 days during transition
- **Test before removal**: Verify new key access before decommissioning old keys

### Access Control
- Script must run as user who owns SSH keys
- NAS permissions must allow user write access
- Maintenance reports contain sensitive connection information

## Future Enhancements

**Planned Improvements**:
- Support for Ed25519 and other modern key types
- Automated key rotation workflow
- Discord/webhook notifications for rotation reminders
- Integration with centralized secrets management
- Automated deployment of new public keys to servers

## Related Documentation

- **Technology Overview**: `/networking/CONTEXT.md`
- **Troubleshooting**: `/networking/troubleshooting.md`
- **Examples**: `/networking/examples/` - SSH configuration patterns
- **Main Instructions**: `/CLAUDE.md` - Context loading rules

## Notes

This script is critical for maintaining SSH key security and ensuring backup availability for disaster recovery. Regular execution (monthly recommended) helps identify aging keys before they become a security risk and maintains a backup history for point-in-time recovery.

The maintenance report format is designed to be both human-readable and actionable, providing specific commands for connectivity testing and clear recommendations for key rotation.