# VM Management Scripts - Operational Context

## Script Overview
This directory contains active operational scripts for VM provisioning, LXC container creation, Docker configuration in containers, and system migration.

## Core Scripts

### VM Post-Installation Provisioning
**Script**: `vm-post-install.sh`
**Purpose**: Automated provisioning of existing VMs with security hardening, SSH keys, and Docker

**Key Features**:
- System updates and essential packages installation
- SSH key deployment (primary + emergency keys)
- SSH security hardening (disable password authentication)
- Docker and Docker Compose installation
- User environment setup with bash aliases
- Automatic security updates configuration

**Usage**:
```bash
./vm-post-install.sh <vm-ip> [ssh-user]

# Example
./vm-post-install.sh 10.10.0.100 cal
```

**Requirements**:
- Target VM must have SSH access enabled initially
- Homelab SSH keys must exist: `~/.ssh/homelab_rsa` and `~/.ssh/emergency_homelab_rsa`
- Initial connection may require password authentication (disabled after provisioning)

**Post-Provision Verification**:
```bash
# Test SSH with key
ssh cal@<vm-ip>

# Verify Docker
docker --version
docker run --rm hello-world

# Check security
sudo sshd -T | grep -E "(passwordauth|pubkeyauth)"
```

### Cloud-Init Automated Provisioning
**File**: `cloud-init-user-data.yaml`
**Purpose**: Fully automated VM provisioning from first boot using Proxmox cloud-init

**Features**:
- User creation with sudo privileges
- SSH keys pre-installed (no password auth needed)
- Automatic package updates
- Docker and Docker Compose installation
- Security hardening from first boot
- Useful bash aliases and environment setup
- Welcome message with system status

**Usage in Proxmox**:
1. Create new VM with cloud-init support
2. Go to Cloud-Init tab in VM settings
3. Copy contents of `cloud-init-user-data.yaml`
4. Paste into "User Data" field
5. Start VM - fully provisioned automatically

**Benefits**:
- Zero-touch provisioning
- Consistent configuration across all VMs
- No password authentication ever enabled
- Faster deployment than post-install script

### Docker AppArmor Fix for LXC
**Script**: `fix-docker-apparmor.sh`
**Purpose**: Add AppArmor=unconfined to docker-compose.yml files for LXC compatibility

**Why Needed**: Docker containers inside LXC containers require AppArmor to be disabled. Without this fix, containers may fail to start or have permission issues.

**Usage**:
```bash
./fix-docker-apparmor.sh <LXC_IP> [COMPOSE_DIR]

# Example - use default directory (/home/cal/container-data)
./fix-docker-apparmor.sh 10.10.0.214

# Example - specify custom directory
./fix-docker-apparmor.sh 10.10.0.214 /home/cal/docker
```

**What It Does**:
1. SSHs into the LXC container
2. Finds all docker-compose.yml files in specified directory
3. Adds `security_opt: ["apparmor=unconfined"]` to each service
4. Creates `.bak` backups of original files before modification

**Safety Features**:
- Creates backups before modifications
- Color-coded output for easy monitoring
- Error handling with detailed logging
- Validates SSH connectivity before proceeding

### LXC Container Creation with Docker
**Script**: `lxc-docker-create.sh`
**Purpose**: Create new LXC containers pre-configured for Docker hosting

**Key Features**:
- Automated LXC container creation in Proxmox
- Docker and Docker Compose pre-installed
- AppArmor configuration for container compatibility
- Network configuration
- Security settings optimized for Docker hosting

**Usage**:
```bash
./lxc-docker-create.sh [options]
```

**Common Use Cases**:
- Creating Docker hosts for specific services (n8n, gitea, etc.)
- Rapid deployment of containerized applications
- Consistent LXC configuration across infrastructure

### LXC Migration Guide
**Document**: `LXC-MIGRATION-GUIDE.md`
**Purpose**: Step-by-step procedures for migrating LXC containers between hosts

**Covers**:
- Pre-migration planning and backups
- LXC configuration export/import
- Storage migration strategies
- Network reconfiguration
- Post-migration validation
- Rollback procedures

**When to Use**:
- Moving containers to new Proxmox host
- Hardware upgrades
- Load balancing across nodes
- Disaster recovery scenarios

## Operational Patterns

### VM Provisioning Workflow

**Option 1: New VMs (Preferred)**
```bash
# 1. Create VM in Proxmox with cloud-init support
# 2. Copy cloud-init-user-data.yaml to User Data field
# 3. Start VM
# 4. Verify provisioning completed:
ssh cal@<vm-ip>
docker --version
```

**Option 2: Existing VMs**
```bash
# 1. Ensure VM has SSH enabled and accessible
# 2. Run post-install script:
./vm-post-install.sh 10.10.0.100 cal

# 3. Verify provisioning:
ssh cal@10.10.0.100
docker --version
```

### LXC Docker Deployment Workflow

**Creating New LXC for Docker**:
```bash
# 1. Create LXC container
./lxc-docker-create.sh --id 220 --hostname docker-app --ip 10.10.0.220

# 2. Deploy docker-compose configurations
scp -r ./app-config/ cal@10.10.0.220:/home/cal/container-data/

# 3. Fix AppArmor compatibility
./fix-docker-apparmor.sh 10.10.0.220

# 4. Start containers
ssh cal@10.10.0.220 "cd /home/cal/container-data/app-config && docker compose up -d"
```

**Existing LXC with Docker Issues**:
```bash
# If containers failing to start in LXC:
./fix-docker-apparmor.sh <LXC_IP>

# Restart affected containers
ssh cal@<LXC_IP> "cd /home/cal/container-data && docker compose restart"
```

### SSH Key Integration

**Both provisioning methods use**:
- **Primary Key**: `~/.ssh/homelab_rsa` - Daily use authentication
- **Emergency Key**: `~/.ssh/emergency_homelab_rsa` - Backup access

**Security Configuration**:
- Password authentication completely disabled after provisioning
- Only key-based SSH access allowed
- Emergency keys provide backup access if primary key fails
- Automatic security updates enabled

**Key Management Integration**:
- Keys managed by `/networking/scripts/ssh_key_maintenance.sh`
- Monthly backups of all SSH keys
- Rotation recommendations for keys > 365 days old

## Configuration Dependencies

### Required Local Files
- `~/.ssh/homelab_rsa` - Primary SSH private key
- `~/.ssh/homelab_rsa.pub` - Primary SSH public key
- `~/.ssh/emergency_homelab_rsa` - Emergency SSH private key
- `~/.ssh/emergency_homelab_rsa.pub` - Emergency SSH public key

### Target VM Requirements
- **For post-install script**: SSH enabled, initial authentication method available
- **For cloud-init**: Proxmox cloud-init support, fresh VM
- **For LXC**: Proxmox host with LXC support

### Network Requirements
- VMs/LXCs on 10.10.0.0/24 network (homelab subnet)
- SSH access (port 22) to target systems
- Internet access on target systems for package installation

## Troubleshooting Context

### Common Issues

**1. vm-post-install.sh Connection Failures**
```bash
# Verify VM is accessible
ping <vm-ip>
nc -z <vm-ip> 22

# Check SSH service on target
ssh <vm-ip> "systemctl status sshd"

# Verify SSH keys exist locally
ls -la ~/.ssh/homelab_rsa*
```

**2. Cloud-Init Not Working**
```bash
# On Proxmox host, check cloud-init support
qm cloudinit dump <vmid>

# On VM, check cloud-init logs
sudo cloud-init status --long
sudo cat /var/log/cloud-init.log
```

**3. Docker Containers Fail in LXC**
```bash
# Symptom: Containers won't start, permission errors
# Solution: Run AppArmor fix
./fix-docker-apparmor.sh <LXC_IP>

# Verify security_opt was added
ssh cal@<LXC_IP> "grep -r 'security_opt' ~/container-data/"

# Check Docker logs
ssh cal@<LXC_IP> "docker compose logs"
```

**4. SSH Key Authentication Fails After Provisioning**
```bash
# Verify key permissions
ls -la ~/.ssh/homelab_rsa
chmod 600 ~/.ssh/homelab_rsa

# Check authorized_keys on target
ssh <vm-ip> "cat ~/.ssh/authorized_keys"

# Test with verbose output
ssh -v cal@<vm-ip>
```

**5. Docker Installation Issues**
```bash
# Check internet connectivity on VM
ssh <vm-ip> "ping -c 3 8.8.8.8"

# Verify Docker GPG key
ssh <vm-ip> "apt-key list | grep -i docker"

# Check Docker service
ssh <vm-ip> "systemctl status docker"

# Manual Docker install if needed
ssh <vm-ip> "curl -fsSL https://get.docker.com | sh"
```

### Diagnostic Commands

```bash
# Post-provisioning validation
ssh cal@<vm-ip> "groups"  # Should include: sudo docker
ssh cal@<vm-ip> "docker run --rm hello-world"
ssh cal@<vm-ip> "sudo sshd -T | grep passwordauth"  # Should be "no"

# Cloud-init status check
ssh cal@<vm-ip> "cloud-init status"
ssh cal@<vm-ip> "cloud-init query -f '{{ds.meta_data.hostname}}'"

# Docker in LXC verification
ssh cal@<LXC_IP> "docker info | grep -i apparmor"
ssh cal@<LXC_IP> "docker compose config" # Validate compose files

# SSH key connectivity test
ssh -o ConnectTimeout=5 cal@<vm-ip> "echo 'SSH OK'"
```

## Integration Points

### External Dependencies
- **Proxmox VE**: For VM/LXC creation and cloud-init support
- **SSH**: For remote provisioning and management
- **Docker**: Installed on target systems
- **Cloud-init**: For automated VM provisioning
- **AppArmor**: Security framework (configured for LXC compatibility)

### File System Dependencies
- **Script Directory**: `/mnt/NV2/Development/claude-home/vm-management/scripts/`
- **SSH Keys**: `~/.ssh/homelab_rsa*`, `~/.ssh/emergency_homelab_rsa*`
- **LXC Compose Directories**: Typically `/home/cal/container-data/` on target
- **Backup Files**: `.bak` files created by AppArmor fix script

### Network Dependencies
- **Management Network**: 10.10.0.0/24 subnet
- **Internet Access**: Required for package installation
- **Proxmox API**: For LXC creation operations
- **DNS**: For hostname resolution

## Security Considerations

### SSH Security
- Password authentication disabled after provisioning
- Only key-based authentication allowed
- Emergency keys provide backup access
- Root login disabled

### Docker Security
- User in docker group (no sudo needed for docker commands)
- AppArmor unconfined in LXC (required for functionality)
- Containers run as non-root when possible
- Network isolation via Docker networks

### VM/LXC Security
- Automatic security updates enabled
- Minimal package installation (only essentials)
- Firewall configuration recommended post-provisioning
- Regular key rotation via SSH key maintenance

## Performance Considerations

### Cloud-Init vs Post-Install
- **Cloud-init**: Faster (0-touch), no manual SSH needed, better for multiple VMs
- **Post-install**: More flexible, works with existing VMs, easier debugging

### LXC vs VM
- **LXC**: Lower overhead, faster startup, shared kernel
- **VM**: Better isolation, GPU passthrough support, different kernels possible

### Docker in LXC
- **Performance**: Near-native, minimal overhead with AppArmor disabled
- **I/O**: Use local storage for best performance, NFS for shared data
- **Networking**: Bridge mode for simplicity, macvlan for direct network access

## Related Documentation

- **Technology Overview**: `/vm-management/CONTEXT.md`
- **Troubleshooting**: `/vm-management/troubleshooting.md`
- **Examples**: `/vm-management/examples/` - Configuration templates
- **SSH Management**: `/networking/scripts/ssh_key_maintenance.sh`
- **Docker Patterns**: `/docker/CONTEXT.md`
- **Main Instructions**: `/CLAUDE.md` - Context loading rules

## Notes

These scripts form the foundation of the homelab VM and LXC provisioning strategy. They ensure consistent configuration, security hardening, and Docker compatibility across all virtualized infrastructure.

The cloud-init approach is preferred for new deployments due to zero-touch provisioning, while the post-install script provides flexibility for existing systems or troubleshooting scenarios.

AppArmor configuration is critical for Docker-in-LXC deployments and should be applied to all LXC containers running Docker to prevent container startup failures.