claude-configs/skills/major-domo/workflows/database-migration.md

# Database Migration Workflow

## Overview

Apply database schema changes safely to production and development environments.

## When to Use

- Schema changes (new tables, columns, constraints)
- Data structure updates
- Index creation/optimization
- Data migrations or transformations

## Pre-Migration Checklist

- [ ] Migration tested on development database
- [ ] Migration SQL file created and reviewed
- [ ] Backup plan confirmed
- [ ] Rollback plan prepared
- [ ] Downtime requirements assessed
- [ ] Affected endpoints/code identified
- [ ] Users notified (if needed)

## Migration File Structure

**Location**: `/mnt/NV2/Development/major-domo/database/migrations/`

**Naming Convention**: `migration_NNN.sql` (e.g., `migration_015.sql`)

**File Structure**:
```sql
-- Migration: Brief description
-- Date: YYYY-MM-DD
-- Author: Your Name

-- Start transaction
BEGIN;

-- Your migration SQL here
ALTER TABLE players ADD COLUMN new_field VARCHAR(100);

-- Commit transaction
COMMIT;

-- Rollback (comment out, for reference)
-- BEGIN;
-- ALTER TABLE players DROP COLUMN new_field;
-- COMMIT;
```

## Development Testing

### 1. Create Migration File

```bash
cd /mnt/NV2/Development/major-domo/database/migrations
nano migration_XXX.sql  # Use next available number
```

### 2. Test on Development Database

**Via Docker**:
```bash
cd /mnt/NV2/Development/major-domo/database

# Apply migration
docker-compose exec postgres psql -U sba_admin -d sba_master -f /migrations/migration_XXX.sql

# Or copy to container and run
docker cp migrations/migration_XXX.sql major-domo-postgres:/tmp/
docker-compose exec postgres psql -U sba_admin -d sba_master -f /tmp/migration_XXX.sql
```

**Via psql directly**:
```bash
psql -h 10.10.0.42 -p 5433 -U sba_admin -d sba_master -f migrations/migration_XXX.sql
```

### 3. Verify Changes

```sql
-- Connect to database
psql -h 10.10.0.42 -p 5433 -U sba_admin -d sba_master

-- Check table structure
\d table_name

-- Verify data
SELECT * FROM table_name LIMIT 5;

-- Check indexes
\di table_name*

-- Verify constraints
\d+ table_name
```

### 4. Test API Endpoints

```bash
# Test affected endpoints
curl -H "Authorization: Bearer $API_TOKEN" \
  http://10.10.0.42:8000/api/v3/affected-endpoint

# Or use API client
cd ~/.claude/skills/major-domo
python api_client.py --env dev --verbose
```

### 5. Test Bot Commands

```bash
cd /mnt/NV2/Development/major-domo/discord-app-v2
python bot.py

# Test commands that use affected tables
```

## Production Migration

### 1. Backup Production Database

**Create Backup**:
```bash
# SSH to production server
ssh akamai

# Create backup with timestamp
pg_dump -h 10.10.0.42 -U sba_admin -d sba_master > \
  ~/backups/sba_master_$(date +%Y%m%d_%H%M%S).sql

# Or via Docker
docker-compose exec postgres pg_dump -U sba_admin sba_master > \
  backups/sba_master_$(date +%Y%m%d_%H%M%S).sql
```

**Verify Backup**:
```bash
ls -lh ~/backups/sba_master_*.sql
head -n 20 ~/backups/sba_master_*.sql
```

### 2. Assess Downtime Requirements

**No Downtime Needed**:
- Adding nullable columns
- Creating indexes (CONCURRENTLY)
- Adding tables not yet used

**Downtime Required**:
- Altering column types
- Adding NOT NULL constraints
- Dropping columns/tables
- Major data transformations

**If Downtime Needed**:
```bash
# Announce in Discord
"⚠️ Database maintenance scheduled for [TIME].
Expected downtime: [DURATION]. Bot will be offline."
```

### 3. Stop API/Bot (if downtime required)

**Stop Discord Bot**:
```bash
ssh akamai
screen -r major-domo
# Ctrl+C to stop bot
```

**Stop API** (if needed):
```bash
cd ~/major-domo/database
docker-compose stop app
```

### 4. Apply Migration to Production

**Via psql**:
```bash
psql -h 10.10.0.42 -U sba_admin -d sba_master -f migrations/migration_XXX.sql
```

**Monitor Output**:
- Watch for errors
- Note any warnings
- Verify completion message

### 5. Verify Production Changes

```sql
-- Connect to production database
psql -h 10.10.0.42 -U sba_admin -d sba_master

-- Check table structure
\d table_name

-- Verify data integrity
SELECT COUNT(*) FROM table_name;
SELECT * FROM table_name WHERE new_field IS NULL LIMIT 10;

-- Check constraints
SELECT conname, contype, conrelid::regclass
FROM pg_constraint
WHERE conrelid = 'table_name'::regclass;
```

### 6. Update Code (if needed)

**Update ORM Models** (`database/app/db_engine.py`):
```python
class Player(BaseModel):
    # ... existing fields ...
    new_field = CharField(null=True, max_length=100)

    class Meta:
        table_name = 'players'
```

**Deploy Code Changes**:
```bash
# Pull latest code
cd ~/major-domo/database
git pull origin main

# Restart API
docker-compose restart app
```

### 7. Restart Services

**Restart API**:
```bash
docker-compose start app

# Verify API is running
curl http://10.10.0.42/api/v3/current
```

**Restart Bot**:
```bash
screen -r major-domo
python bot.py
# Ctrl+A then D to detach
```

### 8. Post-Migration Verification

**Test API Endpoints**:
```bash
# Test affected endpoints
curl -H "Authorization: Bearer $API_TOKEN" \
  http://10.10.0.42/api/v3/players?season=12&limit=5

# Test with API client
cd ~/.claude/skills/major-domo
python api_client.py --env prod --verbose
```

**Test Bot Commands**:
- Verify commands work
- Check for error messages
- Test edge cases

**Monitor Logs**:
```bash
# API logs
docker-compose logs -f app

# Bot logs
tail -f ~/major-domo/discord-app-v2/logs/discord.log
```

## Rollback Procedure

### If Migration Fails

**Immediate Rollback**:
```sql
-- Connect to database
psql -h 10.10.0.42 -U sba_admin -d sba_master

-- Start transaction
BEGIN;

-- Rollback migration (reverse operations)
ALTER TABLE players DROP COLUMN new_field;

-- Commit rollback
COMMIT;
```

### Restore from Backup

**If Critical Failure**:
```bash
# Stop services
docker-compose stop app

# Restore database from backup
psql -h 10.10.0.42 -U sba_admin -d sba_master < \
  ~/backups/sba_master_YYYYMMDD_HHMMSS.sql

# Restart services
docker-compose start app
```

## Common Migration Patterns

### Adding a Column

```sql
-- Add nullable column first
ALTER TABLE players ADD COLUMN new_field VARCHAR(100);

-- Optionally populate data
UPDATE players SET new_field = 'default_value' WHERE condition;

-- Later, make NOT NULL if needed (separate migration)
-- ALTER TABLE players ALTER COLUMN new_field SET NOT NULL;
```

### Creating an Index

```sql
-- Use CONCURRENTLY to avoid locking (no downtime)
CREATE INDEX CONCURRENTLY idx_players_new_field
ON players(new_field);
```

### Renaming a Column

```sql
-- Rename column
ALTER TABLE players RENAME COLUMN old_name TO new_name;

-- Note: Update ORM models and code immediately
```

### Adding a Foreign Key

```sql
-- Add column
ALTER TABLE players ADD COLUMN league_id INTEGER;

-- Add foreign key constraint
ALTER TABLE players
ADD CONSTRAINT fk_players_league
FOREIGN KEY (league_id) REFERENCES leagues(id);
```

### Data Transformation

```sql
-- Complex data migrations in transaction
BEGIN;

-- Create temporary column
ALTER TABLE players ADD COLUMN temp_field TEXT;

-- Transform data
UPDATE players
SET temp_field = CONCAT(first_name, ' ', last_name);

-- Drop old columns
ALTER TABLE players DROP COLUMN first_name;
ALTER TABLE players DROP COLUMN last_name;

-- Rename temp column
ALTER TABLE players RENAME COLUMN temp_field TO full_name;

COMMIT;
```

## Best Practices

1. **Always Test on Dev First**: Never run untested migrations on production
2. **Use Transactions**: Wrap migrations in BEGIN/COMMIT blocks
3. **Backup Before Migration**: Always have a recent backup
4. **Small, Incremental Changes**: Break large migrations into smaller steps
5. **Document Changes**: Add comments explaining why and what
6. **Version Control**: Commit migration files to git
7. **Test Rollback**: Verify rollback SQL works on dev
8. **Monitor Performance**: Check query performance after schema changes
9. **Update Documentation**: Reflect schema changes in docs

## Migration Checklist Template

```
Migration: [Brief Description]
Date: [YYYY-MM-DD]
Developer: [Your Name]

Pre-Migration:
[ ] Migration file created
[ ] Tested on dev database
[ ] API tested with changes
[ ] Bot tested with changes
[ ] Backup created
[ ] Rollback plan prepared
[ ] Downtime assessment complete
[ ] Users notified (if needed)

Migration:
[ ] Services stopped (if needed)
[ ] Migration applied to production
[ ] Changes verified in database
[ ] ORM models updated
[ ] Code deployed
[ ] Services restarted

Post-Migration:
[ ] API endpoints tested
[ ] Bot commands tested
[ ] Logs monitored
[ ] No errors detected
[ ] Performance verified
[ ] Documentation updated

Rollback (if needed):
[ ] Rollback SQL prepared
[ ] Backup restoration tested
[ ] Services can be restored
```

## Environment-Specific Notes

### Development Database
- **Host**: 10.10.0.42
- **Port**: 5433 (Docker)
- **Database**: sba_master
- **User**: sba_admin
- **Docker Container**: major-domo-postgres

### Production Database
- **Host**: 10.10.0.42
- **Port**: 5432
- **Database**: sba_master
- **User**: sba_admin
- **Access**: Via SSH to Akamai Linode

## Troubleshooting

### Permission Errors
```sql
-- Grant permissions to user
GRANT ALL PRIVILEGES ON TABLE table_name TO sba_admin;
```

### Lock Errors
```sql
-- Check for locks
SELECT * FROM pg_locks WHERE NOT granted;

-- Find blocking queries
SELECT pid, query FROM pg_stat_activity WHERE state != 'idle';
```

### Connection Issues
```bash
# Check PostgreSQL is running
docker-compose ps

# Check logs
docker-compose logs postgres
```

---

**Last Updated**: 2025-11-10
**Maintainer**: Cal Corum