claude-configs/skills/major-domo/workflows/database-migration.md

Database Migration Workflow

Overview

Apply database schema changes safely to production and development environments.

When to Use

• Schema changes (new tables, columns, constraints)
• Data structure updates
• Index creation/optimization
• Data migrations or transformations

Pre-Migration Checklist

• Migration tested on development database
• Migration SQL file created and reviewed
• Backup plan confirmed
• Rollback plan prepared
• Downtime requirements assessed
• Affected endpoints/code identified
• Users notified (if needed)

Migration File Structure

Location: /mnt/NV2/Development/major-domo/database/migrations/

Naming Convention: migration_NNN.sql (e.g., migration_015.sql)

File Structure:

-- Migration: Brief description
-- Date: YYYY-MM-DD
-- Author: Your Name

-- Start transaction
BEGIN;

-- Your migration SQL here
ALTER TABLE players ADD COLUMN new_field VARCHAR(100);

-- Commit transaction
COMMIT;

-- Rollback (comment out, for reference)
-- BEGIN;
-- ALTER TABLE players DROP COLUMN new_field;
-- COMMIT;

Development Testing

1. Create Migration File

cd /mnt/NV2/Development/major-domo/database/migrations
nano migration_XXX.sql  # Use next available number

2. Test on Development Database

Via Docker:

cd /mnt/NV2/Development/major-domo/database

# Apply migration
docker-compose exec postgres psql -U sba_admin -d sba_master -f /migrations/migration_XXX.sql

# Or copy to container and run
docker cp migrations/migration_XXX.sql major-domo-postgres:/tmp/
docker-compose exec postgres psql -U sba_admin -d sba_master -f /tmp/migration_XXX.sql

Via psql directly:

psql -h 10.10.0.42 -p 5433 -U sba_admin -d sba_master -f migrations/migration_XXX.sql

3. Verify Changes

-- Connect to database
psql -h 10.10.0.42 -p 5433 -U sba_admin -d sba_master

-- Check table structure
\d table_name

-- Verify data
SELECT * FROM table_name LIMIT 5;

-- Check indexes
\di table_name*

-- Verify constraints
\d+ table_name

4. Test API Endpoints

# Test affected endpoints
curl -H "Authorization: Bearer $API_TOKEN" \
  http://10.10.0.42:8000/api/v3/affected-endpoint

# Or use API client
cd ~/.claude/skills/major-domo
python api_client.py --env dev --verbose

5. Test Bot Commands

cd /mnt/NV2/Development/major-domo/discord-app-v2
python bot.py

# Test commands that use affected tables

Production Migration

1. Backup Production Database

Create Backup:

# SSH to production server
ssh akamai

# Create backup with timestamp
pg_dump -h 10.10.0.42 -U sba_admin -d sba_master > \
  ~/backups/sba_master_$(date +%Y%m%d_%H%M%S).sql

# Or via Docker
docker-compose exec postgres pg_dump -U sba_admin sba_master > \
  backups/sba_master_$(date +%Y%m%d_%H%M%S).sql

Verify Backup:

ls -lh ~/backups/sba_master_*.sql
head -n 20 ~/backups/sba_master_*.sql

2. Assess Downtime Requirements

No Downtime Needed:

• Adding nullable columns
• Creating indexes (CONCURRENTLY)
• Adding tables not yet used

Downtime Required:

• Altering column types
• Adding NOT NULL constraints
• Dropping columns/tables
• Major data transformations

If Downtime Needed:

# Announce in Discord
"⚠️ Database maintenance scheduled for [TIME].
Expected downtime: [DURATION]. Bot will be offline."

3. Stop API/Bot (if downtime required)

Stop Discord Bot:

ssh akamai
screen -r major-domo
# Ctrl+C to stop bot

Stop API (if needed):

cd ~/major-domo/database
docker-compose stop app

4. Apply Migration to Production

Via psql:

psql -h 10.10.0.42 -U sba_admin -d sba_master -f migrations/migration_XXX.sql

Monitor Output:

• Watch for errors
• Note any warnings
• Verify completion message

5. Verify Production Changes

-- Connect to production database
psql -h 10.10.0.42 -U sba_admin -d sba_master

-- Check table structure
\d table_name

-- Verify data integrity
SELECT COUNT(*) FROM table_name;
SELECT * FROM table_name WHERE new_field IS NULL LIMIT 10;

-- Check constraints
SELECT conname, contype, conrelid::regclass
FROM pg_constraint
WHERE conrelid = 'table_name'::regclass;

6. Update Code (if needed)

Update ORM Models (database/app/db_engine.py):

class Player(BaseModel):
    # ... existing fields ...
    new_field = CharField(null=True, max_length=100)

    class Meta:
        table_name = 'players'

Deploy Code Changes:

# Pull latest code
cd ~/major-domo/database
git pull origin main

# Restart API
docker-compose restart app

7. Restart Services

Restart API:

docker-compose start app

# Verify API is running
curl http://10.10.0.42/api/v3/current

Restart Bot:

screen -r major-domo
python bot.py
# Ctrl+A then D to detach

8. Post-Migration Verification

Test API Endpoints:

# Test affected endpoints
curl -H "Authorization: Bearer $API_TOKEN" \
  http://10.10.0.42/api/v3/players?season=12&limit=5

# Test with API client
cd ~/.claude/skills/major-domo
python api_client.py --env prod --verbose

Test Bot Commands:

• Verify commands work
• Check for error messages
• Test edge cases

Monitor Logs:

# API logs
docker-compose logs -f app

# Bot logs
tail -f ~/major-domo/discord-app-v2/logs/discord.log

Rollback Procedure

If Migration Fails

Immediate Rollback:

-- Connect to database
psql -h 10.10.0.42 -U sba_admin -d sba_master

-- Start transaction
BEGIN;

-- Rollback migration (reverse operations)
ALTER TABLE players DROP COLUMN new_field;

-- Commit rollback
COMMIT;

Restore from Backup

If Critical Failure:

# Stop services
docker-compose stop app

# Restore database from backup
psql -h 10.10.0.42 -U sba_admin -d sba_master < \
  ~/backups/sba_master_YYYYMMDD_HHMMSS.sql

# Restart services
docker-compose start app

Common Migration Patterns

Adding a Column

-- Add nullable column first
ALTER TABLE players ADD COLUMN new_field VARCHAR(100);

-- Optionally populate data
UPDATE players SET new_field = 'default_value' WHERE condition;

-- Later, make NOT NULL if needed (separate migration)
-- ALTER TABLE players ALTER COLUMN new_field SET NOT NULL;

Creating an Index

-- Use CONCURRENTLY to avoid locking (no downtime)
CREATE INDEX CONCURRENTLY idx_players_new_field
ON players(new_field);

Renaming a Column

-- Rename column
ALTER TABLE players RENAME COLUMN old_name TO new_name;

-- Note: Update ORM models and code immediately

Adding a Foreign Key

-- Add column
ALTER TABLE players ADD COLUMN league_id INTEGER;

-- Add foreign key constraint
ALTER TABLE players
ADD CONSTRAINT fk_players_league
FOREIGN KEY (league_id) REFERENCES leagues(id);

Data Transformation

-- Complex data migrations in transaction
BEGIN;

-- Create temporary column
ALTER TABLE players ADD COLUMN temp_field TEXT;

-- Transform data
UPDATE players
SET temp_field = CONCAT(first_name, ' ', last_name);

-- Drop old columns
ALTER TABLE players DROP COLUMN first_name;
ALTER TABLE players DROP COLUMN last_name;

-- Rename temp column
ALTER TABLE players RENAME COLUMN temp_field TO full_name;

COMMIT;

Best Practices

1. Always Test on Dev First: Never run untested migrations on production
2. Use Transactions: Wrap migrations in BEGIN/COMMIT blocks
3. Backup Before Migration: Always have a recent backup
4. Small, Incremental Changes: Break large migrations into smaller steps
5. Document Changes: Add comments explaining why and what
6. Version Control: Commit migration files to git
7. Test Rollback: Verify rollback SQL works on dev
8. Monitor Performance: Check query performance after schema changes
9. Update Documentation: Reflect schema changes in docs

Migration Checklist Template

Migration: [Brief Description]
Date: [YYYY-MM-DD]
Developer: [Your Name]

Pre-Migration:
[ ] Migration file created
[ ] Tested on dev database
[ ] API tested with changes
[ ] Bot tested with changes
[ ] Backup created
[ ] Rollback plan prepared
[ ] Downtime assessment complete
[ ] Users notified (if needed)

Migration:
[ ] Services stopped (if needed)
[ ] Migration applied to production
[ ] Changes verified in database
[ ] ORM models updated
[ ] Code deployed
[ ] Services restarted

Post-Migration:
[ ] API endpoints tested
[ ] Bot commands tested
[ ] Logs monitored
[ ] No errors detected
[ ] Performance verified
[ ] Documentation updated

Rollback (if needed):
[ ] Rollback SQL prepared
[ ] Backup restoration tested
[ ] Services can be restored

Environment-Specific Notes

Development Database

• Host: 10.10.0.42
• Port: 5433 (Docker)
• Database: sba_master
• User: sba_admin
• Docker Container: major-domo-postgres

Production Database

• Host: 10.10.0.42
• Port: 5432
• Database: sba_master
• User: sba_admin
• Access: Via SSH to Akamai Linode

Troubleshooting

Permission Errors

-- Grant permissions to user
GRANT ALL PRIVILEGES ON TABLE table_name TO sba_admin;

Lock Errors

-- Check for locks
SELECT * FROM pg_locks WHERE NOT granted;

-- Find blocking queries
SELECT pid, query FROM pg_stat_activity WHERE state != 'idle';

Connection Issues

# Check PostgreSQL is running
docker-compose ps

# Check logs
docker-compose logs postgres

---

Last Updated: 2025-11-10 Maintainer: Cal Corum