From b063b73f92b4bf7dd903dd91e0e8bd74dd3491a4 Mon Sep 17 00:00:00 2001 From: Cal Corum Date: Tue, 27 Jan 2026 15:27:18 -0600 Subject: [PATCH] Address documentation gaps from review - Added Prerequisites section with software/infrastructure requirements - Added pre-migration steps: stop API, password file creation - Added PostgreSQL 15+ schema permission grant - Added post-migration VACUUM ANALYZE step - Added Troubleshooting section with common errors and solutions - Added verification queries for debugging - Added expected record counts table - Added connection string for external tools - Expanded checklist with pre/post sections - Added monitoring guidance - Clarified CRITICAL warnings about data consistency --- docs/POSTGRES_MIGRATION_GUIDE.md | 285 ++++++++++++++++++++++++------- 1 file changed, 226 insertions(+), 59 deletions(-) diff --git a/docs/POSTGRES_MIGRATION_GUIDE.md b/docs/POSTGRES_MIGRATION_GUIDE.md index 113fe6e..5476488 100644 --- a/docs/POSTGRES_MIGRATION_GUIDE.md +++ b/docs/POSTGRES_MIGRATION_GUIDE.md @@ -5,11 +5,32 @@ This document captures lessons learned from test migrations and provides a step-by-step guide for production deployment. **Migration Branch:** `postgres-migration` -**Target:** PostgreSQL 17 on `sba_postgres` container +**Target:** PostgreSQL 17 (compatible with PostgreSQL 14+) **Source:** SQLite `storage/pd_master.db` --- +## Prerequisites + +### Software Requirements + +| Component | Version | Notes | +|-----------|---------|-------| +| Python | 3.8+ | Required for type hints, f-strings | +| PostgreSQL | 14+ | Tested on PostgreSQL 17 | +| Peewee | 3.15+ | PostgreSQL connection pooling | +| psycopg2-binary | 2.9+ | PostgreSQL driver | +| Docker | 20+ | Container runtime | + +### Infrastructure Requirements + +- PostgreSQL container running on `sba_postgres` +- Docker network `dev-sba-database_default` exists +- ~500MB disk space for migration (2x SQLite size temporarily) +- 30 minute maintenance window + +--- + ## Final Test Results ### Migration Summary (January 27, 2026) @@ -20,7 +41,18 @@ This document captures lessons learned from test migrations and provides a step- | **Records Inserted** | 549,788 | | **Records Skipped** | 168,463 (orphaned FK) | | **Duration** | ~21 minutes | -| **gamerewards** | 10/10 ✅ | +| **gamerewards** | 10/10 | + +### Expected Record Counts (Approximate) + +| Table | Expected Count | +|-------|---------------| +| player | ~13,400 | +| team | ~105 | +| card | ~33,400 | +| gamerewards | 10 | +| stratplay | ~14,000 | +| stratgame | ~3,700 | ### Tables with Orphaned Data (Expected) @@ -35,23 +67,23 @@ These tables have records that reference deleted teams/games - PostgreSQL correc | stratplay | 418,523 | 13,963 | 404,560 | Deleted games | | decision | 36,900 | 24,551 | 12,349 | Deleted games | -### All Tested Endpoints ✅ +### All Tested Endpoints | Endpoint | Status | |----------|--------| -| `/api/v2/teams` | ✅ 200 | -| `/api/v2/players` | ✅ 200 | -| `/api/v2/cards` | ✅ 200 | -| `/api/v2/cardsets` | ✅ 200 | -| `/api/v2/games` | ✅ 200 | -| `/api/v2/decisions` | ✅ 200 | -| `/api/v2/decisions/rest` | ✅ 200 | -| `/api/v2/current` | ✅ 200 | -| `/api/v2/gamerewards` | ✅ 200 | -| `/api/v2/plays` | ✅ 200 | -| `/api/v2/plays/batting?group_by=player` | ✅ 200 | -| `/api/v2/plays/pitching?group_by=player` | ✅ 200 | -| `/api/v2/plays/game-summary/{id}` | ✅ 200 | +| `/api/v2/teams` | 200 | +| `/api/v2/players` | 200 | +| `/api/v2/cards` | 200 | +| `/api/v2/cardsets` | 200 | +| `/api/v2/games` | 200 | +| `/api/v2/decisions` | 200 | +| `/api/v2/decisions/rest` | 200 | +| `/api/v2/current` | 200 | +| `/api/v2/gamerewards` | 200 | +| `/api/v2/plays` | 200 | +| `/api/v2/plays/batting?group_by=player` | 200 | +| `/api/v2/plays/pitching?group_by=player` | 200 | +| `/api/v2/plays/game-summary/{id}` | 200 | --- @@ -129,23 +161,27 @@ MIGRATION_ORDER = [ ## Production Migration Plan -### Prerequisites - -1. **PostgreSQL Server Running** - - Container: `sba_postgres` (PostgreSQL 17) - - Network: `dev-sba-database_default` - -2. **Docker Image Built and Pushed** - ```bash - docker build -t manticorum67/paper-dynasty-database:postgres-migration . - docker push manticorum67/paper-dynasty-database:postgres-migration - ``` - -### Step 1: Create Database and User +### Step 1: Pre-Migration Preparation ```bash ssh sba-db +# Verify PostgreSQL is running +docker ps | grep sba_postgres + +# Verify network exists +docker network ls | grep sba-database + +# Create password file (avoid shell history) +cat > /tmp/.pg_credentials << 'EOF' +POSTGRES_PASSWORD=YOUR_SECURE_PASSWORD +EOF +chmod 600 /tmp/.pg_credentials +``` + +### Step 2: Create Database and User + +```bash # Create user docker exec sba_postgres psql -U sba_admin -d postgres -c \ "CREATE USER pd_admin WITH PASSWORD 'YOUR_SECURE_PASSWORD';" @@ -156,23 +192,27 @@ docker exec sba_postgres psql -U sba_admin -d postgres -c \ docker exec sba_postgres psql -U sba_admin -d postgres -c \ "GRANT ALL PRIVILEGES ON DATABASE pd_master TO pd_admin;" + +# PostgreSQL 15+ requires explicit schema grant +docker exec sba_postgres psql -U sba_admin -d pd_master -c \ + "GRANT ALL ON SCHEMA public TO pd_admin;" ``` -### Step 2: Create Logs Directory +### Step 3: Create Logs Directory ```bash -mkdir -p /path/to/logs/database -chmod 777 /path/to/logs /path/to/logs/database +mkdir -p /home/cal/container-data/dev-sba-database/logs/database +chmod 777 /home/cal/container-data/dev-sba-database/logs /home/cal/container-data/dev-sba-database/logs/database ``` -### Step 3: Create Schema +### Step 4: Create Schema ```bash docker pull manticorum67/paper-dynasty-database:postgres-migration docker run --rm \ --network dev-sba-database_default \ - -v /path/to/logs:/usr/src/app/logs \ + -v /home/cal/container-data/dev-sba-database/logs:/usr/src/app/logs \ -e DATABASE_TYPE=postgresql \ -e POSTGRES_HOST=sba_postgres \ -e POSTGRES_DB=pd_master \ @@ -186,24 +226,43 @@ db.create_tables([Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, Pack print('Tables created successfully') db.close() " + +# Verify tables created +docker exec sba_postgres psql -U pd_admin -d pd_master -c \ + "SELECT COUNT(*) as table_count FROM pg_tables WHERE schemaname='public';" +# Expected: 29 tables ``` -### Step 4: Copy Production SQLite +### Step 5: Stop Production API and Copy SQLite + +**CRITICAL: Stop the API to ensure data consistency during copy.** ```bash -# Copy from production container -docker cp pd_database_v2:/usr/src/app/storage/pd_master.db /path/to/storage/pd_master.db +# Stop production API +docker stop pd_database_v2 + +# Wait for writes to complete +sleep 5 + +# Copy database +docker cp pd_database_v2:/usr/src/app/storage/pd_master.db \ + /home/cal/container-data/dev-sba-database/storage/pd_master_migration.db + +# Verify copy integrity +ls -la /home/cal/container-data/dev-sba-database/storage/pd_master_migration.db ``` -### Step 5: Run Migration +### Step 6: Run Migration + +**WARNING: Do NOT restart the API until migration is complete and verified.** ```bash -# Run in background (takes ~21 minutes) +# Run migration (takes ~21 minutes) nohup docker run --rm \ --network dev-sba-database_default \ - -v /path/to/storage:/usr/src/app/storage \ - -v /path/to/logs:/usr/src/app/logs \ - -v /path/to/scripts:/usr/src/app/scripts \ + -v /home/cal/container-data/dev-sba-database/storage:/usr/src/app/storage \ + -v /home/cal/container-data/dev-sba-database/logs:/usr/src/app/logs \ + -v /home/cal/container-data/dev-sba-database/scripts:/usr/src/app/scripts \ -e DATABASE_TYPE=postgresql \ -e POSTGRES_HOST=sba_postgres \ -e POSTGRES_DB=pd_master \ @@ -211,13 +270,13 @@ nohup docker run --rm \ -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \ -e POSTGRES_PORT=5432 \ manticorum67/paper-dynasty-database:postgres-migration \ - python scripts/migrate_to_postgres.py --sqlite-path storage/pd_master.db > /tmp/migration.log 2>&1 & + python scripts/migrate_to_postgres.py --sqlite-path storage/pd_master_migration.db > /tmp/migration.log 2>&1 & # Monitor progress tail -f /tmp/migration.log ``` -### Step 6: Verify Migration +### Step 7: Verify Migration ```bash # Check summary @@ -232,9 +291,29 @@ UNION ALL SELECT 'gamerewards', COUNT(*) FROM gamerewards UNION ALL SELECT 'stratplay', COUNT(*) FROM stratplay ORDER BY tbl; " + +# Verify gamerewards (should be 10) +docker exec sba_postgres psql -U pd_admin -d pd_master -c \ + "SELECT id, name FROM gamerewards ORDER BY id;" + +# Verify data integrity +docker exec sba_postgres psql -U pd_admin -d pd_master -c " +SELECT MIN(created), MAX(created) FROM stratgame WHERE created IS NOT NULL; +" ``` -### Step 7: Start API Container +### Step 8: Post-Migration Optimization + +```bash +# Analyze tables for query optimization +docker exec sba_postgres psql -U pd_admin -d pd_master -c "VACUUM ANALYZE;" + +# Verify connection health +docker exec sba_postgres psql -U pd_admin -d pd_master -c \ + "SELECT version(), current_database(), current_user;" +``` + +### Step 9: Start API Container ```bash docker run -d \ @@ -247,11 +326,17 @@ docker run -d \ -e POSTGRES_USER=pd_admin \ -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \ -e POSTGRES_PORT=5432 \ - -v /path/to/logs:/usr/src/app/logs \ + -v /home/cal/container-data/dev-sba-database/logs:/usr/src/app/logs \ manticorum67/paper-dynasty-database:postgres-migration + +# Wait for startup +sleep 10 + +# Check logs +docker logs pd_postgres_api 2>&1 | tail -20 ``` -### Step 8: Test Endpoints +### Step 10: Test Endpoints ```bash # Basic endpoints @@ -261,56 +346,122 @@ for ep in "teams?limit=1" "players?limit=1" "cards?limit=1" "games?limit=1" "cur echo done -# GROUP BY endpoints (critical) +# GROUP BY endpoints (critical - these had PostgreSQL-specific fixes) +echo "Testing GROUP BY endpoints..." curl -s "http://localhost:8100/api/v2/plays/batting?season=10&group_by=player&limit=1" | head -c 200 +echo curl -s "http://localhost:8100/api/v2/plays/pitching?season=10&group_by=player&limit=1" | head -c 200 +echo +curl -s "http://localhost:8100/api/v2/decisions/rest?team_id=2&season=10&limit=3" ``` --- ## Rollback Plan -To switch back to SQLite: +If issues are discovered after migration: ```bash # Stop PostgreSQL API docker stop pd_postgres_api +docker rm pd_postgres_api -# Start SQLite API (original image) +# Restart SQLite API (original image) +docker start pd_database_v2 + +# Or start fresh SQLite container docker run -d \ --name pd_sqlite_api \ -p 8100:80 \ - -v /path/to/storage:/usr/src/app/storage \ + -v /home/cal/container-data/dev-sba-database/storage:/usr/src/app/storage \ manticorum67/paper-dynasty-database:latest ``` +The original SQLite database is preserved and unmodified. + +--- + +## Troubleshooting + +### Common Errors + +| Error | Cause | Solution | +|-------|-------|----------| +| `permission denied for schema public` | PostgreSQL 15+ changed permissions | `GRANT ALL ON SCHEMA public TO pd_admin;` | +| `connection refused` | Container not on same network | Verify `docker network ls` shows both containers | +| `relation does not exist` | Schema not created | Re-run Step 4 (Create Schema) | +| `sequence does not exist` | Sequence naming mismatch | Check `player_player_id_seq` vs `player_id_seq` | +| FK violation during migration | Parent record deleted | Expected - see "Tables with Orphaned Data" | + +### Verification Queries + +```sql +-- Check table counts +SELECT schemaname, relname, n_live_tup +FROM pg_stat_user_tables +ORDER BY relname; + +-- Check active connections +SELECT pid, usename, application_name, state +FROM pg_stat_activity +WHERE datname = 'pd_master'; + +-- Check for slow queries (if pg_stat_statements enabled) +SELECT query, calls, mean_exec_time +FROM pg_stat_statements +ORDER BY mean_exec_time DESC +LIMIT 10; +``` + +### Connection String + +For debugging with external tools: +``` +postgresql://pd_admin:PASSWORD@sba_postgres:5432/pd_master +``` + --- ## Production Deployment Checklist +### Pre-Migration - [ ] Backup production SQLite database - [ ] Schedule maintenance window (~30 minutes) +- [ ] Notify users of downtime +- [ ] Verify PostgreSQL container is healthy +- [ ] Verify Docker network exists + +### Migration - [ ] Create PostgreSQL database and user -- [ ] Create schema -- [ ] Copy fresh SQLite from production +- [ ] Grant schema permissions (PostgreSQL 15+) +- [ ] Create schema (29 tables) +- [ ] **Stop production API** +- [ ] Copy SQLite database - [ ] Run migration script - [ ] Verify migration summary (21/27 tables, ~550K records) -- [ ] Verify gamerewards has 10 records +- [ ] Verify gamerewards count matches SQLite +- [ ] Run VACUUM ANALYZE + +### Post-Migration - [ ] Start PostgreSQL API container -- [ ] Test all endpoints (especially GROUP BY endpoints) +- [ ] Test all basic endpoints (200 response) +- [ ] Test GROUP BY endpoints (batting, pitching, decisions/rest) +- [ ] Verify game-summary endpoint - [ ] Update DNS/load balancer to point to new container -- [ ] Monitor for 24 hours -- [ ] Remove old SQLite container +- [ ] Monitor logs for 1 hour +- [ ] Monitor for 24 hours before removing SQLite backup --- ## Known Limitations -1. **Orphaned historical data not migrated** - Records from deleted teams/games are correctly rejected by PostgreSQL FK constraints. This is expected and doesn't affect current gameplay. +1. **Orphaned historical data not migrated** - Records from deleted teams/games are correctly rejected by PostgreSQL FK constraints. This is expected and doesn't affect current gameplay. The original SQLite backup preserves this data if ever needed. 2. **Legacy tables excluded** - `battingstat` and `pitchingstat` are not migrated (legacy, unused by current app). -3. **Roster table mostly empty** - Most roster records reference deleted cards. Current rosters work correctly. +3. **Roster table mostly empty** - Most roster records reference deleted cards. Current active rosters work correctly. + +4. **Skipped records are permanent** - The 168,463 skipped records will not exist in PostgreSQL. Keep SQLite backup indefinitely. --- @@ -323,3 +474,19 @@ Database: pd_master User: pd_admin Network: dev-sba-database_default ``` + +--- + +## Monitoring (Post-Deployment) + +### Key Metrics to Watch + +1. **API Response Times** - Should be comparable or faster than SQLite +2. **Connection Pool Usage** - `max_connections=20` in db_engine.py +3. **Disk Usage** - PostgreSQL data directory growth +4. **Error Logs** - Check for FK violations, connection errors + +### Log Locations + +- API logs: `/usr/src/app/logs/` (mounted volume) +- PostgreSQL logs: `docker logs sba_postgres`