cal/paper-dynasty-database
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Address documentation gaps from review

Browse Source 
- Added Prerequisites section with software/infrastructure requirements
- Added pre-migration steps: stop API, password file creation
- Added PostgreSQL 15+ schema permission grant
- Added post-migration VACUUM ANALYZE step
- Added Troubleshooting section with common errors and solutions
- Added verification queries for debugging
- Added expected record counts table
- Added connection string for external tools
- Expanded checklist with pre/post sections
- Added monitoring guidance
- Clarified CRITICAL warnings about data consistency
This commit is contained in:
Cal Corum 2026-01-27 15:27:18 -06:00
parent ea5c047b15
commit b063b73f92
1 changed files with 226 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

285
docs/POSTGRES_MIGRATION_GUIDE.md
View File

@ -5,11 +5,32 @@
This document captures lessons learned from test migrations and provides a step-by-step guide for production deployment. This document captures lessons learned from test migrations and provides a step-by-step guide for production deployment.
**Migration Branch:** `postgres-migration` **Migration Branch:** `postgres-migration`
**Target:** PostgreSQL 17 on `sba_postgres` container **Target:** PostgreSQL 17 (compatible with PostgreSQL 14+)
**Source:** SQLite `storage/pd_master.db` **Source:** SQLite `storage/pd_master.db`
--- ---
## Prerequisites
### Software Requirements
| Component | Version | Notes |
|-----------|---------|-------|
| Python | 3.8+ | Required for type hints, f-strings |
| PostgreSQL | 14+ | Tested on PostgreSQL 17 |
| Peewee | 3.15+ | PostgreSQL connection pooling |
| psycopg2-binary | 2.9+ | PostgreSQL driver |
| Docker | 20+ | Container runtime |
### Infrastructure Requirements
- PostgreSQL container running on `sba_postgres`
- Docker network `dev-sba-database_default` exists
- ~500MB disk space for migration (2x SQLite size temporarily)
- 30 minute maintenance window
---
## Final Test Results ## Final Test Results
### Migration Summary (January 27, 2026) ### Migration Summary (January 27, 2026)
@ -20,7 +41,18 @@ This document captures lessons learned from test migrations and provides a step-
| **Records Inserted** | 549,788 | | **Records Inserted** | 549,788 |
| **Records Skipped** | 168,463 (orphaned FK) | | **Records Skipped** | 168,463 (orphaned FK) |
| **Duration** | ~21 minutes | | **Duration** | ~21 minutes |
| **gamerewards** | 10/10 ✅ | | **gamerewards** | 10/10 |
### Expected Record Counts (Approximate)
| Table | Expected Count |
|-------|---------------|
| player | ~13,400 |
| team | ~105 |
| card | ~33,400 |
| gamerewards | 10 |
| stratplay | ~14,000 |
| stratgame | ~3,700 |
### Tables with Orphaned Data (Expected) ### Tables with Orphaned Data (Expected)
@ -35,23 +67,23 @@ These tables have records that reference deleted teams/games - PostgreSQL correc
| stratplay | 418,523 | 13,963 | 404,560 | Deleted games | | stratplay | 418,523 | 13,963 | 404,560 | Deleted games |
| decision | 36,900 | 24,551 | 12,349 | Deleted games | | decision | 36,900 | 24,551 | 12,349 | Deleted games |
### All Tested Endpoints ✅ ### All Tested Endpoints
| Endpoint | Status | | Endpoint | Status |
|----------|--------| |----------|--------|
| `/api/v2/teams` | ✅ 200 | | `/api/v2/teams` | 200 |
| `/api/v2/players` | ✅ 200 | | `/api/v2/players` | 200 |
| `/api/v2/cards` | ✅ 200 | | `/api/v2/cards` | 200 |
| `/api/v2/cardsets` | ✅ 200 | | `/api/v2/cardsets` | 200 |
| `/api/v2/games` | ✅ 200 | | `/api/v2/games` | 200 |
| `/api/v2/decisions` | ✅ 200 | | `/api/v2/decisions` | 200 |
| `/api/v2/decisions/rest` | ✅ 200 | | `/api/v2/decisions/rest` | 200 |
| `/api/v2/current` | ✅ 200 | | `/api/v2/current` | 200 |
| `/api/v2/gamerewards` | ✅ 200 | | `/api/v2/gamerewards` | 200 |
| `/api/v2/plays` | ✅ 200 | | `/api/v2/plays` | 200 |
| `/api/v2/plays/batting?group_by=player` | ✅ 200 | | `/api/v2/plays/batting?group_by=player` | 200 |
| `/api/v2/plays/pitching?group_by=player` | ✅ 200 | | `/api/v2/plays/pitching?group_by=player` | 200 |
| `/api/v2/plays/game-summary/{id}` | ✅ 200 | | `/api/v2/plays/game-summary/{id}` | 200 |
--- ---
@ -129,23 +161,27 @@ MIGRATION_ORDER = [
## Production Migration Plan ## Production Migration Plan
### Prerequisites ### Step 1: Pre-Migration Preparation
1. **PostgreSQL Server Running**
- Container: `sba_postgres` (PostgreSQL 17)
- Network: `dev-sba-database_default`
2. **Docker Image Built and Pushed**
```bash
docker build -t manticorum67/paper-dynasty-database:postgres-migration .
docker push manticorum67/paper-dynasty-database:postgres-migration
```
### Step 1: Create Database and User
```bash ```bash
ssh sba-db ssh sba-db
# Verify PostgreSQL is running
docker ps | grep sba_postgres
# Verify network exists
docker network ls | grep sba-database
# Create password file (avoid shell history)
cat > /tmp/.pg_credentials << 'EOF'
POSTGRES_PASSWORD=YOUR_SECURE_PASSWORD
EOF
chmod 600 /tmp/.pg_credentials
```
### Step 2: Create Database and User
```bash
# Create user # Create user
docker exec sba_postgres psql -U sba_admin -d postgres -c \ docker exec sba_postgres psql -U sba_admin -d postgres -c \
"CREATE USER pd_admin WITH PASSWORD 'YOUR_SECURE_PASSWORD';" "CREATE USER pd_admin WITH PASSWORD 'YOUR_SECURE_PASSWORD';"
@ -156,23 +192,27 @@ docker exec sba_postgres psql -U sba_admin -d postgres -c \
docker exec sba_postgres psql -U sba_admin -d postgres -c \ docker exec sba_postgres psql -U sba_admin -d postgres -c \
"GRANT ALL PRIVILEGES ON DATABASE pd_master TO pd_admin;" "GRANT ALL PRIVILEGES ON DATABASE pd_master TO pd_admin;"
# PostgreSQL 15+ requires explicit schema grant
docker exec sba_postgres psql -U sba_admin -d pd_master -c \
"GRANT ALL ON SCHEMA public TO pd_admin;"
``` ```
### Step 2: Create Logs Directory ### Step 3: Create Logs Directory
```bash ```bash
mkdir -p /path/to/logs/database mkdir -p /home/cal/container-data/dev-sba-database/logs/database
chmod 777 /path/to/logs /path/to/logs/database chmod 777 /home/cal/container-data/dev-sba-database/logs /home/cal/container-data/dev-sba-database/logs/database
``` ```
### Step 3: Create Schema ### Step 4: Create Schema
```bash ```bash
docker pull manticorum67/paper-dynasty-database:postgres-migration docker pull manticorum67/paper-dynasty-database:postgres-migration
docker run --rm \ docker run --rm \
--network dev-sba-database_default \ --network dev-sba-database_default \
-v /path/to/logs:/usr/src/app/logs \ -v /home/cal/container-data/dev-sba-database/logs:/usr/src/app/logs \
-e DATABASE_TYPE=postgresql \ -e DATABASE_TYPE=postgresql \
-e POSTGRES_HOST=sba_postgres \ -e POSTGRES_HOST=sba_postgres \
-e POSTGRES_DB=pd_master \ -e POSTGRES_DB=pd_master \
@ -186,24 +226,43 @@ db.create_tables([Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, Pack
print('Tables created successfully') print('Tables created successfully')
db.close() db.close()
" "
# Verify tables created
docker exec sba_postgres psql -U pd_admin -d pd_master -c \
"SELECT COUNT(*) as table_count FROM pg_tables WHERE schemaname='public';"
# Expected: 29 tables
``` ```
### Step 4: Copy Production SQLite ### Step 5: Stop Production API and Copy SQLite
**CRITICAL: Stop the API to ensure data consistency during copy.**
```bash ```bash
# Copy from production container # Stop production API
docker cp pd_database_v2:/usr/src/app/storage/pd_master.db /path/to/storage/pd_master.db docker stop pd_database_v2
# Wait for writes to complete
sleep 5
# Copy database
docker cp pd_database_v2:/usr/src/app/storage/pd_master.db \
/home/cal/container-data/dev-sba-database/storage/pd_master_migration.db
# Verify copy integrity
ls -la /home/cal/container-data/dev-sba-database/storage/pd_master_migration.db
``` ```
### Step 5: Run Migration ### Step 6: Run Migration
**WARNING: Do NOT restart the API until migration is complete and verified.**
```bash ```bash
# Run in background (takes ~21 minutes) # Run migration (takes ~21 minutes)
nohup docker run --rm \ nohup docker run --rm \
--network dev-sba-database_default \ --network dev-sba-database_default \
-v /path/to/storage:/usr/src/app/storage \ -v /home/cal/container-data/dev-sba-database/storage:/usr/src/app/storage \
-v /path/to/logs:/usr/src/app/logs \ -v /home/cal/container-data/dev-sba-database/logs:/usr/src/app/logs \
-v /path/to/scripts:/usr/src/app/scripts \ -v /home/cal/container-data/dev-sba-database/scripts:/usr/src/app/scripts \
-e DATABASE_TYPE=postgresql \ -e DATABASE_TYPE=postgresql \
-e POSTGRES_HOST=sba_postgres \ -e POSTGRES_HOST=sba_postgres \
-e POSTGRES_DB=pd_master \ -e POSTGRES_DB=pd_master \
@ -211,13 +270,13 @@ nohup docker run --rm \
-e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \ -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
-e POSTGRES_PORT=5432 \ -e POSTGRES_PORT=5432 \
manticorum67/paper-dynasty-database:postgres-migration \ manticorum67/paper-dynasty-database:postgres-migration \
python scripts/migrate_to_postgres.py --sqlite-path storage/pd_master.db > /tmp/migration.log 2>&1 & python scripts/migrate_to_postgres.py --sqlite-path storage/pd_master_migration.db > /tmp/migration.log 2>&1 &
# Monitor progress # Monitor progress
tail -f /tmp/migration.log tail -f /tmp/migration.log
``` ```
### Step 6: Verify Migration ### Step 7: Verify Migration
```bash ```bash
# Check summary # Check summary
@ -232,9 +291,29 @@ UNION ALL SELECT 'gamerewards', COUNT(*) FROM gamerewards
UNION ALL SELECT 'stratplay', COUNT(*) FROM stratplay UNION ALL SELECT 'stratplay', COUNT(*) FROM stratplay
ORDER BY tbl; ORDER BY tbl;
" "
# Verify gamerewards (should be 10)
docker exec sba_postgres psql -U pd_admin -d pd_master -c \
"SELECT id, name FROM gamerewards ORDER BY id;"
# Verify data integrity
docker exec sba_postgres psql -U pd_admin -d pd_master -c "
SELECT MIN(created), MAX(created) FROM stratgame WHERE created IS NOT NULL;
"
``` ```
### Step 7: Start API Container ### Step 8: Post-Migration Optimization
```bash
# Analyze tables for query optimization
docker exec sba_postgres psql -U pd_admin -d pd_master -c "VACUUM ANALYZE;"
# Verify connection health
docker exec sba_postgres psql -U pd_admin -d pd_master -c \
"SELECT version(), current_database(), current_user;"
```
### Step 9: Start API Container
```bash ```bash
docker run -d \ docker run -d \
@ -247,11 +326,17 @@ docker run -d \
-e POSTGRES_USER=pd_admin \ -e POSTGRES_USER=pd_admin \
-e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \ -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
-e POSTGRES_PORT=5432 \ -e POSTGRES_PORT=5432 \
-v /path/to/logs:/usr/src/app/logs \ -v /home/cal/container-data/dev-sba-database/logs:/usr/src/app/logs \
manticorum67/paper-dynasty-database:postgres-migration manticorum67/paper-dynasty-database:postgres-migration
# Wait for startup
sleep 10
# Check logs
docker logs pd_postgres_api 2>&1 | tail -20
``` ```
### Step 8: Test Endpoints ### Step 10: Test Endpoints
```bash ```bash
# Basic endpoints # Basic endpoints
@ -261,56 +346,122 @@ for ep in "teams?limit=1" "players?limit=1" "cards?limit=1" "games?limit=1" "cur
echo echo
done done
# GROUP BY endpoints (critical) # GROUP BY endpoints (critical - these had PostgreSQL-specific fixes)
echo "Testing GROUP BY endpoints..."
curl -s "http://localhost:8100/api/v2/plays/batting?season=10&group_by=player&limit=1" | head -c 200 curl -s "http://localhost:8100/api/v2/plays/batting?season=10&group_by=player&limit=1" | head -c 200
echo
curl -s "http://localhost:8100/api/v2/plays/pitching?season=10&group_by=player&limit=1" | head -c 200 curl -s "http://localhost:8100/api/v2/plays/pitching?season=10&group_by=player&limit=1" | head -c 200
echo
curl -s "http://localhost:8100/api/v2/decisions/rest?team_id=2&season=10&limit=3"
``` ```
--- ---
## Rollback Plan ## Rollback Plan
To switch back to SQLite: If issues are discovered after migration:
```bash ```bash
# Stop PostgreSQL API # Stop PostgreSQL API
docker stop pd_postgres_api docker stop pd_postgres_api
docker rm pd_postgres_api
# Start SQLite API (original image) # Restart SQLite API (original image)
docker start pd_database_v2
# Or start fresh SQLite container
docker run -d \ docker run -d \
--name pd_sqlite_api \ --name pd_sqlite_api \
-p 8100:80 \ -p 8100:80 \
-v /path/to/storage:/usr/src/app/storage \ -v /home/cal/container-data/dev-sba-database/storage:/usr/src/app/storage \
manticorum67/paper-dynasty-database:latest manticorum67/paper-dynasty-database:latest
``` ```
The original SQLite database is preserved and unmodified.
---
## Troubleshooting
### Common Errors
| Error | Cause | Solution |
|-------|-------|----------|
| `permission denied for schema public` | PostgreSQL 15+ changed permissions | `GRANT ALL ON SCHEMA public TO pd_admin;` |
| `connection refused` | Container not on same network | Verify `docker network ls` shows both containers |
| `relation does not exist` | Schema not created | Re-run Step 4 (Create Schema) |
| `sequence does not exist` | Sequence naming mismatch | Check `player_player_id_seq` vs `player_id_seq` |
| FK violation during migration | Parent record deleted | Expected - see "Tables with Orphaned Data" |
### Verification Queries
```sql
-- Check table counts
SELECT schemaname, relname, n_live_tup
FROM pg_stat_user_tables
ORDER BY relname;
-- Check active connections
SELECT pid, usename, application_name, state
FROM pg_stat_activity
WHERE datname = 'pd_master';
-- Check for slow queries (if pg_stat_statements enabled)
SELECT query, calls, mean_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;
```
### Connection String
For debugging with external tools:
```
postgresql://pd_admin:PASSWORD@sba_postgres:5432/pd_master
```
--- ---
## Production Deployment Checklist ## Production Deployment Checklist
### Pre-Migration
- [ ] Backup production SQLite database - [ ] Backup production SQLite database
- [ ] Schedule maintenance window (~30 minutes) - [ ] Schedule maintenance window (~30 minutes)
- [ ] Notify users of downtime
- [ ] Verify PostgreSQL container is healthy
- [ ] Verify Docker network exists
### Migration
- [ ] Create PostgreSQL database and user - [ ] Create PostgreSQL database and user
- [ ] Create schema - [ ] Grant schema permissions (PostgreSQL 15+)
- [ ] Copy fresh SQLite from production - [ ] Create schema (29 tables)
- [ ] **Stop production API**
- [ ] Copy SQLite database
- [ ] Run migration script - [ ] Run migration script
- [ ] Verify migration summary (21/27 tables, ~550K records) - [ ] Verify migration summary (21/27 tables, ~550K records)
- [ ] Verify gamerewards has 10 records - [ ] Verify gamerewards count matches SQLite
- [ ] Run VACUUM ANALYZE
### Post-Migration
- [ ] Start PostgreSQL API container - [ ] Start PostgreSQL API container
- [ ] Test all endpoints (especially GROUP BY endpoints) - [ ] Test all basic endpoints (200 response)
- [ ] Test GROUP BY endpoints (batting, pitching, decisions/rest)
- [ ] Verify game-summary endpoint
- [ ] Update DNS/load balancer to point to new container - [ ] Update DNS/load balancer to point to new container
- [ ] Monitor for 24 hours - [ ] Monitor logs for 1 hour
- [ ] Remove old SQLite container - [ ] Monitor for 24 hours before removing SQLite backup
--- ---
## Known Limitations ## Known Limitations
1. **Orphaned historical data not migrated** - Records from deleted teams/games are correctly rejected by PostgreSQL FK constraints. This is expected and doesn't affect current gameplay. 1. **Orphaned historical data not migrated** - Records from deleted teams/games are correctly rejected by PostgreSQL FK constraints. This is expected and doesn't affect current gameplay. The original SQLite backup preserves this data if ever needed.
2. **Legacy tables excluded** - `battingstat` and `pitchingstat` are not migrated (legacy, unused by current app). 2. **Legacy tables excluded** - `battingstat` and `pitchingstat` are not migrated (legacy, unused by current app).
3. **Roster table mostly empty** - Most roster records reference deleted cards. Current rosters work correctly. 3. **Roster table mostly empty** - Most roster records reference deleted cards. Current active rosters work correctly.
4. **Skipped records are permanent** - The 168,463 skipped records will not exist in PostgreSQL. Keep SQLite backup indefinitely.
--- ---
@ -323,3 +474,19 @@ Database: pd_master
User: pd_admin User: pd_admin
Network: dev-sba-database_default Network: dev-sba-database_default
``` ```
---
## Monitoring (Post-Deployment)
### Key Metrics to Watch
1. **API Response Times** - Should be comparable or faster than SQLite
2. **Connection Pool Usage** - `max_connections=20` in db_engine.py
3. **Disk Usage** - PostgreSQL data directory growth
4. **Error Logs** - Check for FK violations, connection errors
### Log Locations
- API logs: `/usr/src/app/logs/` (mounted volume)
- PostgreSQL logs: `docker logs sba_postgres`