cal/paper-dynasty-database
1
0
Fork 0
Code Issues 7 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Update migration guide with finalized production plan

Browse Source 
- Updated test results with final migration metrics (549,788 records)
- Documented all tested endpoints with status
- Added complete step-by-step production migration plan
- Included rollback procedures and deployment checklist
- Documented known limitations and connection details
This commit is contained in:
Cal Corum 2026-01-27 14:03:05 -06:00
parent 94c8564282
commit ea5c047b15
1 changed files with 196 additions and 198 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

394
docs/POSTGRES_MIGRATION_GUIDE.md
View File

@ -10,15 +10,57 @@ This document captures lessons learned from test migrations and provides a step-
--- ---
## Code Changes Required ## Final Test Results
### 1. BigIntegerField for Discord IDs ### Migration Summary (January 27, 2026)
Discord snowflake IDs exceed PostgreSQL INTEGER max (2.1 billion). Changed to BIGINT: | Metric | Value |
|--------|-------|
| **Tables Successful** | 21/27 |
| **Records Inserted** | 549,788 |
| **Records Skipped** | 168,463 (orphaned FK) |
| **Duration** | ~21 minutes |
| **gamerewards** | 10/10 ✅ |
### Tables with Orphaned Data (Expected)
These tables have records that reference deleted teams/games - PostgreSQL correctly rejects them:
| Table | SQLite | PostgreSQL | Skipped | Reason |
|-------|--------|------------|---------|--------|
| pack | 25,224 | 14,708 | 10,516 | Deleted teams |
| card | 71,380 | 33,412 | 37,968 | Deleted packs/teams |
| result | 2,235 | 1,224 | 1,011 | Deleted teams |
| stratgame | 5,292 | 3,760 | 1,532 | Deleted teams |
| stratplay | 418,523 | 13,963 | 404,560 | Deleted games |
| decision | 36,900 | 24,551 | 12,349 | Deleted games |
### All Tested Endpoints ✅
| Endpoint | Status |
|----------|--------|
| `/api/v2/teams` | ✅ 200 |
| `/api/v2/players` | ✅ 200 |
| `/api/v2/cards` | ✅ 200 |
| `/api/v2/cardsets` | ✅ 200 |
| `/api/v2/games` | ✅ 200 |
| `/api/v2/decisions` | ✅ 200 |
| `/api/v2/decisions/rest` | ✅ 200 |
| `/api/v2/current` | ✅ 200 |
| `/api/v2/gamerewards` | ✅ 200 |
| `/api/v2/plays` | ✅ 200 |
| `/api/v2/plays/batting?group_by=player` | ✅ 200 |
| `/api/v2/plays/pitching?group_by=player` | ✅ 200 |
| `/api/v2/plays/game-summary/{id}` | ✅ 200 |
---
## Code Changes Summary
### 1. Database Schema (`app/db_engine.py`)
**BigIntegerField for Discord IDs** (exceed INTEGER max of 2.1 billion):
```python ```python
# app/db_engine.py
# Line 179 - Current.live_scoreboard # Line 179 - Current.live_scoreboard
live_scoreboard = BigIntegerField() # Discord channel ID live_scoreboard = BigIntegerField() # Discord channel ID
@ -26,175 +68,138 @@ live_scoreboard = BigIntegerField() # Discord channel ID
gmid = BigIntegerField() # Discord user ID gmid = BigIntegerField() # Discord user ID
``` ```
### 2. Nullable DateTimeField for Optional Timestamps **Nullable DateTimeField**:
```python ```python
# Line 715 - GauntletRun.ended # Line 715 - GauntletRun.ended
ended = DateTimeField(null=True) # NULL means run not yet ended ended = DateTimeField(null=True) # NULL means run not yet ended
``` ```
### 3. Migration Script Type Conversions ### 2. PostgreSQL Query Compatibility (`app/routers_v2/stratplays.py`)
The migration script (`scripts/migrate_to_postgres.py`) handles: **FK NULL checks** - use `_id` suffix to avoid FK lookup:
```python
# Wrong (triggers FK lookup, fails if row missing):
if x.batter:
model_to_dict(x.batter)
**Boolean Columns** (SQLite 0/1 → PostgreSQL True/False): # Correct:
- event.active if x.batter_id:
- cardset.for_purchase, in_packs, ranked_legal model_to_dict(x.batter)
- team.has_guide ```
- packtype.available
- result.ranked, short_game
- stratgame.ranked, short_game, forfeit
- stratplay.is_go_ahead, is_tied, is_new_inning
- decision.is_start
- battingcard.steal_auto
- notification.ack
**DateTime Columns** (Unix ms → PostgreSQL timestamp): **Boolean aggregation** - PostgreSQL can't sum booleans:
- pack.open_time ```python
- battingstat.created # Wrong:
- pitchingstat.created fn.SUM(Decision.is_start)
- result.created
- stratgame.created, ended
- notification.created
- gauntletrun.created, ended
- paperdex.created
- reward.created
- award.created
**Reserved Words** (quoted in SQL): # Correct:
- notification.desc → "desc" fn.SUM(Case(None, [(Decision.is_start == True, 1)], 0))
```
**Explicit GROUP BY** - PostgreSQL requires all non-aggregated columns:
```python
# Added .group_by(Decision.pitcher) to aggregate queries
```
### 3. PostgreSQL Query Compatibility (`app/routers_v2/decisions.py`)
**Missing GROUP BY on aggregate query**:
```python
# Added .group_by(StratPlay.pitcher, StratPlay.game) to the rest endpoint query
```
### 4. Migration Script (`scripts/migrate_to_postgres.py`)
**Migration order fix** - `gamerewards` moved after `player`:
```python
MIGRATION_ORDER = [
...
"player", # Tier 3
"gamerewards", # Tier 3b - depends on player
...
]
```
**Type conversions handled**:
- Boolean columns: SQLite 0/1 → PostgreSQL True/False
- DateTime columns: Unix ms → PostgreSQL timestamp
- Reserved words: `notification.desc` → `"desc"`
--- ---
## Pre-Migration Data Cleanup ## Production Migration Plan
### Required Cleanup Steps ### Prerequisites
Run these SQL commands on SQLite before migration to maximize data integrity: 1. **PostgreSQL Server Running**
- Container: `sba_postgres` (PostgreSQL 17)
- Network: `dev-sba-database_default`
```sql 2. **Docker Image Built and Pushed**
-- 1. Fix orphaned cards with invalid pack_id (set to NULL) ```bash
UPDATE card SET pack_id = NULL docker build -t manticorum67/paper-dynasty-database:postgres-migration .
WHERE pack_id IS NOT NULL docker push manticorum67/paper-dynasty-database:postgres-migration
AND pack_id NOT IN (SELECT id FROM pack); ```
-- 2. Delete packs from deleted teams ### Step 1: Create Database and User
DELETE FROM pack
WHERE team_id NOT IN (SELECT id FROM team);
-- 3. Delete cards from deleted teams
DELETE FROM card
WHERE team_id NOT IN (SELECT id FROM team);
-- 4. Delete duplicate stratplay records (keep lowest id)
DELETE FROM stratplay WHERE id NOT IN (
SELECT MIN(id) FROM stratplay GROUP BY game_id, play_num
);
-- 5. Delete roster entries for deleted cards
DELETE FROM roster
WHERE card_1 NOT IN (SELECT id FROM card)
OR card_2 NOT IN (SELECT id FROM card)
-- ... continue for all 26 card positions
;
-- 6. Delete battingstat for deleted cards
DELETE FROM battingstat
WHERE card_id NOT IN (SELECT id FROM card);
-- 7. Delete pitchingstat for deleted cards
DELETE FROM pitchingstat
WHERE card_id NOT IN (SELECT id FROM card);
-- 8. Delete stratplay for deleted games
DELETE FROM stratplay
WHERE game_id NOT IN (SELECT id FROM stratgame);
-- 9. Delete decision for deleted games
DELETE FROM decision
WHERE game_id NOT IN (SELECT id FROM stratgame);
```
### Orphan Analysis Queries
Before cleanup, run these to understand scope:
```sql
-- Count orphaned packs
SELECT COUNT(*) FROM pack WHERE team_id NOT IN (SELECT id FROM team);
-- Count orphaned cards (team)
SELECT COUNT(*) FROM card WHERE team_id NOT IN (SELECT id FROM team);
-- Count orphaned cards (pack)
SELECT COUNT(*) FROM card WHERE pack_id IS NOT NULL AND pack_id NOT IN (SELECT id FROM pack);
-- Count duplicate stratplays
SELECT COUNT(*) FROM (
SELECT game_id, play_num FROM stratplay
GROUP BY game_id, play_num HAVING COUNT(*) > 1
);
-- Count orphaned battingstats
SELECT COUNT(*) FROM battingstat WHERE card_id NOT IN (SELECT id FROM card);
-- Count orphaned pitchingstats
SELECT COUNT(*) FROM pitchingstat WHERE card_id NOT IN (SELECT id FROM card);
```
---
## Migration Execution Steps
### Phase 1: Build Docker Image (Local)
```bash
cd /mnt/NV2/Development/paper-dynasty/database
git checkout postgres-migration
docker build -t manticorum67/paper-dynasty-database:postgres-migration .
docker push manticorum67/paper-dynasty-database:postgres-migration
```
### Phase 2: Create PostgreSQL Database (Dev Server)
```bash ```bash
ssh sba-db ssh sba-db
# Create user and database # Create user
docker exec sba_postgres psql -U sba_admin -d postgres -c \ docker exec sba_postgres psql -U sba_admin -d postgres -c \
"CREATE USER pd_admin WITH PASSWORD 'YOUR_PASSWORD';" "CREATE USER pd_admin WITH PASSWORD 'YOUR_SECURE_PASSWORD';"
# Create database
docker exec sba_postgres psql -U sba_admin -d postgres -c \ docker exec sba_postgres psql -U sba_admin -d postgres -c \
"CREATE DATABASE pd_master OWNER pd_admin;" "CREATE DATABASE pd_master OWNER pd_admin;"
docker exec sba_postgres psql -U sba_admin -d postgres -c \ docker exec sba_postgres psql -U sba_admin -d postgres -c \
"GRANT ALL PRIVILEGES ON DATABASE pd_master TO pd_admin;" "GRANT ALL PRIVILEGES ON DATABASE pd_master TO pd_admin;"
``` ```
### Phase 3: Create Schema ### Step 2: Create Logs Directory
```bash
mkdir -p /path/to/logs/database
chmod 777 /path/to/logs /path/to/logs/database
```
### Step 3: Create Schema
```bash ```bash
docker pull manticorum67/paper-dynasty-database:postgres-migration docker pull manticorum67/paper-dynasty-database:postgres-migration
docker run --rm \ docker run --rm \
--network dev-sba-database_default \ --network dev-sba-database_default \
-v /home/cal/container-data/dev-sba-database/logs:/usr/src/app/logs \ -v /path/to/logs:/usr/src/app/logs \
-e DATABASE_TYPE=postgresql \ -e DATABASE_TYPE=postgresql \
-e POSTGRES_HOST=sba_postgres \ -e POSTGRES_HOST=sba_postgres \
-e POSTGRES_DB=pd_master \ -e POSTGRES_DB=pd_master \
-e POSTGRES_USER=pd_admin \ -e POSTGRES_USER=pd_admin \
-e POSTGRES_PASSWORD='YOUR_PASSWORD' \ -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
-e POSTGRES_PORT=5432 \ -e POSTGRES_PORT=5432 \
manticorum67/paper-dynasty-database:postgres-migration \ manticorum67/paper-dynasty-database:postgres-migration \
python -c " python -c "
from app.db_engine import db, Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision from app.db_engine import db, Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision
db.create_tables([Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision]) db.create_tables([Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision])
print('Tables created successfully')
db.close() db.close()
" "
``` ```
### Phase 4: Run Migration ### Step 4: Copy Production SQLite
```bash ```bash
docker run --rm \ # Copy from production container
docker cp pd_database_v2:/usr/src/app/storage/pd_master.db /path/to/storage/pd_master.db
```
### Step 5: Run Migration
```bash
# Run in background (takes ~21 minutes)
nohup docker run --rm \
--network dev-sba-database_default \ --network dev-sba-database_default \
-v /path/to/storage:/usr/src/app/storage \ -v /path/to/storage:/usr/src/app/storage \
-v /path/to/logs:/usr/src/app/logs \ -v /path/to/logs:/usr/src/app/logs \
@ -203,24 +208,33 @@ docker run --rm \
-e POSTGRES_HOST=sba_postgres \ -e POSTGRES_HOST=sba_postgres \
-e POSTGRES_DB=pd_master \ -e POSTGRES_DB=pd_master \
-e POSTGRES_USER=pd_admin \ -e POSTGRES_USER=pd_admin \
-e POSTGRES_PASSWORD='YOUR_PASSWORD' \ -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
-e POSTGRES_PORT=5432 \ -e POSTGRES_PORT=5432 \
manticorum67/paper-dynasty-database:postgres-migration \ manticorum67/paper-dynasty-database:postgres-migration \
python scripts/migrate_to_postgres.py --sqlite-path storage/pd_master.db python scripts/migrate_to_postgres.py --sqlite-path storage/pd_master.db > /tmp/migration.log 2>&1 &
# Monitor progress
tail -f /tmp/migration.log
``` ```
### Phase 5: Verify Migration ### Step 6: Verify Migration
```sql ```bash
-- Check table counts # Check summary
grep -E '(MIGRATION SUMMARY|Tables:|Records:|Duration:)' /tmp/migration.log
# Check key table counts
docker exec sba_postgres psql -U pd_admin -d pd_master -c "
SELECT 'player' as tbl, COUNT(*) FROM player SELECT 'player' as tbl, COUNT(*) FROM player
UNION ALL SELECT 'team', COUNT(*) FROM team UNION ALL SELECT 'team', COUNT(*) FROM team
UNION ALL SELECT 'card', COUNT(*) FROM card UNION ALL SELECT 'card', COUNT(*) FROM card
UNION ALL SELECT 'gamerewards', COUNT(*) FROM gamerewards
UNION ALL SELECT 'stratplay', COUNT(*) FROM stratplay UNION ALL SELECT 'stratplay', COUNT(*) FROM stratplay
ORDER BY tbl; ORDER BY tbl;
"
``` ```
### Phase 6: Start API with PostgreSQL ### Step 7: Start API Container
```bash ```bash
docker run -d \ docker run -d \
@ -231,64 +245,26 @@ docker run -d \
-e POSTGRES_HOST=sba_postgres \ -e POSTGRES_HOST=sba_postgres \
-e POSTGRES_DB=pd_master \ -e POSTGRES_DB=pd_master \
-e POSTGRES_USER=pd_admin \ -e POSTGRES_USER=pd_admin \
-e POSTGRES_PASSWORD='YOUR_PASSWORD' \ -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
-e POSTGRES_PORT=5432 \ -e POSTGRES_PORT=5432 \
-e API_TOKEN=$API_TOKEN \ -v /path/to/logs:/usr/src/app/logs \
manticorum67/paper-dynasty-database:postgres-migration manticorum67/paper-dynasty-database:postgres-migration
``` ```
--- ### Step 8: Test Endpoints
## Test Results Summary ```bash
# Basic endpoints
for ep in "teams?limit=1" "players?limit=1" "cards?limit=1" "games?limit=1" "current" "gamerewards"; do
echo -n "$ep: "
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8100/api/v2/$ep"
echo
done
### Test Migration #3 (Fresh Production Copy + Minimal Cleanup) # GROUP BY endpoints (critical)
curl -s "http://localhost:8100/api/v2/plays/batting?season=10&group_by=player&limit=1" | head -c 200
| Metric | Value | curl -s "http://localhost:8100/api/v2/plays/pitching?season=10&group_by=player&limit=1" | head -c 200
|--------|-------| ```
| Tables Successful | 23/29 (79%) |
| Records Inserted | 562,489 |
| Records Skipped | 218,387 |
| Duration | 25 minutes |
**Pre-migration cleanup performed:**
- Deleted 1,953 orphaned battingstats (invalid card_id)
- Deleted 437 orphaned pitchingstats (invalid card_id)
**Records skipped during migration (expected):**
- These are historical records from deleted teams/cards from previous seasons
- FK integrity is enforced in PostgreSQL, so orphaned records are correctly rejected
- The API functions correctly with the migrated data
### Working Endpoints (All Tested)
- /api/v2/current
- /api/v2/teams
- /api/v2/players
- /api/v2/cards
- /api/v2/cardsets
- /api/v2/games
- /api/v2/decisions
- /api/v2/plays/batting (with group_by=player)
- /api/v2/plays/pitching (with group_by=player)
### Fixed Issues (This Session)
1. **NULL handling for GROUP BY foreign keys** (`app/routers_v2/stratplays.py`)
- Changed `if x.batter` → `if x.batter_id` (line ~718)
- Changed `if x.batter_team` → `if x.batter_team_id` (line ~725)
- Changed `if x.pitcher` → `if x.pitcher_id` (line ~1100)
- Changed `if x.pitcher_team` → `if x.pitcher_team_id` (line ~1103)
- **Why:** Checking `x.foreign_key` triggers FK lookup which fails if the referenced row doesn't exist. Checking `x.foreign_key_id` only checks the raw integer value.
2. **Boolean SUM for PostgreSQL** (`app/routers_v2/stratplays.py`)
- Changed `fn.SUM(Decision.is_start)` → `fn.SUM(Case(None, [(Decision.is_start == True, 1)], 0))`
- **Why:** PostgreSQL cannot sum booleans directly; must cast to integer first.
3. **Missing GROUP BY on Decision query** (`app/routers_v2/stratplays.py`)
- Added `.group_by(Decision.pitcher)` to the `all_dec` query
- **Why:** PostgreSQL requires explicit GROUP BY for all non-aggregate columns (SQLite was lenient).
4. **Import Case from db_engine** (`app/routers_v2/stratplays.py`)
- Added `Case` to imports for PostgreSQL-compatible boolean aggregation
--- ---
@ -297,16 +273,14 @@ docker run -d \
To switch back to SQLite: To switch back to SQLite:
```bash ```bash
# Stop PostgreSQL container # Stop PostgreSQL API
docker stop pd_postgres_api docker stop pd_postgres_api
# Start with SQLite # Start SQLite API (original image)
docker run -d \ docker run -d \
--name pd_sqlite_api \ --name pd_sqlite_api \
-p 8100:80 \ -p 8100:80 \
-v /path/to/storage:/usr/src/app/storage \ -v /path/to/storage:/usr/src/app/storage \
-e DATABASE_TYPE=sqlite \
-e API_TOKEN=$API_TOKEN \
manticorum67/paper-dynasty-database:latest manticorum67/paper-dynasty-database:latest
``` ```
@ -314,14 +288,38 @@ docker run -d \
## Production Deployment Checklist ## Production Deployment Checklist
- [ ] Backup production SQLite - [ ] Backup production SQLite database
- [ ] Run orphan analysis on production data - [ ] Schedule maintenance window (~30 minutes)
- [ ] Execute cleanup SQL - [ ] Create PostgreSQL database and user
- [ ] Verify cleanup counts - [ ] Create schema
- [ ] Schedule maintenance window - [ ] Copy fresh SQLite from production
- [ ] Run migration - [ ] Run migration script
- [ ] Verify record counts - [ ] Verify migration summary (21/27 tables, ~550K records)
- [ ] Test all endpoints - [ ] Verify gamerewards has 10 records
- [ ] Update docker-compose.yml with PostgreSQL config - [ ] Start PostgreSQL API container
- [ ] Test all endpoints (especially GROUP BY endpoints)
- [ ] Update DNS/load balancer to point to new container
- [ ] Monitor for 24 hours - [ ] Monitor for 24 hours
- [ ] Remove SQLite dependency (optional) - [ ] Remove old SQLite container
---
## Known Limitations
1. **Orphaned historical data not migrated** - Records from deleted teams/games are correctly rejected by PostgreSQL FK constraints. This is expected and doesn't affect current gameplay.
2. **Legacy tables excluded** - `battingstat` and `pitchingstat` are not migrated (legacy, unused by current app).
3. **Roster table mostly empty** - Most roster records reference deleted cards. Current rosters work correctly.
---
## Connection Details (Dev Environment)
```
PostgreSQL Host: sba_postgres
PostgreSQL Port: 5432
Database: pd_master
User: pd_admin
Network: dev-sba-database_default
```