paper-dynasty-database/docs/POSTGRES_MIGRATION_GUIDE.md

Paper Dynasty PostgreSQL Migration Guide

Overview

This document captures lessons learned from test migrations and provides a step-by-step guide for production deployment.

Migration Branch: postgres-migration
Target: PostgreSQL 17 on sba_postgres container
Source: SQLite storage/pd_master.db

---

Final Test Results

Migration Summary (January 27, 2026)

Metric	Value
Tables Successful	21/27
Records Inserted	549,788
Records Skipped	168,463 (orphaned FK)
Duration	~21 minutes
gamerewards	10/10 ✅

Tables with Orphaned Data (Expected)

These tables have records that reference deleted teams/games - PostgreSQL correctly rejects them:

Table	SQLite	PostgreSQL	Skipped	Reason
pack	25,224	14,708	10,516	Deleted teams
card	71,380	33,412	37,968	Deleted packs/teams
result	2,235	1,224	1,011	Deleted teams
stratgame	5,292	3,760	1,532	Deleted teams
stratplay	418,523	13,963	404,560	Deleted games
decision	36,900	24,551	12,349	Deleted games

All Tested Endpoints ✅

Endpoint	Status
/api/v2/teams	✅ 200
/api/v2/players	✅ 200
/api/v2/cards	✅ 200
/api/v2/cardsets	✅ 200
/api/v2/games	✅ 200
/api/v2/decisions	✅ 200
/api/v2/decisions/rest	✅ 200
/api/v2/current	✅ 200
/api/v2/gamerewards	✅ 200
/api/v2/plays	✅ 200
/api/v2/plays/batting?group_by=player	✅ 200
/api/v2/plays/pitching?group_by=player	✅ 200
/api/v2/plays/game-summary/{id}	✅ 200

---

Code Changes Summary

1. Database Schema (app/db_engine.py)

BigIntegerField for Discord IDs (exceed INTEGER max of 2.1 billion):

# Line 179 - Current.live_scoreboard
live_scoreboard = BigIntegerField()  # Discord channel ID

# Line 368 - Team.gmid  
gmid = BigIntegerField()  # Discord user ID

Nullable DateTimeField:

# Line 715 - GauntletRun.ended
ended = DateTimeField(null=True)  # NULL means run not yet ended

2. PostgreSQL Query Compatibility (app/routers_v2/stratplays.py)

FK NULL checks - use _id suffix to avoid FK lookup:

# Wrong (triggers FK lookup, fails if row missing):
if x.batter:
    model_to_dict(x.batter)

# Correct:
if x.batter_id:
    model_to_dict(x.batter)

Boolean aggregation - PostgreSQL can't sum booleans:

# Wrong:
fn.SUM(Decision.is_start)

# Correct:
fn.SUM(Case(None, [(Decision.is_start == True, 1)], 0))

Explicit GROUP BY - PostgreSQL requires all non-aggregated columns:

# Added .group_by(Decision.pitcher) to aggregate queries

3. PostgreSQL Query Compatibility (app/routers_v2/decisions.py)

Missing GROUP BY on aggregate query:

# Added .group_by(StratPlay.pitcher, StratPlay.game) to the rest endpoint query

4. Migration Script (scripts/migrate_to_postgres.py)

Migration order fix - gamerewards moved after player:

MIGRATION_ORDER = [
    ...
    "player",        # Tier 3
    "gamerewards",   # Tier 3b - depends on player
    ...
]

Type conversions handled:

• Boolean columns: SQLite 0/1 → PostgreSQL True/False
• DateTime columns: Unix ms → PostgreSQL timestamp
• Reserved words: notification.desc → "desc"

---

Production Migration Plan

Prerequisites

1. PostgreSQL Server Running

   • Container: sba_postgres (PostgreSQL 17)
   • Network: dev-sba-database_default

2. Docker Image Built and Pushed

   docker build -t manticorum67/paper-dynasty-database:postgres-migration .
   docker push manticorum67/paper-dynasty-database:postgres-migration
   

Step 1: Create Database and User

ssh sba-db

# Create user
docker exec sba_postgres psql -U sba_admin -d postgres -c \
  "CREATE USER pd_admin WITH PASSWORD 'YOUR_SECURE_PASSWORD';"

# Create database
docker exec sba_postgres psql -U sba_admin -d postgres -c \
  "CREATE DATABASE pd_master OWNER pd_admin;"

docker exec sba_postgres psql -U sba_admin -d postgres -c \
  "GRANT ALL PRIVILEGES ON DATABASE pd_master TO pd_admin;"

Step 2: Create Logs Directory

mkdir -p /path/to/logs/database
chmod 777 /path/to/logs /path/to/logs/database

Step 3: Create Schema

docker pull manticorum67/paper-dynasty-database:postgres-migration

docker run --rm \
  --network dev-sba-database_default \
  -v /path/to/logs:/usr/src/app/logs \
  -e DATABASE_TYPE=postgresql \
  -e POSTGRES_HOST=sba_postgres \
  -e POSTGRES_DB=pd_master \
  -e POSTGRES_USER=pd_admin \
  -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
  -e POSTGRES_PORT=5432 \
  manticorum67/paper-dynasty-database:postgres-migration \
  python -c "
from app.db_engine import db, Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision
db.create_tables([Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision])
print('Tables created successfully')
db.close()
"

Step 4: Copy Production SQLite

# Copy from production container
docker cp pd_database_v2:/usr/src/app/storage/pd_master.db /path/to/storage/pd_master.db

Step 5: Run Migration

# Run in background (takes ~21 minutes)
nohup docker run --rm \
  --network dev-sba-database_default \
  -v /path/to/storage:/usr/src/app/storage \
  -v /path/to/logs:/usr/src/app/logs \
  -v /path/to/scripts:/usr/src/app/scripts \
  -e DATABASE_TYPE=postgresql \
  -e POSTGRES_HOST=sba_postgres \
  -e POSTGRES_DB=pd_master \
  -e POSTGRES_USER=pd_admin \
  -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
  -e POSTGRES_PORT=5432 \
  manticorum67/paper-dynasty-database:postgres-migration \
  python scripts/migrate_to_postgres.py --sqlite-path storage/pd_master.db > /tmp/migration.log 2>&1 &

# Monitor progress
tail -f /tmp/migration.log

Step 6: Verify Migration

# Check summary
grep -E '(MIGRATION SUMMARY|Tables:|Records:|Duration:)' /tmp/migration.log

# Check key table counts
docker exec sba_postgres psql -U pd_admin -d pd_master -c "
SELECT 'player' as tbl, COUNT(*) FROM player
UNION ALL SELECT 'team', COUNT(*) FROM team
UNION ALL SELECT 'card', COUNT(*) FROM card
UNION ALL SELECT 'gamerewards', COUNT(*) FROM gamerewards
UNION ALL SELECT 'stratplay', COUNT(*) FROM stratplay
ORDER BY tbl;
"

Step 7: Start API Container

docker run -d \
  --name pd_postgres_api \
  --network dev-sba-database_default \
  -p 8100:80 \
  -e DATABASE_TYPE=postgresql \
  -e POSTGRES_HOST=sba_postgres \
  -e POSTGRES_DB=pd_master \
  -e POSTGRES_USER=pd_admin \
  -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
  -e POSTGRES_PORT=5432 \
  -v /path/to/logs:/usr/src/app/logs \
  manticorum67/paper-dynasty-database:postgres-migration

Step 8: Test Endpoints

# Basic endpoints
for ep in "teams?limit=1" "players?limit=1" "cards?limit=1" "games?limit=1" "current" "gamerewards"; do
  echo -n "$ep: "
  curl -s -o /dev/null -w "%{http_code}" "http://localhost:8100/api/v2/$ep"
  echo
done

# GROUP BY endpoints (critical)
curl -s "http://localhost:8100/api/v2/plays/batting?season=10&group_by=player&limit=1" | head -c 200
curl -s "http://localhost:8100/api/v2/plays/pitching?season=10&group_by=player&limit=1" | head -c 200

---

Rollback Plan

To switch back to SQLite:

# Stop PostgreSQL API
docker stop pd_postgres_api

# Start SQLite API (original image)
docker run -d \
  --name pd_sqlite_api \
  -p 8100:80 \
  -v /path/to/storage:/usr/src/app/storage \
  manticorum67/paper-dynasty-database:latest

---

Production Deployment Checklist

• Backup production SQLite database
• Schedule maintenance window (~30 minutes)
• Create PostgreSQL database and user
• Create schema
• Copy fresh SQLite from production
• Run migration script
• Verify migration summary (21/27 tables, ~550K records)
• Verify gamerewards has 10 records
• Start PostgreSQL API container
• Test all endpoints (especially GROUP BY endpoints)
• Update DNS/load balancer to point to new container
• Monitor for 24 hours
• Remove old SQLite container

---

Known Limitations

1. Orphaned historical data not migrated - Records from deleted teams/games are correctly rejected by PostgreSQL FK constraints. This is expected and doesn't affect current gameplay.

2. Legacy tables excluded - battingstat and pitchingstat are not migrated (legacy, unused by current app).

3. Roster table mostly empty - Most roster records reference deleted cards. Current rosters work correctly.

---

Connection Details (Dev Environment)

PostgreSQL Host: sba_postgres
PostgreSQL Port: 5432
Database: pd_master
User: pd_admin
Network: dev-sba-database_default