# Paper Dynasty PostgreSQL Migration Guide

## Overview

This document captures lessons learned from test migrations and provides a step-by-step guide for production deployment.

**Migration Branch:** `postgres-migration`  
**Target:** PostgreSQL 17 on `sba_postgres` container  
**Source:** SQLite `storage/pd_master.db`

---

## Final Test Results

### Migration Summary (January 27, 2026)

| Metric | Value |
|--------|-------|
| **Tables Successful** | 21/27 |
| **Records Inserted** | 549,788 |
| **Records Skipped** | 168,463 (orphaned FK) |
| **Duration** | ~21 minutes |
| **gamerewards** | 10/10 ✅ |

### Tables with Orphaned Data (Expected)

These tables have records that reference deleted teams/games - PostgreSQL correctly rejects them:

| Table | SQLite | PostgreSQL | Skipped | Reason |
|-------|--------|------------|---------|--------|
| pack | 25,224 | 14,708 | 10,516 | Deleted teams |
| card | 71,380 | 33,412 | 37,968 | Deleted packs/teams |
| result | 2,235 | 1,224 | 1,011 | Deleted teams |
| stratgame | 5,292 | 3,760 | 1,532 | Deleted teams |
| stratplay | 418,523 | 13,963 | 404,560 | Deleted games |
| decision | 36,900 | 24,551 | 12,349 | Deleted games |

### All Tested Endpoints ✅

| Endpoint | Status |
|----------|--------|
| `/api/v2/teams` | ✅ 200 |
| `/api/v2/players` | ✅ 200 |
| `/api/v2/cards` | ✅ 200 |
| `/api/v2/cardsets` | ✅ 200 |
| `/api/v2/games` | ✅ 200 |
| `/api/v2/decisions` | ✅ 200 |
| `/api/v2/decisions/rest` | ✅ 200 |
| `/api/v2/current` | ✅ 200 |
| `/api/v2/gamerewards` | ✅ 200 |
| `/api/v2/plays` | ✅ 200 |
| `/api/v2/plays/batting?group_by=player` | ✅ 200 |
| `/api/v2/plays/pitching?group_by=player` | ✅ 200 |
| `/api/v2/plays/game-summary/{id}` | ✅ 200 |

---

## Code Changes Summary

### 1. Database Schema (`app/db_engine.py`)

**BigIntegerField for Discord IDs** (exceed INTEGER max of 2.1 billion):
```python
# Line 179 - Current.live_scoreboard
live_scoreboard = BigIntegerField()  # Discord channel ID

# Line 368 - Team.gmid  
gmid = BigIntegerField()  # Discord user ID
```

**Nullable DateTimeField**:
```python
# Line 715 - GauntletRun.ended
ended = DateTimeField(null=True)  # NULL means run not yet ended
```

### 2. PostgreSQL Query Compatibility (`app/routers_v2/stratplays.py`)

**FK NULL checks** - use `_id` suffix to avoid FK lookup:
```python
# Wrong (triggers FK lookup, fails if row missing):
if x.batter:
    model_to_dict(x.batter)

# Correct:
if x.batter_id:
    model_to_dict(x.batter)
```

**Boolean aggregation** - PostgreSQL can't sum booleans:
```python
# Wrong:
fn.SUM(Decision.is_start)

# Correct:
fn.SUM(Case(None, [(Decision.is_start == True, 1)], 0))
```

**Explicit GROUP BY** - PostgreSQL requires all non-aggregated columns:
```python
# Added .group_by(Decision.pitcher) to aggregate queries
```

### 3. PostgreSQL Query Compatibility (`app/routers_v2/decisions.py`)

**Missing GROUP BY on aggregate query**:
```python
# Added .group_by(StratPlay.pitcher, StratPlay.game) to the rest endpoint query
```

### 4. Migration Script (`scripts/migrate_to_postgres.py`)

**Migration order fix** - `gamerewards` moved after `player`:
```python
MIGRATION_ORDER = [
    ...
    "player",        # Tier 3
    "gamerewards",   # Tier 3b - depends on player
    ...
]
```

**Type conversions handled**:
- Boolean columns: SQLite 0/1 → PostgreSQL True/False
- DateTime columns: Unix ms → PostgreSQL timestamp
- Reserved words: `notification.desc` → `"desc"`

---

## Production Migration Plan

### Prerequisites

1. **PostgreSQL Server Running**
   - Container: `sba_postgres` (PostgreSQL 17)
   - Network: `dev-sba-database_default`

2. **Docker Image Built and Pushed**
   ```bash
   docker build -t manticorum67/paper-dynasty-database:postgres-migration .
   docker push manticorum67/paper-dynasty-database:postgres-migration
   ```

### Step 1: Create Database and User

```bash
ssh sba-db

# Create user
docker exec sba_postgres psql -U sba_admin -d postgres -c \
  "CREATE USER pd_admin WITH PASSWORD 'YOUR_SECURE_PASSWORD';"

# Create database
docker exec sba_postgres psql -U sba_admin -d postgres -c \
  "CREATE DATABASE pd_master OWNER pd_admin;"

docker exec sba_postgres psql -U sba_admin -d postgres -c \
  "GRANT ALL PRIVILEGES ON DATABASE pd_master TO pd_admin;"
```

### Step 2: Create Logs Directory

```bash
mkdir -p /path/to/logs/database
chmod 777 /path/to/logs /path/to/logs/database
```

### Step 3: Create Schema

```bash
docker pull manticorum67/paper-dynasty-database:postgres-migration

docker run --rm \
  --network dev-sba-database_default \
  -v /path/to/logs:/usr/src/app/logs \
  -e DATABASE_TYPE=postgresql \
  -e POSTGRES_HOST=sba_postgres \
  -e POSTGRES_DB=pd_master \
  -e POSTGRES_USER=pd_admin \
  -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
  -e POSTGRES_PORT=5432 \
  manticorum67/paper-dynasty-database:postgres-migration \
  python -c "
from app.db_engine import db, Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision
db.create_tables([Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision])
print('Tables created successfully')
db.close()
"
```

### Step 4: Copy Production SQLite

```bash
# Copy from production container
docker cp pd_database_v2:/usr/src/app/storage/pd_master.db /path/to/storage/pd_master.db
```

### Step 5: Run Migration

```bash
# Run in background (takes ~21 minutes)
nohup docker run --rm \
  --network dev-sba-database_default \
  -v /path/to/storage:/usr/src/app/storage \
  -v /path/to/logs:/usr/src/app/logs \
  -v /path/to/scripts:/usr/src/app/scripts \
  -e DATABASE_TYPE=postgresql \
  -e POSTGRES_HOST=sba_postgres \
  -e POSTGRES_DB=pd_master \
  -e POSTGRES_USER=pd_admin \
  -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
  -e POSTGRES_PORT=5432 \
  manticorum67/paper-dynasty-database:postgres-migration \
  python scripts/migrate_to_postgres.py --sqlite-path storage/pd_master.db > /tmp/migration.log 2>&1 &

# Monitor progress
tail -f /tmp/migration.log
```

### Step 6: Verify Migration

```bash
# Check summary
grep -E '(MIGRATION SUMMARY|Tables:|Records:|Duration:)' /tmp/migration.log

# Check key table counts
docker exec sba_postgres psql -U pd_admin -d pd_master -c "
SELECT 'player' as tbl, COUNT(*) FROM player
UNION ALL SELECT 'team', COUNT(*) FROM team
UNION ALL SELECT 'card', COUNT(*) FROM card
UNION ALL SELECT 'gamerewards', COUNT(*) FROM gamerewards
UNION ALL SELECT 'stratplay', COUNT(*) FROM stratplay
ORDER BY tbl;
"
```

### Step 7: Start API Container

```bash
docker run -d \
  --name pd_postgres_api \
  --network dev-sba-database_default \
  -p 8100:80 \
  -e DATABASE_TYPE=postgresql \
  -e POSTGRES_HOST=sba_postgres \
  -e POSTGRES_DB=pd_master \
  -e POSTGRES_USER=pd_admin \
  -e POSTGRES_PASSWORD='YOUR_SECURE_PASSWORD' \
  -e POSTGRES_PORT=5432 \
  -v /path/to/logs:/usr/src/app/logs \
  manticorum67/paper-dynasty-database:postgres-migration
```

### Step 8: Test Endpoints

```bash
# Basic endpoints
for ep in "teams?limit=1" "players?limit=1" "cards?limit=1" "games?limit=1" "current" "gamerewards"; do
  echo -n "$ep: "
  curl -s -o /dev/null -w "%{http_code}" "http://localhost:8100/api/v2/$ep"
  echo
done

# GROUP BY endpoints (critical)
curl -s "http://localhost:8100/api/v2/plays/batting?season=10&group_by=player&limit=1" | head -c 200
curl -s "http://localhost:8100/api/v2/plays/pitching?season=10&group_by=player&limit=1" | head -c 200
```

---

## Rollback Plan

To switch back to SQLite:

```bash
# Stop PostgreSQL API
docker stop pd_postgres_api

# Start SQLite API (original image)
docker run -d \
  --name pd_sqlite_api \
  -p 8100:80 \
  -v /path/to/storage:/usr/src/app/storage \
  manticorum67/paper-dynasty-database:latest
```

---

## Production Deployment Checklist

- [ ] Backup production SQLite database
- [ ] Schedule maintenance window (~30 minutes)
- [ ] Create PostgreSQL database and user
- [ ] Create schema
- [ ] Copy fresh SQLite from production
- [ ] Run migration script
- [ ] Verify migration summary (21/27 tables, ~550K records)
- [ ] Verify gamerewards has 10 records
- [ ] Start PostgreSQL API container
- [ ] Test all endpoints (especially GROUP BY endpoints)
- [ ] Update DNS/load balancer to point to new container
- [ ] Monitor for 24 hours
- [ ] Remove old SQLite container

---

## Known Limitations

1. **Orphaned historical data not migrated** - Records from deleted teams/games are correctly rejected by PostgreSQL FK constraints. This is expected and doesn't affect current gameplay.

2. **Legacy tables excluded** - `battingstat` and `pitchingstat` are not migrated (legacy, unused by current app).

3. **Roster table mostly empty** - Most roster records reference deleted cards. Current rosters work correctly.

---

## Connection Details (Dev Environment)

```
PostgreSQL Host: sba_postgres
PostgreSQL Port: 5432
Database: pd_master
User: pd_admin
Network: dev-sba-database_default
```