paper-dynasty-database/POSTGRES_MIGRATION_PLAN.md

PostgreSQL Migration Plan for Paper Dynasty Database

Created: 2025-11-07 Status: Planning Phase Estimated Effort: 3-5 days Risk Level: Medium

Table of Contents

1. Executive Summary
2. Migration Context
3. Critical Issues Identified
4. Detailed Code Changes Required
5. Migration Strategy
6. Testing Plan
7. Rollback Plan
8. Timeline

---

Executive Summary

Current State

• Database: SQLite 3.x with WAL mode
• Database File: storage/pd_master.db (~104 MB)
• ORM: Peewee 3.x
• Models: 40 database models
• API Endpoints: 30+ routers in app/routers_v2/

Target State

• Database: PostgreSQL 14+
• Connection Pooling: PooledPostgresqlDatabase
• Models: Same 40 models with explicit table names
• API Endpoints: Updated for PostgreSQL compatibility

Why Migrate?

Based on sister project (Major Domo) experience:

• Better concurrency and performance under load
• Production-ready transaction handling
• Better data integrity constraints
• Scalability for future growth

Key Metrics

• Files Requiring Changes: 7 must-modify + 15 test-thoroughly
• Models Missing table_name: 40/40 (100%)
• GROUP BY Issues: ~30 instances in 2 files
• SQLite-Specific Code: 3 critical instances

---

Migration Context

Sister Project Lessons Learned

The Major Domo project (SBA database) successfully migrated from SQLite to PostgreSQL in August 2025. Key findings:

What Went Well

✅ 74.6% immediate success rate on endpoint testing ✅ Zero data loss during migration ✅ All failures were environment/state differences, not migration bugs ✅ Peewee ORM handled most SQL differences automatically

Critical Issues Encountered

⚠️ PostgreSQL GROUP BY strictness - All non-aggregated SELECT fields must be in GROUP BY ⚠️ Table naming - Models must explicitly define Meta.table_name ⚠️ Connection pooling - Required configuration changes

Code Changes Required (Major Domo)

• Fixed ~20 GROUP BY clauses in stratplay.py
• Added table_name to all models
• Changed from SqliteDatabase to PooledPostgresqlDatabase
• Updated migrator from SqliteMigrator to PostgresqlMigrator

---

Critical Issues Identified

1. GROUP BY Queries (BLOCKING - Must Fix)

PostgreSQL Requirement: All non-aggregated SELECT fields MUST appear in GROUP BY clause

Location: app/routers_v2/stratplays.py

Issue 1: Batting Totals Query (Lines 342-456)

Current Code:

bat_plays = (
    StratPlay
        .select(
            StratPlay.batter,      # Field 1 - not aggregated
            StratPlay.game,        # Field 2 - not aggregated
            fn.SUM(StratPlay.pa).alias('sum_pa'),  # Aggregate
            # ... many more aggregates ...
            StratPlay.batter_team,  # Field 3 - not aggregated
        )
        .where(...)
)

# Later, conditionally group by:
if group_by == 'player':
    bat_plays = bat_plays.group_by(StratPlay.batter)  # ONLY batter!

Problem: When group_by='player', PostgreSQL will reject this because:

• SELECT includes: StratPlay.batter, StratPlay.game, StratPlay.batter_team
• GROUP BY only includes: StratPlay.batter
• StratPlay.game and StratPlay.batter_team are neither aggregated nor grouped

Sister Project Solution: Conditionally build SELECT fields based on group_by parameter

Example Fix Pattern (from Major Domo):

# Build SELECT fields conditionally based on group_by
base_select_fields = [
    fn.SUM(StratPlay.pa).alias('sum_pa'),
    fn.SUM(StratPlay.ab).alias('sum_ab'),
    # ... all aggregates ...
]

# Add non-aggregated fields based on grouping type
if group_by in ['player', 'playerteam', 'playergame']:
    base_select_fields.insert(0, StratPlay.batter)
if group_by in ['team', 'playerteam', 'teamgame']:
    base_select_fields.append(StratPlay.batter_team)
if group_by in ['playergame', 'teamgame']:
    base_select_fields.append(StratPlay.game)

bat_plays = StratPlay.select(*base_select_fields).where(...)

Affected Functions:

• get_batting_totals() - Lines 342-456 (bat_plays, run_plays)
• get_pitching_totals() - Lines 645-733 (pit_plays)
• get_game_summary() - Lines 911-929 (all_batters, all_pitchers)

Estimated Effort: 4-6 hours (complex logic, must test all group_by modes)

---

2. Database Models Missing table_name (BLOCKING - Must Fix)

PostgreSQL Best Practice: Explicitly define table names to avoid Peewee naming inconsistencies

Location: app/db_engine.py

All 40 Models Affected:

Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack,
Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex,
Reward, GameRewards, Notification, GauntletReward, GauntletRun,
BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings,
CardPosition, StratGame, StratPlay, Decision, Scouting, ...

Required Change (for EACH model):

class Current(BaseModel):
    season = IntegerField()
    week = IntegerField(default=0)
    # ... other fields ...

    class Meta:
        database = db
        table_name = 'current'  # ADD THIS LINE

Estimated Effort: 1-2 hours (repetitive, low risk)

---

3. Database Connection Configuration (BLOCKING - Must Fix)

Location: app/db_engine.py (Lines 11-18)

Current Code:

db = SqliteDatabase(
    'storage/pd_master.db',
    pragmas={
        'journal_mode': 'wal',
        'cache_size': -1 * 64000,
        'synchronous': 0
    }
)

Required Changes:

Option A: Environment-Based Configuration (Recommended)

import os
from playhouse.pool import PooledPostgresqlDatabase

DATABASE_TYPE = os.environ.get('DATABASE_TYPE', 'sqlite')

if DATABASE_TYPE.lower() == 'postgresql':
    db = PooledPostgresqlDatabase(
        os.environ.get('POSTGRES_DB', 'pd_master'),
        user=os.environ.get('POSTGRES_USER', 'pd_admin'),
        password=os.environ.get('POSTGRES_PASSWORD'),
        host=os.environ.get('POSTGRES_HOST', 'localhost'),
        port=int(os.environ.get('POSTGRES_PORT', '5432')),
        max_connections=20,
        stale_timeout=300,  # 5 minutes
        timeout=0,
        autoconnect=True,
        autorollback=True  # Automatically rollback failed transactions
    )
else:
    # Keep SQLite for local development/testing
    db = SqliteDatabase(
        'storage/pd_master.db',
        pragmas={
            'journal_mode': 'wal',
            'cache_size': -1 * 64000,
            'synchronous': 0
        }
    )

Environment Variables Required:

DATABASE_TYPE=postgresql
POSTGRES_HOST=localhost  # or production host
POSTGRES_DB=pd_master
POSTGRES_USER=pd_admin
POSTGRES_PASSWORD=secure_password_here
POSTGRES_PORT=5432

Estimated Effort: 1 hour

---

4. Migration Script Updates (BLOCKING - Must Fix)

Location: db_migrations.py (Line 5)

Current Code:

from playhouse.migrate import *
migrator = SqliteMigrator(db_engine.db)

Required Changes:

from playhouse.migrate import *

# Determine which migrator to use based on database type
if isinstance(db_engine.db, PostgresqlDatabase):
    migrator = PostgresqlMigrator(db_engine.db)
else:
    migrator = SqliteMigrator(db_engine.db)

Estimated Effort: 30 minutes

---

Detailed Code Changes Required

Priority 1: CRITICAL (Must Fix Before Migration)

File: app/db_engine.py

Change 1: Database Connection

• Lines 11-18: Replace SQLite configuration with environment-based PostgreSQL/SQLite
• Add import: from playhouse.pool import PooledPostgresqlDatabase
• See detailed code in Section 3

Change 2: Add table_name to All Models

• Add table_name = 'model_name_lowercase' to Meta class for all 40 models
• Use snake_case for multi-word models (e.g., BattingCard → 'batting_card')

Model Name Mapping:

Current → 'current'
Rarity → 'rarity'
Event → 'event'
Cardset → 'cardset'
MlbPlayer → 'mlb_player'
Player → 'player'
Team → 'team'
PackType → 'pack_type'
Pack → 'pack'
Card → 'card'
Roster → 'roster'
Result → 'result'
BattingStat → 'batting_stat'
PitchingStat → 'pitching_stat'
Award → 'award'
Paperdex → 'paperdex'
Reward → 'reward'
GameRewards → 'game_rewards'
Notification → 'notification'
GauntletReward → 'gauntlet_reward'
GauntletRun → 'gauntlet_run'
BattingCard → 'batting_card'
BattingCardRatings → 'batting_card_ratings'
PitchingCard → 'pitching_card'
PitchingCardRatings → 'pitching_card_ratings'
CardPosition → 'card_position'
StratGame → 'strat_game'
StratPlay → 'strat_play'
Decision → 'decision'
Scouting → 'scouting'
# ... (add remaining models)

File: app/routers_v2/stratplays.py

Change 1: Fix get_batting_totals() GROUP BY (Lines 342-456)

Replace lines 342-456 with conditional SELECT field building:

# Determine which fields to include in SELECT based on group_by parameter
base_select_fields = [
    fn.SUM(StratPlay.pa).alias('sum_pa'),
    fn.SUM(StratPlay.ab).alias('sum_ab'),
    fn.SUM(StratPlay.run).alias('sum_run'),
    fn.SUM(StratPlay.hit).alias('sum_hit'),
    fn.SUM(StratPlay.rbi).alias('sum_rbi'),
    fn.SUM(StratPlay.double).alias('sum_double'),
    fn.SUM(StratPlay.triple).alias('sum_triple'),
    fn.SUM(StratPlay.homerun).alias('sum_hr'),
    fn.SUM(StratPlay.bb).alias('sum_bb'),
    fn.SUM(StratPlay.so).alias('sum_so'),
    fn.SUM(StratPlay.hbp).alias('sum_hbp'),
    fn.SUM(StratPlay.sac).alias('sum_sac'),
    fn.SUM(StratPlay.ibb).alias('sum_ibb'),
    fn.SUM(StratPlay.gidp).alias('sum_gidp'),
    fn.SUM(StratPlay.sb).alias('sum_sb'),
    fn.SUM(StratPlay.cs).alias('sum_cs'),
    fn.SUM(StratPlay.bphr).alias('sum_bphr'),
    fn.SUM(StratPlay.bpfo).alias('sum_bpfo'),
    fn.SUM(StratPlay.bp1b).alias('sum_bp1b'),
    fn.SUM(StratPlay.bplo).alias('sum_bplo'),
    fn.SUM(StratPlay.wpa).alias('sum_wpa'),
    fn.SUM(StratPlay.re24).alias('sum_re24'),
    # ... all other aggregates ...
]

# Conditionally add non-aggregated fields based on group_by
if group_by in ['player', 'playerteam', 'playergame', 'playergtype', 'playerteamgtype']:
    base_select_fields.insert(0, StratPlay.batter)
if group_by in ['team', 'playerteam', 'teamgame', 'playerteamgtype']:
    base_select_fields.append(StratPlay.batter_team)
if group_by in ['playergame', 'teamgame']:
    base_select_fields.append(StratPlay.game)

bat_plays = (
    StratPlay
        .select(*base_select_fields)
        .where((StratPlay.game << season_games) & (StratPlay.batter.is_null(False)))
        .having(fn.SUM(StratPlay.pa) >= min_pa)
)

# Similar pattern for run_plays
run_select_fields = [
    fn.SUM(StratPlay.sb).alias('sum_sb'),
    fn.SUM(StratPlay.cs).alias('sum_cs'),
    fn.SUM(StratPlay.pick_off).alias('sum_pick'),
    fn.SUM(StratPlay.wpa).alias('sum_wpa'),
    fn.SUM(StratPlay.re24).alias('sum_re24')
]

if group_by in ['player', 'playerteam', 'playergame', 'playergtype', 'playerteamgtype']:
    run_select_fields.insert(0, StratPlay.runner)
if group_by in ['team', 'playerteam', 'teamgame', 'playerteamgtype']:
    run_select_fields.append(StratPlay.runner_team)
if group_by in ['playergame', 'teamgame']:
    run_select_fields.append(StratPlay.game)

run_plays = (
    StratPlay
        .select(*run_select_fields)
        .where((StratPlay.game << season_games) & (StratPlay.runner.is_null(False)))
)

Change 2: Fix get_pitching_totals() GROUP BY (Lines 645-733)

Apply same pattern as batting totals - conditionally build SELECT based on group_by

Change 3: Fix get_game_summary() GROUP BY (Lines 911-929)

Review and ensure GROUP BY includes all non-aggregated SELECT fields

File: db_migrations.py

Change: Update Migrator Selection (Line 5)

from playhouse.migrate import *
from app.db_engine import db

# Automatically select correct migrator based on database type
if hasattr(db, '__class__') and 'Postgres' in db.__class__.__name__:
    migrator = PostgresqlMigrator(db)
else:
    migrator = SqliteMigrator(db)

---

Priority 2: HIGH (Test Thoroughly After Migration)

Files with .on_conflict_replace() (15 files)

Potential Issue: PostgreSQL uses different syntax than SQLite for conflict resolution

SQLite: INSERT OR REPLACE INTO ... PostgreSQL: INSERT ... ON CONFLICT ... DO UPDATE SET ...

Peewee Should Handle This Automatically, but test these endpoints carefully:

1. app/routers_v2/players.py:765 - Player.insert_many(batch).on_conflict_replace().execute()
2. app/routers_v2/battingcardratings.py:549 - BattingCardRatings.insert_many(batch).on_conflict_replace().execute()
3. app/routers_v2/battingcards.py:134 - BattingCard.insert_many(batch).on_conflict_replace().execute()
4. app/routers_v2/pitchingcardratings.py:551 - Similar pattern
5. app/routers_v2/pitchingcards.py:135 - Similar pattern
6. app/routers_v2/stratplays.py:1082 - StratPlay.insert_many(batch).on_conflict_replace().execute()
7. Additional UPDATE/INSERT operations in teams.py, cards.py, admin.py

Testing Strategy:

• Create integration tests for each bulk insert operation
• Verify data integrity after conflicts
• Test with duplicate key scenarios

Random Function Usage

Location: app/routers_v2/players.py:236

Current Code:

all_players = Player.select().order_by(fn.Random())

Status: Should work (both SQLite and PostgreSQL use RANDOM()) Action: Test to confirm behavior is identical

---

Priority 3: MEDIUM (Monitor During Migration)

Date/Time Handling

Status: ✅ GOOD - All using Python's datetime library Action: Verify timestamp storage/retrieval works identically

All .execute() Calls

Action: Monitor logs during testing for any SQL errors

---

Migration Strategy

Phase 1: Preparation (Day 1)

1.1 Set Up PostgreSQL Development Environment

# Install PostgreSQL
sudo dnf install postgresql-server postgresql-contrib  # Fedora/RHEL
# OR
sudo apt install postgresql postgresql-contrib  # Debian/Ubuntu

# Initialize database
sudo postgresql-setup --initdb  # Fedora/RHEL
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Create database and user
sudo -u postgres psql
CREATE DATABASE pd_master;
CREATE USER pd_admin WITH PASSWORD 'secure_password';
GRANT ALL PRIVILEGES ON DATABASE pd_master TO pd_admin;
\q

1.2 Install Python PostgreSQL Adapter

pip install psycopg2-binary  # or psycopg2 (requires compilation)

1.3 Create Development Branch

git checkout -b postgres-migration

1.4 Backup Current SQLite Database

cp storage/pd_master.db storage/pd_master_backup_$(date +%Y%m%d).db
sqlite3 storage/pd_master.db .dump > storage/pd_master_backup_$(date +%Y%m%d).sql

---

Phase 2: Code Changes (Day 1-2)

2.1 Update Database Configuration

• Modify app/db_engine.py with environment-based PostgreSQL support
• Add all table_name values to model Meta classes
• Update db_migrations.py with PostgreSQL migrator support

2.2 Fix GROUP BY Queries

• Update app/routers_v2/stratplays.py:
  • Fix get_batting_totals() - conditional SELECT fields
  • Fix get_pitching_totals() - conditional SELECT fields
  • Fix get_game_summary() - verify GROUP BY completeness
• Review app/routers_v2/batstats.py - verify GROUP BY (likely OK)
• Review app/routers_v2/pitstats.py - verify GROUP BY (likely OK)

2.3 Verify Other Routers

• Review all routers for any SQLite-specific code
• Check for any hardcoded SQL that needs updating

Estimated Time: 6-8 hours

---

Phase 3: Data Migration (Day 2)

Option A: SQLite to PostgreSQL Data Transfer (Recommended)

Step 1: Export Schema

# Let Peewee create PostgreSQL tables
DATABASE_TYPE=postgresql python -c "
from app.db_engine import db, Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision, Scouting
db.create_tables([Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision, Scouting])
print('PostgreSQL schema created')
"

Step 2: Create Migration Script

Create scripts/migrate_sqlite_to_postgres.py:

"""
SQLite to PostgreSQL Data Migration Script

Migrates all data from SQLite (storage/pd_master.db) to PostgreSQL.
"""
import os
import logging
from peewee import SqliteDatabase, PostgresqlDatabase
from playhouse.pool import PooledPostgresqlDatabase
from app.db_engine import (
    Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack,
    Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex,
    Reward, GameRewards, Notification, GauntletReward, GauntletRun,
    BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings,
    CardPosition, StratGame, StratPlay, Decision, Scouting
)

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Source database (SQLite)
sqlite_db = SqliteDatabase('storage/pd_master.db')

# Target database (PostgreSQL)
postgres_db = PooledPostgresqlDatabase(
    os.environ.get('POSTGRES_DB', 'pd_master'),
    user=os.environ.get('POSTGRES_USER', 'pd_admin'),
    password=os.environ.get('POSTGRES_PASSWORD'),
    host=os.environ.get('POSTGRES_HOST', 'localhost'),
    port=int(os.environ.get('POSTGRES_PORT', '5432'))
)

# List of models in dependency order (foreign keys last)
MODELS = [
    Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType,
    # Add all models in correct dependency order
]

def migrate_model(model_class, batch_size=1000):
    """Migrate a single model from SQLite to PostgreSQL"""
    logger.info(f"Migrating {model_class.__name__}...")

    # Bind to SQLite to read
    model_class._meta.database = sqlite_db

    total = model_class.select().count()
    logger.info(f"  Total records: {total}")

    migrated = 0

    # Read from SQLite in batches
    for i in range(0, total, batch_size):
        batch = list(model_class.select().offset(i).limit(batch_size).dicts())

        if not batch:
            break

        # Bind to PostgreSQL to write
        model_class._meta.database = postgres_db

        # Insert batch into PostgreSQL
        with postgres_db.atomic():
            model_class.insert_many(batch).execute()

        migrated += len(batch)
        logger.info(f"  Progress: {migrated}/{total}")

    logger.info(f"✓ {model_class.__name__} migration complete")

def main():
    """Run full migration"""
    logger.info("Starting SQLite → PostgreSQL migration")

    # Create PostgreSQL tables
    postgres_db.create_tables(MODELS)
    logger.info("PostgreSQL tables created")

    # Migrate each model
    for model in MODELS:
        try:
            migrate_model(model)
        except Exception as e:
            logger.error(f"✗ Failed to migrate {model.__name__}: {e}")
            raise

    logger.info("Migration complete!")

if __name__ == '__main__':
    main()

Step 3: Run Migration

export DATABASE_TYPE=postgresql
export POSTGRES_HOST=localhost
export POSTGRES_DB=pd_master
export POSTGRES_USER=pd_admin
export POSTGRES_PASSWORD=your_secure_password
export POSTGRES_PORT=5432

python scripts/migrate_sqlite_to_postgres.py

Option B: pg_loader (Alternative)

Can use pgloader for automatic migration, but requires testing.

Estimated Time: 2-4 hours (including verification)

---

Phase 4: Testing (Day 3)

4.1 Unit Testing Strategy

Create tests/test_postgres_migration.py:

"""
PostgreSQL Migration Test Suite

Tests all critical endpoints against PostgreSQL database.
"""
import pytest
from fastapi.testclient import TestClient
from app.main import app

client = TestClient(app)

class TestGroupByQueries:
    """Test all GROUP BY query variations"""

    def test_batting_totals_group_by_player(self):
        """Test batting totals with group_by=player"""
        response = client.get("/api/v2/stratplays/batting?group_by=player&limit=10")
        assert response.status_code == 200
        data = response.json()
        assert len(data) <= 10

    def test_batting_totals_group_by_team(self):
        """Test batting totals with group_by=team"""
        response = client.get("/api/v2/stratplays/batting?group_by=team")
        assert response.status_code == 200

    def test_batting_totals_group_by_playerteam(self):
        """Test batting totals with group_by=playerteam"""
        response = client.get("/api/v2/stratplays/batting?group_by=playerteam&limit=10")
        assert response.status_code == 200

    def test_batting_totals_group_by_playergame(self):
        """Test batting totals with group_by=playergame"""
        response = client.get("/api/v2/stratplays/batting?group_by=playergame&limit=10")
        assert response.status_code == 200

    # Add tests for all group_by variations

    def test_pitching_totals_all_variations(self):
        """Test pitching totals with all group_by variations"""
        variations = ['player', 'team', 'playerteam', 'playergame', 'teamgame',
                     'league', 'gtype', 'playergtype', 'playerteamgtype']
        for variation in variations:
            response = client.get(f"/api/v2/stratplays/pitching?group_by={variation}&limit=5")
            assert response.status_code == 200, f"Failed for group_by={variation}"

class TestConflictResolution:
    """Test .on_conflict_replace() operations"""

    def test_player_bulk_insert(self):
        """Test player bulk insert with conflicts"""
        # Implementation depends on your API
        pass

    def test_card_ratings_insert(self):
        """Test card ratings bulk insert"""
        pass

class TestDataIntegrity:
    """Verify data was migrated correctly"""

    def test_record_counts(self):
        """Verify record counts match SQLite"""
        # Query each table and compare counts
        pass

    def test_random_sampling(self):
        """Sample random records and verify data"""
        pass

# Run with: pytest tests/test_postgres_migration.py -v

4.2 Integration Testing

Test All Endpoints:

# Use Major Domo's test pattern - comprehensive API testing
python tests/comprehensive_api_test.py  # Create based on Major Domo pattern

Expected Results:

• All GROUP BY queries return data without errors
• All bulk insert operations complete successfully
• Data integrity verified (counts, relationships, values)
• Performance comparable or better than SQLite

4.3 Load Testing (Optional)

# Use Apache Bench or similar
ab -n 1000 -c 10 http://localhost:8000/api/v2/players/

Estimated Time: 4-6 hours

---

Phase 5: Production Deployment (Day 4)

5.1 Production PostgreSQL Setup

On Production Server:

# Install PostgreSQL
sudo apt install postgresql postgresql-contrib

# Create production database
sudo -u postgres psql
CREATE DATABASE pd_master_prod;
CREATE USER pd_prod WITH PASSWORD 'strong_production_password';
GRANT ALL PRIVILEGES ON DATABASE pd_master_prod TO pd_prod;
ALTER DATABASE pd_master_prod OWNER TO pd_prod;
\q

# Configure PostgreSQL for production
sudo nano /etc/postgresql/14/main/postgresql.conf
# Set: max_connections = 100
# Set: shared_buffers = 256MB
# Set: effective_cache_size = 1GB
# Set: maintenance_work_mem = 64MB
# Set: checkpoint_completion_target = 0.9

sudo systemctl restart postgresql

5.2 Migrate Production Data

Option A: Direct Migration (Downtime Required)

# Stop application
sudo systemctl stop paper-dynasty-api

# Run migration script
export DATABASE_TYPE=postgresql
export POSTGRES_HOST=localhost
export POSTGRES_DB=pd_master_prod
export POSTGRES_USER=pd_prod
export POSTGRES_PASSWORD=strong_production_password

python scripts/migrate_sqlite_to_postgres.py

# Verify migration
python scripts/verify_migration.py

# Update environment variables
sudo nano /etc/systemd/system/paper-dynasty-api.service
# Add PostgreSQL environment variables

# Start application
sudo systemctl start paper-dynasty-api

Option B: Blue-Green Deployment (Zero Downtime)

• Set up PostgreSQL database
• Migrate data to PostgreSQL while SQLite is still serving traffic
• Switch DNS/load balancer to new PostgreSQL instance
• Monitor for issues, rollback if needed

5.3 Post-Deployment Monitoring

Monitor for 24-48 hours:

• Database connection pool usage
• Query performance (compare to SQLite baseline)
• Error rates in logs
• API response times
• Database CPU/memory usage

Estimated Time: 2-3 hours + monitoring

---

Testing Plan

Unit Tests

Create comprehensive unit tests for:

• All GROUP BY query variations
• Model CRUD operations
• Bulk insert with conflicts
• Transaction rollback behavior

Integration Tests

Test all 30+ API endpoints:

• GET requests with various filters
• POST/PUT/DELETE operations
• Bulk operations
• Edge cases (empty results, null values)

Data Integrity Tests

Verify:

• Record counts match between SQLite and PostgreSQL
• Foreign key relationships maintained
• Data values identical (random sampling)
• Aggregation results consistent

Performance Tests

Compare:

• Query execution times (SQLite vs PostgreSQL)
• API response times
• Concurrent request handling
• Database connection overhead

Success Criteria:

• ✅ 100% of unit tests pass
• ✅ 100% of integration tests pass
• ✅ 100% data integrity verification
• ✅ Performance equal or better than SQLite

---

Rollback Plan

If Issues Discovered During Testing

Action: Fix issues in postgres-migration branch, don't merge to main

If Issues Discovered After Production Deployment

Immediate Rollback Steps:

# Stop application
sudo systemctl stop paper-dynasty-api

# Restore environment to SQLite
sudo nano /etc/systemd/system/paper-dynasty-api.service
# Remove DATABASE_TYPE=postgresql
# OR set DATABASE_TYPE=sqlite

# Start application
sudo systemctl start paper-dynasty-api

# Verify SQLite is working
curl http://localhost:8000/api/v2/current

Data Recovery:

• SQLite database backup exists: storage/pd_master_backup_YYYYMMDD.db
• SQL dump exists: storage/pd_master_backup_YYYYMMDD.sql
• Recent data added to PostgreSQL can be extracted with pg_dump

Estimated Rollback Time: 5-10 minutes

---

Timeline

Day 1: Preparation & Initial Code Changes (8 hours)

• ✅ Hour 1-2: Set up PostgreSQL development environment
• ✅ Hour 2-3: Create development branch, backup SQLite database
• ✅ Hour 3-5: Update database configuration in db_engine.py
• ✅ Hour 5-6: Add table_name to all 40 models
• ✅ Hour 6-8: Update db_migrations.py, verify no syntax errors

Day 2: Fix GROUP BY & Data Migration (8 hours)

• ✅ Hour 1-3: Fix GROUP BY in stratplays.py (get_batting_totals)
• ✅ Hour 3-4: Fix GROUP BY in stratplays.py (get_pitching_totals)
• ✅ Hour 4-5: Fix GROUP BY in stratplays.py (get_game_summary)
• ✅ Hour 5-6: Create data migration script
• ✅ Hour 6-8: Run migration, verify data transferred correctly

Day 3: Testing (8 hours)

• ✅ Hour 1-2: Create unit tests for GROUP BY queries
• ✅ Hour 2-4: Run comprehensive integration tests on all endpoints
• ✅ Hour 4-6: Test all .on_conflict_replace() operations
• ✅ Hour 6-7: Performance testing and comparison
• ✅ Hour 7-8: Fix any issues discovered, retest

Day 4: Production Deployment (4 hours + monitoring)

• ✅ Hour 1: Set up production PostgreSQL database
• ✅ Hour 2-3: Migrate production data (with downtime)
• ✅ Hour 3-4: Deploy application, verify functionality
• ✅ 24-48 hours: Monitor production for issues

Total Estimated Effort: 28 hours (3.5 days) + monitoring

---

Risk Assessment

High Risk Items

Risk: GROUP BY queries fail in production Mitigation: Comprehensive testing of all group_by variations, sister project proven pattern Likelihood: Low (sister project solved this successfully)

Risk: Data migration fails or corrupts data Mitigation: Multiple backups, verification scripts, rollback plan ready Likelihood: Low (well-tested migration pattern)

Risk: Performance degrades after migration Mitigation: Performance testing before deployment, connection pooling configured Likelihood: Very Low (PostgreSQL typically faster for concurrent operations)

Medium Risk Items

Risk: .on_conflict_replace() syntax differences cause issues Mitigation: Peewee should handle this, but test thoroughly Likelihood: Low (Peewee abstracts this)

Risk: Docker/deployment configuration issues Mitigation: Test in staging environment first Likelihood: Medium (new environment variables required)

Low Risk Items

Risk: fn.Random() behaves differently Mitigation: Simple test confirms compatibility Likelihood: Very Low (standard SQL function)

---

Success Metrics

Technical Metrics

• ✅ All 40 models successfully created in PostgreSQL
• ✅ All endpoints return 2XX status codes
• ✅ GROUP BY queries execute without errors
• ✅ Data integrity: 100% record count match
• ✅ Performance: Equal or better response times

Business Metrics

• ✅ Zero data loss during migration
• ✅ Minimal downtime (< 1 hour if using Option A)
• ✅ No user-facing errors after deployment
• ✅ Application stability maintained

---

Appendix

A. Environment Variables Reference

Development:

DATABASE_TYPE=postgresql
POSTGRES_HOST=localhost
POSTGRES_DB=pd_master
POSTGRES_USER=pd_admin
POSTGRES_PASSWORD=dev_password_here
POSTGRES_PORT=5432
LOG_LEVEL=INFO

Production:

DATABASE_TYPE=postgresql
POSTGRES_HOST=production_host_or_ip
POSTGRES_DB=pd_master_prod
POSTGRES_USER=pd_prod
POSTGRES_PASSWORD=strong_production_password
POSTGRES_PORT=5432
LOG_LEVEL=WARNING

B. Quick Reference: SQLite vs PostgreSQL

Feature	SQLite	PostgreSQL
Connection	File-based	Network-based
Concurrency	Limited (single writer)	Excellent (MVCC)
GROUP BY	Permissive	Strict (SQL standard)
AUTOINCREMENT	AUTOINCREMENT keyword	SERIAL/BIGSERIAL
RANDOM	RANDOM()	RANDOM() (same)
PRAGMA	Yes	No
INSERT OR REPLACE	Native	ON CONFLICT DO UPDATE

C. Sister Project Success Metrics

From Major Domo migration (August 2025):

• Success Rate: 74.6% immediate compatibility
• Data Loss: 0%
• Migration Time: ~3 days
• Critical Issues: 0 (all failures were environment differences)
• Production Ready: Yes

---

Next Steps

1. Review this plan with team/stakeholders
2. Set migration date (recommend 1-2 weeks out)
3. Create tickets for each phase
4. Begin Day 1 preparation tasks
5. Schedule testing time with QA team (if applicable)

---

Document Version: 1.0 Last Updated: 2025-11-07 Author: Claude (AI Assistant) Based On: Major Domo PostgreSQL migration (August 2025)