paper-dynasty-database/MIGRATION_PROGRESS.md

PostgreSQL Migration Progress

Branch: postgres-migration Date Started: 2025-11-07 Status: ✅ Code Changes Complete - Ready for Testing

---

✅ Completed Tasks

1. Database Configuration (app/db_engine.py)

• ✅ Added environment-based PostgreSQL/SQLite configuration
• ✅ Implemented PooledPostgresqlDatabase for PostgreSQL connections
• ✅ Maintained backward compatibility with SQLite for local development
• ✅ Connection pooling configured (20 max connections, 5-minute stale timeout)

Environment Variables Required:

DATABASE_TYPE=postgresql  # or 'sqlite' for local dev
POSTGRES_HOST=localhost
POSTGRES_DB=pd_master
POSTGRES_USER=pd_admin
POSTGRES_PASSWORD=your_secure_password
POSTGRES_PORT=5432

2. Model Table Names (app/db_engine.py)

• ✅ Added table_name to all 30 models' Meta classes
• ✅ Ensures explicit table naming for PostgreSQL compatibility

Models Updated:

Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack,
Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward,
GameRewards, Notification, GauntletReward, GauntletRun, BattingCard,
BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition,
StratGame, StratPlay, Decision

3. Migration Script (db_migrations.py)

• ✅ Updated to auto-detect database type
• ✅ Automatically selects PostgresqlMigrator or SqliteMigrator

4. GROUP BY Query Fixes (app/routers_v2/stratplays.py)

• ✅ Fixed get_batting_totals() - conditionally builds SELECT fields based on group_by parameter
• ✅ Fixed get_pitching_totals() - same conditional SELECT pattern
• ✅ Ensures all non-aggregated SELECT fields are in GROUP BY clause

Pattern Applied:

# Build SELECT fields conditionally based on group_by
base_select_fields = [fn.SUM(...), fn.COUNT(...), ...]  # All aggregates

# Add non-aggregated fields only if they'll be in GROUP BY
if group_by in ['player', 'playerteam', 'playergame']:
    base_select_fields.insert(0, StratPlay.batter)
if group_by in ['team', 'playerteam', 'teamgame']:
    base_select_fields.append(StratPlay.batter_team)
# etc...

query = StratPlay.select(*base_select_fields).where(...)

5. Dependencies (requirements.txt)

• ✅ Added psycopg2-binary for PostgreSQL adapter

---

📝 Git Commit Summary

git log --oneline
0d123be CLAUDE: Add psycopg2-binary to requirements for PostgreSQL support
1517283 CLAUDE: Fix GROUP BY queries for PostgreSQL compatibility
f6e8aa7 CLAUDE: Add PostgreSQL support and table names to models

Total Changes:

• 3 commits
• 5 files modified
• ~1,300 lines changed (mostly in db_engine.py and stratplays.py)

---

🔍 Next Steps (Not Yet Started)

Phase 1: Local Testing (Estimated: 2-3 hours)

1.1 Install PostgreSQL Locally

# Fedora/RHEL
sudo dnf install postgresql-server postgresql-contrib
sudo postgresql-setup --initdb
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Create database
sudo -u postgres psql
CREATE DATABASE pd_master_test;
CREATE USER pd_admin WITH PASSWORD 'test_password';
GRANT ALL PRIVILEGES ON DATABASE pd_master_test TO pd_admin;
\q

1.2 Install Python Dependencies

pip install psycopg2-binary

1.3 Test Database Connection

# Set environment variables
export DATABASE_TYPE=postgresql
export POSTGRES_HOST=localhost
export POSTGRES_DB=pd_master_test
export POSTGRES_USER=pd_admin
export POSTGRES_PASSWORD=test_password
export POSTGRES_PORT=5432

# Test connection
python -c "from app.db_engine import db; db.connect(); print('✓ PostgreSQL connection successful'); db.close()"

1.4 Create PostgreSQL Schema

# Let Peewee create all tables
python -c "
from app.db_engine import db, Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision

db.connect()
db.create_tables([
    Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack,
    Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward,
    GameRewards, Notification, GauntletReward, GauntletRun, BattingCard,
    BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition,
    StratGame, StratPlay, Decision
])
print('✓ PostgreSQL schema created')
db.close()
"

1.5 Test GROUP BY Queries

Create test script: tests/test_postgres_groupby.py

"""Test all GROUP BY variations work with PostgreSQL"""
from fastapi.testclient import TestClient
from app.main import app

client = TestClient(app)

def test_batting_totals_all_groupby_modes():
    """Test all group_by variations for batting totals"""
    group_by_modes = ['player', 'team', 'playerteam', 'playergame',
                     'teamgame', 'league', 'gtype', 'playergtype',
                     'playerteamgtype']

    for mode in group_by_modes:
        print(f"Testing group_by={mode}...")
        response = client.get(f"/api/v2/stratplays/batting?group_by={mode}&limit=5")
        assert response.status_code == 200, f"Failed for group_by={mode}"
        print(f"  ✓ {mode} passed")

    print("\n✓ All batting GROUP BY modes passed!")

def test_pitching_totals_all_groupby_modes():
    """Test all group_by variations for pitching totals"""
    group_by_modes = ['player', 'team', 'playerteam', 'playergame',
                     'teamgame', 'league', 'gtype', 'playergtype',
                     'playerteamgtype']

    for mode in group_by_modes:
        print(f"Testing group_by={mode}...")
        response = client.get(f"/api/v2/stratplays/pitching?group_by={mode}&limit=5")
        assert response.status_code == 200, f"Failed for group_by={mode}"
        print(f"  ✓ {mode} passed")

    print("\n✓ All pitching GROUP BY modes passed!")

if __name__ == '__main__':
    test_batting_totals_all_groupby_modes()
    test_pitching_totals_all_groupby_modes()
    print("\n🎉 All PostgreSQL GROUP BY tests passed!")

Run tests:

export DATABASE_TYPE=postgresql
# ... set other POSTGRES_* env vars ...
python tests/test_postgres_groupby.py

---

Phase 2: Data Migration (Estimated: 2-4 hours)

2.1 Create Data Migration Script

File: scripts/migrate_sqlite_to_postgres.py

Already outlined in POSTGRES_MIGRATION_PLAN.md (lines 380-449).

Key steps:

1. Connect to both SQLite and PostgreSQL
2. Iterate through models in dependency order
3. Read from SQLite in batches (1000 records)
4. Write to PostgreSQL with transactions
5. Verify record counts match

2.2 Run Migration

export DATABASE_TYPE=sqlite  # Read from SQLite
python scripts/migrate_sqlite_to_postgres.py

2.3 Verify Migration

# Compare record counts
python scripts/verify_migration.py

---

Phase 3: Integration Testing (Estimated: 4-6 hours)

3.1 Test All API Endpoints

• Test all 30+ routers in app/routers_v2/
• Verify data integrity
• Compare responses with SQLite baseline

3.2 Test Bulk Operations

• Test all .on_conflict_replace() operations
• Verify conflict resolution works correctly
• Test in files:
  • app/routers_v2/players.py
  • app/routers_v2/battingcardratings.py
  • app/routers_v2/battingcards.py
  • app/routers_v2/pitchingcardratings.py
  • app/routers_v2/pitchingcards.py
  • app/routers_v2/stratplays.py

3.3 Load Testing (Optional)

# Test concurrent requests
ab -n 1000 -c 10 http://localhost:8000/api/v2/players/

---

Phase 4: Production Deployment (Estimated: 2-3 hours)

4.1 Production PostgreSQL Setup

• Install PostgreSQL on production server
• Configure production database
• Set up backups

4.2 Deploy Application

• Update environment variables
• Restart application
• Monitor logs

4.3 Post-Deployment Monitoring

• Monitor for 24-48 hours
• Check error rates
• Verify performance metrics

---

📊 Risk Assessment

Low Risk ✅

• ✅ Database configuration (tested pattern from sister project)
• ✅ Model table names (straightforward addition)
• ✅ GROUP BY fixes (proven pattern from Major Domo)
• ✅ Migration script auto-detection (simple logic)

Medium Risk ⚠️

• ⚠️ Data migration (requires careful testing)
• ⚠️ .on_conflict_replace() operations (Peewee should handle, but test)
• ⚠️ Production deployment timing (requires downtime)

Mitigations

• Comprehensive testing before production
• SQLite backup before migration
• Rollback plan ready (documented in POSTGRES_MIGRATION_PLAN.md)
• Can continue using SQLite if issues arise

---

🎯 Success Criteria

Before merging to main:

• PostgreSQL connection established and working
• All models created successfully in PostgreSQL
• All GROUP BY query variations tested and passing
• Data migration script tested with sample data
• All API endpoints return 200 status codes
• Data integrity verified (counts, relationships, values)
• Performance equal or better than SQLite

---

📚 Reference Documents

1. POSTGRES_MIGRATION_PLAN.md - Comprehensive 62-page migration guide
2. Sister Project: /mnt/NV2/Development/major-domo/database/
   • MIGRATION_TEST_ANALYSIS_20250819.md - Test results (74.6% immediate success)
   • CLAUDE.md - Production PostgreSQL configuration

---

🔧 Rollback Plan

If issues are discovered:

During Testing (Before Merge)

• Switch back to SQLite by setting DATABASE_TYPE=sqlite
• Fix issues in postgres-migration branch
• Re-test

After Production Deployment

See POSTGRES_MIGRATION_PLAN.md Section: "Rollback Plan" (lines 558-580)

Quick rollback:

# Stop application
sudo systemctl stop paper-dynasty-api

# Revert to SQLite
export DATABASE_TYPE=sqlite

# Start application
sudo systemctl start paper-dynasty-api

SQLite backup locations:

• storage/pd_master_backup_YYYYMMDD.db
• storage/pd_master_backup_YYYYMMDD.sql

---

📝 Notes

Code Quality

• All code changes follow sister project's proven patterns
• Added comprehensive comments explaining PostgreSQL requirements
• Maintained backward compatibility with SQLite

Performance Considerations

• Connection pooling configured (20 max connections)
• Peewee ORM handles most query optimizations
• PostgreSQL should perform better under concurrent load

Lessons from Sister Project (Major Domo)

• 100% of core endpoints working after migration
• All failures were environment differences, not code issues
• GROUP BY fixes were the only critical changes needed
• Data integrity maintained perfectly

---

Last Updated: 2025-11-07 Branch Status: Ready for testing Next Action: Begin Phase 1 (Local Testing)