CLAUDE: Add migration progress tracking document

MIGRATION_PROGRESS.md

@@ -0,0 +1,364 @@
+# PostgreSQL Migration Progress
+
+**Branch:** `postgres-migration`
+**Date Started:** 2025-11-07
+**Status:** ✅ **Code Changes Complete - Ready for Testing**
+
+---
+
+## ✅ Completed Tasks
+
+### 1. Database Configuration (app/db_engine.py)
+- ✅ Added environment-based PostgreSQL/SQLite configuration
+- ✅ Implemented `PooledPostgresqlDatabase` for PostgreSQL connections
+- ✅ Maintained backward compatibility with SQLite for local development
+- ✅ Connection pooling configured (20 max connections, 5-minute stale timeout)
+
+**Environment Variables Required:**
+```bash
+DATABASE_TYPE=postgresql  # or 'sqlite' for local dev
+POSTGRES_HOST=localhost
+POSTGRES_DB=pd_master
+POSTGRES_USER=pd_admin
+POSTGRES_PASSWORD=your_secure_password
+POSTGRES_PORT=5432
+```
+
+### 2. Model Table Names (app/db_engine.py)
+- ✅ Added `table_name` to all 30 models' Meta classes
+- ✅ Ensures explicit table naming for PostgreSQL compatibility
+
+**Models Updated:**
+```
+Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack,
+Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward,
+GameRewards, Notification, GauntletReward, GauntletRun, BattingCard,
+BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition,
+StratGame, StratPlay, Decision
+```
+
+### 3. Migration Script (db_migrations.py)
+- ✅ Updated to auto-detect database type
+- ✅ Automatically selects `PostgresqlMigrator` or `SqliteMigrator`
+
+### 4. GROUP BY Query Fixes (app/routers_v2/stratplays.py)
+- ✅ Fixed `get_batting_totals()` - conditionally builds SELECT fields based on `group_by` parameter
+- ✅ Fixed `get_pitching_totals()` - same conditional SELECT pattern
+- ✅ Ensures all non-aggregated SELECT fields are in GROUP BY clause
+
+**Pattern Applied:**
+```python
+# Build SELECT fields conditionally based on group_by
+base_select_fields = [fn.SUM(...), fn.COUNT(...), ...]  # All aggregates
+
+# Add non-aggregated fields only if they'll be in GROUP BY
+if group_by in ['player', 'playerteam', 'playergame']:
+    base_select_fields.insert(0, StratPlay.batter)
+if group_by in ['team', 'playerteam', 'teamgame']:
+    base_select_fields.append(StratPlay.batter_team)
+# etc...
+
+query = StratPlay.select(*base_select_fields).where(...)
+```
+
+### 5. Dependencies (requirements.txt)
+- ✅ Added `psycopg2-binary` for PostgreSQL adapter
+
+---
+
+## 📝 Git Commit Summary
+
+```bash
+git log --oneline
+0d123be CLAUDE: Add psycopg2-binary to requirements for PostgreSQL support
+1517283 CLAUDE: Fix GROUP BY queries for PostgreSQL compatibility
+f6e8aa7 CLAUDE: Add PostgreSQL support and table names to models
+```
+
+**Total Changes:**
+- 3 commits
+- 5 files modified
+- ~1,300 lines changed (mostly in db_engine.py and stratplays.py)
+
+---
+
+## 🔍 Next Steps (Not Yet Started)
+
+### Phase 1: Local Testing (Estimated: 2-3 hours)
+
+#### 1.1 Install PostgreSQL Locally
+```bash
+# Fedora/RHEL
+sudo dnf install postgresql-server postgresql-contrib
+sudo postgresql-setup --initdb
+sudo systemctl start postgresql
+sudo systemctl enable postgresql
+
+# Create database
+sudo -u postgres psql
+CREATE DATABASE pd_master_test;
+CREATE USER pd_admin WITH PASSWORD 'test_password';
+GRANT ALL PRIVILEGES ON DATABASE pd_master_test TO pd_admin;
+\q
+```
+
+#### 1.2 Install Python Dependencies
+```bash
+pip install psycopg2-binary
+```
+
+#### 1.3 Test Database Connection
+```bash
+# Set environment variables
+export DATABASE_TYPE=postgresql
+export POSTGRES_HOST=localhost
+export POSTGRES_DB=pd_master_test
+export POSTGRES_USER=pd_admin
+export POSTGRES_PASSWORD=test_password
+export POSTGRES_PORT=5432
+
+# Test connection
+python -c "from app.db_engine import db; db.connect(); print('✓ PostgreSQL connection successful'); db.close()"
+```
+
+#### 1.4 Create PostgreSQL Schema
+```bash
+# Let Peewee create all tables
+python -c "
+from app.db_engine import db, Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack, Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward, GameRewards, Notification, GauntletReward, GauntletRun, BattingCard, BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition, StratGame, StratPlay, Decision
+
+db.connect()
+db.create_tables([
+    Current, Rarity, Event, Cardset, MlbPlayer, Player, Team, PackType, Pack,
+    Card, Roster, Result, BattingStat, PitchingStat, Award, Paperdex, Reward,
+    GameRewards, Notification, GauntletReward, GauntletRun, BattingCard,
+    BattingCardRatings, PitchingCard, PitchingCardRatings, CardPosition,
+    StratGame, StratPlay, Decision
+])
+print('✓ PostgreSQL schema created')
+db.close()
+"
+```
+
+#### 1.5 Test GROUP BY Queries
+**Create test script:** `tests/test_postgres_groupby.py`
+```python
+"""Test all GROUP BY variations work with PostgreSQL"""
+from fastapi.testclient import TestClient
+from app.main import app
+
+client = TestClient(app)
+
+def test_batting_totals_all_groupby_modes():
+    """Test all group_by variations for batting totals"""
+    group_by_modes = ['player', 'team', 'playerteam', 'playergame',
+                     'teamgame', 'league', 'gtype', 'playergtype',
+                     'playerteamgtype']
+
+    for mode in group_by_modes:
+        print(f"Testing group_by={mode}...")
+        response = client.get(f"/api/v2/stratplays/batting?group_by={mode}&limit=5")
+        assert response.status_code == 200, f"Failed for group_by={mode}"
+        print(f"  ✓ {mode} passed")
+
+    print("\n✓ All batting GROUP BY modes passed!")
+
+def test_pitching_totals_all_groupby_modes():
+    """Test all group_by variations for pitching totals"""
+    group_by_modes = ['player', 'team', 'playerteam', 'playergame',
+                     'teamgame', 'league', 'gtype', 'playergtype',
+                     'playerteamgtype']
+
+    for mode in group_by_modes:
+        print(f"Testing group_by={mode}...")
+        response = client.get(f"/api/v2/stratplays/pitching?group_by={mode}&limit=5")
+        assert response.status_code == 200, f"Failed for group_by={mode}"
+        print(f"  ✓ {mode} passed")
+
+    print("\n✓ All pitching GROUP BY modes passed!")
+
+if __name__ == '__main__':
+    test_batting_totals_all_groupby_modes()
+    test_pitching_totals_all_groupby_modes()
+    print("\n🎉 All PostgreSQL GROUP BY tests passed!")
+```
+
+**Run tests:**
+```bash
+export DATABASE_TYPE=postgresql
+# ... set other POSTGRES_* env vars ...
+python tests/test_postgres_groupby.py
+```
+
+---
+
+### Phase 2: Data Migration (Estimated: 2-4 hours)
+
+#### 2.1 Create Data Migration Script
+
+**File:** `scripts/migrate_sqlite_to_postgres.py`
+
+Already outlined in `POSTGRES_MIGRATION_PLAN.md` (lines 380-449).
+
+Key steps:
+1. Connect to both SQLite and PostgreSQL
+2. Iterate through models in dependency order
+3. Read from SQLite in batches (1000 records)
+4. Write to PostgreSQL with transactions
+5. Verify record counts match
+
+#### 2.2 Run Migration
+```bash
+export DATABASE_TYPE=sqlite  # Read from SQLite
+python scripts/migrate_sqlite_to_postgres.py
+```
+
+#### 2.3 Verify Migration
+```bash
+# Compare record counts
+python scripts/verify_migration.py
+```
+
+---
+
+### Phase 3: Integration Testing (Estimated: 4-6 hours)
+
+#### 3.1 Test All API Endpoints
+- Test all 30+ routers in `app/routers_v2/`
+- Verify data integrity
+- Compare responses with SQLite baseline
+
+#### 3.2 Test Bulk Operations
+- Test all `.on_conflict_replace()` operations
+- Verify conflict resolution works correctly
+- Test in files:
+  - `app/routers_v2/players.py`
+  - `app/routers_v2/battingcardratings.py`
+  - `app/routers_v2/battingcards.py`
+  - `app/routers_v2/pitchingcardratings.py`
+  - `app/routers_v2/pitchingcards.py`
+  - `app/routers_v2/stratplays.py`
+
+#### 3.3 Load Testing (Optional)
+```bash
+# Test concurrent requests
+ab -n 1000 -c 10 http://localhost:8000/api/v2/players/
+```
+
+---
+
+### Phase 4: Production Deployment (Estimated: 2-3 hours)
+
+#### 4.1 Production PostgreSQL Setup
+- Install PostgreSQL on production server
+- Configure production database
+- Set up backups
+
+#### 4.2 Deploy Application
+- Update environment variables
+- Restart application
+- Monitor logs
+
+#### 4.3 Post-Deployment Monitoring
+- Monitor for 24-48 hours
+- Check error rates
+- Verify performance metrics
+
+---
+
+## 📊 Risk Assessment
+
+### Low Risk ✅
+- ✅ Database configuration (tested pattern from sister project)
+- ✅ Model table names (straightforward addition)
+- ✅ GROUP BY fixes (proven pattern from Major Domo)
+- ✅ Migration script auto-detection (simple logic)
+
+### Medium Risk ⚠️
+- ⚠️ Data migration (requires careful testing)
+- ⚠️ `.on_conflict_replace()` operations (Peewee should handle, but test)
+- ⚠️ Production deployment timing (requires downtime)
+
+### Mitigations
+- Comprehensive testing before production
+- SQLite backup before migration
+- Rollback plan ready (documented in POSTGRES_MIGRATION_PLAN.md)
+- Can continue using SQLite if issues arise
+
+---
+
+## 🎯 Success Criteria
+
+Before merging to `main`:
+- [ ] PostgreSQL connection established and working
+- [ ] All models created successfully in PostgreSQL
+- [ ] All GROUP BY query variations tested and passing
+- [ ] Data migration script tested with sample data
+- [ ] All API endpoints return 200 status codes
+- [ ] Data integrity verified (counts, relationships, values)
+- [ ] Performance equal or better than SQLite
+
+---
+
+## 📚 Reference Documents
+
+1. **POSTGRES_MIGRATION_PLAN.md** - Comprehensive 62-page migration guide
+2. **Sister Project:** `/mnt/NV2/Development/major-domo/database/`
+   - **MIGRATION_TEST_ANALYSIS_20250819.md** - Test results (74.6% immediate success)
+   - **CLAUDE.md** - Production PostgreSQL configuration
+
+---
+
+## 🔧 Rollback Plan
+
+If issues are discovered:
+
+### During Testing (Before Merge)
+- Switch back to SQLite by setting `DATABASE_TYPE=sqlite`
+- Fix issues in `postgres-migration` branch
+- Re-test
+
+### After Production Deployment
+See `POSTGRES_MIGRATION_PLAN.md` Section: "Rollback Plan" (lines 558-580)
+
+Quick rollback:
+```bash
+# Stop application
+sudo systemctl stop paper-dynasty-api
+
+# Revert to SQLite
+export DATABASE_TYPE=sqlite
+
+# Start application
+sudo systemctl start paper-dynasty-api
+```
+
+SQLite backup locations:
+- `storage/pd_master_backup_YYYYMMDD.db`
+- `storage/pd_master_backup_YYYYMMDD.sql`
+
+---
+
+## 📝 Notes
+
+### Code Quality
+- All code changes follow sister project's proven patterns
+- Added comprehensive comments explaining PostgreSQL requirements
+- Maintained backward compatibility with SQLite
+
+### Performance Considerations
+- Connection pooling configured (20 max connections)
+- Peewee ORM handles most query optimizations
+- PostgreSQL should perform better under concurrent load
+
+### Lessons from Sister Project (Major Domo)
+- 100% of core endpoints working after migration
+- All failures were environment differences, not code issues
+- GROUP BY fixes were the only critical changes needed
+- Data integrity maintained perfectly
+
+---
+
+**Last Updated:** 2025-11-07
+**Branch Status:** Ready for testing
+**Next Action:** Begin Phase 1 (Local Testing)