This commit adds support for the new Retrosheet CSV format and resolves
multiple data processing issues in retrosheet_data.py.
New Features:
- Created retrosheet_transformer.py with smart caching system
- Transforms new Retrosheet CSV format to legacy format
- Checks file timestamps to avoid redundant transformations
- Caches normalized data for instant subsequent loads (~5s → <1s)
- Handles column mapping: gid→game_id, bathand→batter_hand, etc.
- Derives event_type from multiple boolean columns
- Converts handedness values R/L → r/l
- Explicitly sets string dtypes for hit_val, hit_location, batted_ball_type
Configuration Updates:
- Updated retrosheet_data.py for 2005 season data
- START_DATE: 19980301 → 20050403 (2005 Opening Day)
- END_DATE: 19980430 → 20051002 (2005 Regular Season End)
- SEASON_PCT: 28/162 → 162/162 (full season)
- MIN_PA_VL/VR: 20/40 → 50/75 (full season minimums)
- CARDSET_ID: Updated for 2005 cardsets
- EVENTS_FILENAME: Updated to use retrosheets_events_2005.csv
Bug Fixes:
1. Multi-team player duplicates
- Players traded during season had duplicate rows (one per team + combined)
- Added filtering to keep only combined totals (2TM, 3TM, etc.)
- Prevents duplicate key_bbref values in ratings dataframes
2. Column name conflicts
- Fixed Tm column conflict when merging periph_stats and defense_p
- Drop duplicate Tm from defense data before merge
3. Pitcher rating calculations (pitchers/calcs_pitcher.py)
- Fixed "truth value is ambiguous" error in min() comparisons
- Explicitly convert pandas values to float before min() operations
4. Dictionary column corruption in ratings
- Fixed ratings_vL and ratings_vR corruption during DataFrame merges
- Only merge specific columns (key_bbref, player_id, card_id) instead of full DataFrame
- Removed unnecessary .set_index() calls from post_batting_cards() and post_pitching_cards()
Documentation:
- Updated CLAUDE.md with comprehensive troubleshooting section
- Added Retrosheet transformation documentation
- Documented defense CSV requirements and column naming
- Added configuration checklist for retrosheet_data.py
- Documented common issues: multi-team players, dictionary corruption, string types
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit adds default OPS value constants and type hints to key functions,
improving code documentation and IDE support.
## Changes Made
1. **Add default OPS constants** (creation_helpers.py)
- DEFAULT_BATTER_OPS: Default OPS by rarity (1-5)
- DEFAULT_STARTER_OPS: Default OPS-against for starters (99, 1-5)
- DEFAULT_RELIEVER_OPS: Default OPS-against for relievers (99, 1-5)
- Comprehensive comments explaining usage
- Single source of truth for fallback values
2. **Update batters/creation.py**
- Import DEFAULT_BATTER_OPS
- Replace 6 hardcoded if-checks with clean loop over constants
- Add type hints to post_player_updates function
- Import Dict from typing
3. **Update pitchers/creation.py**
- Import DEFAULT_STARTER_OPS and DEFAULT_RELIEVER_OPS
- Replace 12 hardcoded if-checks with clean loops over constants
- Add type hints to post_player_updates function
- Import Dict from typing
4. **Add typing import** (creation_helpers.py)
- Import Dict, List, Tuple, Optional for type hints
- Enables type hints throughout helper functions
## Impact
### Before
```python
# Scattered hardcoded values (batters)
if 1 not in average_ops:
average_ops[1] = 1.066
if 2 not in average_ops:
average_ops[2] = 0.938
# ... 4 more if-checks
# Scattered hardcoded values (pitchers)
if 99 not in sp_average_ops:
sp_average_ops[99] = 0.388
# ... 5 more if-checks for starters
# ... 6 more if-checks for relievers
```
### After
```python
# Clean, data-driven approach (batters)
for rarity, default_ops in DEFAULT_BATTER_OPS.items():
if rarity not in average_ops:
average_ops[rarity] = default_ops
# Clean, data-driven approach (pitchers)
for rarity, default_ops in DEFAULT_STARTER_OPS.items():
if rarity not in sp_average_ops:
sp_average_ops[rarity] = default_ops
for rarity, default_ops in DEFAULT_RELIEVER_OPS.items():
if rarity not in rp_average_ops:
rp_average_ops[rarity] = default_ops
```
### Benefits
✅ Eliminates 18 if-checks across batters and pitchers
✅ Single source of truth for default OPS values
✅ Easy to modify values (change constant, not scattered code)
✅ Self-documenting with clear constant names and comments
✅ Type hints improve IDE support and catch errors early
✅ Function signatures now document expected types
✅ Consistent with other recent refactorings
## Test Results
✅ 42/42 tests pass
✅ All existing functionality preserved
✅ 100% backward compatible
## Files Modified
- creation_helpers.py: +35 lines (3 constants + typing import)
- batters/creation.py: -4 lines net (cleaner code + type hints)
- pitchers/creation.py: -8 lines net (cleaner code + type hints)
**Net change:** More constants, less scattered magic numbers, better types.
Part of ongoing refactoring to reduce code fragility.
