# Data Layer - External API Integration & Caching

## Overview

The data layer provides external data integration for the Paper Dynasty game engine. It handles communication with league REST APIs to fetch team rosters, player data, and submit completed game results.

**Status**: 🚧 **NOT YET IMPLEMENTED** - This directory is currently empty and awaits implementation.

**Purpose**:
- Fetch team/roster data from league-specific REST APIs (SBA and PD)
- Retrieve detailed player/card information
- Submit completed game results to league systems
- Cache frequently accessed data to reduce API calls
- Abstract API differences between leagues

## Planned Architecture

```
app/data/
├── __init__.py              # Public API exports
├── api_client.py            # Base API client with HTTP operations
├── sba_client.py            # SBA League API wrapper
├── pd_client.py             # PD League API wrapper
└── cache.py                 # Optional caching layer (Redis or in-memory)
```

## Integration Points

### With Game Engine
```python
# Game engine needs player data at game start
from app.data import get_api_client

# Get league-specific client
api_client = get_api_client(league_id="sba")

# Fetch roster for game
roster = await api_client.get_team_roster(team_id=123)

# Create lineup from roster data
lineup = create_lineup_from_roster(roster)
```

### With Database Layer
```python
# Store fetched data in database
from app.database.operations import DatabaseOperations

db_ops = DatabaseOperations()

# Fetch and persist player data
player_data = await api_client.get_player(player_id=456)
await db_ops.store_player_metadata(player_data)
```

### With Player Models
```python
# Parse API responses into typed player models
from app.models import SbaPlayer, PdPlayer

# SBA league
sba_data = await sba_client.get_player(player_id=123)
player = SbaPlayer.from_api_response(sba_data)

# PD league (with scouting data)
pd_data = await pd_client.get_player(player_id=456)
batting_data = await pd_client.get_batting_card(player_id=456)
pitching_data = await pd_client.get_pitching_card(player_id=456)
player = PdPlayer.from_api_response(pd_data, batting_data, pitching_data)
```

## API Clients Design

### Base API Client Pattern

**Location**: `app/data/api_client.py` (not yet created)

**Purpose**: Abstract HTTP client with common patterns for all league APIs.

**Key Features**:
- Async HTTP requests using `httpx` or `aiohttp`
- Automatic retry logic with exponential backoff
- Request/response logging
- Error handling and custom exceptions
- Authentication header management
- Rate limiting protection

**Example Implementation**:
```python
import httpx
import logging
from typing import Dict, Any, Optional
from abc import ABC, abstractmethod

logger = logging.getLogger(f'{__name__}.BaseApiClient')

class ApiClientError(Exception):
    """Base exception for API client errors"""
    pass

class ApiConnectionError(ApiClientError):
    """Raised when API connection fails"""
    pass

class ApiAuthenticationError(ApiClientError):
    """Raised when authentication fails"""
    pass

class ApiNotFoundError(ApiClientError):
    """Raised when resource not found (404)"""
    pass

class BaseApiClient(ABC):
    """Abstract base class for league API clients

    Provides common HTTP operations and error handling.
    Subclasses implement league-specific endpoints.
    """

    def __init__(self, base_url: str, api_key: Optional[str] = None):
        self.base_url = base_url.rstrip('/')
        self.api_key = api_key
        self._client: Optional[httpx.AsyncClient] = None

    async def __aenter__(self):
        """Async context manager entry"""
        self._client = httpx.AsyncClient(
            base_url=self.base_url,
            headers=self._get_headers(),
            timeout=30.0
        )
        return self

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        """Async context manager exit"""
        if self._client:
            await self._client.aclose()

    def _get_headers(self) -> Dict[str, str]:
        """Get request headers including auth"""
        headers = {
            "Accept": "application/json",
            "Content-Type": "application/json"
        }
        if self.api_key:
            headers["Authorization"] = f"Bearer {self.api_key}"
        return headers

    async def _get(self, endpoint: str, params: Optional[Dict] = None) -> Dict[str, Any]:
        """Execute GET request with error handling

        Args:
            endpoint: API endpoint path (e.g., "/teams/123")
            params: Optional query parameters

        Returns:
            Parsed JSON response

        Raises:
            ApiConnectionError: Network/connection error
            ApiAuthenticationError: Auth failure (401, 403)
            ApiNotFoundError: Resource not found (404)
            ApiClientError: Other API errors
        """
        if not self._client:
            raise RuntimeError("Client not initialized. Use 'async with' context manager.")

        url = endpoint if endpoint.startswith('/') else f'/{endpoint}'

        try:
            logger.debug(f"GET {url} with params {params}")
            response = await self._client.get(url, params=params)
            response.raise_for_status()

            data = response.json()
            logger.debug(f"Response: {data}")
            return data

        except httpx.HTTPStatusError as e:
            if e.response.status_code == 404:
                raise ApiNotFoundError(f"Resource not found: {url}") from e
            elif e.response.status_code in (401, 403):
                raise ApiAuthenticationError(f"Authentication failed: {e.response.text}") from e
            else:
                raise ApiClientError(f"HTTP {e.response.status_code}: {e.response.text}") from e

        except httpx.RequestError as e:
            logger.error(f"Connection error for {url}: {e}")
            raise ApiConnectionError(f"Failed to connect to API: {e}") from e

    async def _post(self, endpoint: str, data: Dict[str, Any]) -> Dict[str, Any]:
        """Execute POST request with error handling"""
        if not self._client:
            raise RuntimeError("Client not initialized. Use 'async with' context manager.")

        url = endpoint if endpoint.startswith('/') else f'/{endpoint}'

        try:
            logger.debug(f"POST {url} with data {data}")
            response = await self._client.post(url, json=data)
            response.raise_for_status()

            result = response.json()
            logger.debug(f"Response: {result}")
            return result

        except httpx.HTTPStatusError as e:
            if e.response.status_code in (401, 403):
                raise ApiAuthenticationError(f"Authentication failed: {e.response.text}") from e
            else:
                raise ApiClientError(f"HTTP {e.response.status_code}: {e.response.text}") from e

        except httpx.RequestError as e:
            logger.error(f"Connection error for {url}: {e}")
            raise ApiConnectionError(f"Failed to connect to API: {e}") from e

    # Abstract methods for subclasses to implement
    @abstractmethod
    async def get_team(self, team_id: int) -> Dict[str, Any]:
        """Fetch team details"""
        pass

    @abstractmethod
    async def get_team_roster(self, team_id: int) -> Dict[str, Any]:
        """Fetch team roster with all cards/players"""
        pass

    @abstractmethod
    async def get_player(self, player_id: int) -> Dict[str, Any]:
        """Fetch player details"""
        pass

    @abstractmethod
    async def submit_game_result(self, game_data: Dict[str, Any]) -> Dict[str, Any]:
        """Submit completed game results to league system"""
        pass
```

### SBA League API Client

**Location**: `app/data/sba_client.py` (not yet created)

**API Base URL**: `https://api.sba.manticorum.com` (from config)

**Key Endpoints**:
- `GET /teams/:id` - Team details
- `GET /teams/:id/roster` - Team roster
- `GET /players/:id` - Player details (simple model)
- `POST /games/submit` - Submit completed game

**Example Implementation**:
```python
from typing import Dict, Any
from .api_client import BaseApiClient

class SbaApiClient(BaseApiClient):
    """SBA League API client

    Handles communication with SBA REST API for team/player data.
    Uses simple player model (id, name, image, positions).
    """

    async def get_team(self, team_id: int) -> Dict[str, Any]:
        """Fetch SBA team details

        Args:
            team_id: SBA team ID

        Returns:
            Team data including name, manager, season
        """
        return await self._get(f"/teams/{team_id}")

    async def get_team_roster(self, team_id: int) -> Dict[str, Any]:
        """Fetch SBA team roster

        Args:
            team_id: SBA team ID

        Returns:
            Roster data with list of players
        """
        return await self._get(f"/teams/{team_id}/roster")

    async def get_player(self, player_id: int) -> Dict[str, Any]:
        """Fetch SBA player details

        Args:
            player_id: SBA player ID

        Returns:
            Player data (id, name, image, positions, wara, team)
        """
        return await self._get(f"/players/{player_id}")

    async def submit_game_result(self, game_data: Dict[str, Any]) -> Dict[str, Any]:
        """Submit completed SBA game to league system

        Args:
            game_data: Complete game data including plays, stats, final score

        Returns:
            Confirmation response from API
        """
        return await self._post("/games/submit", game_data)
```

### PD League API Client

**Location**: `app/data/pd_client.py` (not yet created)

**API Base URL**: `https://api.pd.manticorum.com` (from config)

**Key Endpoints**:
- `GET /api/v2/teams/:id` - Team details
- `GET /api/v2/teams/:id/roster` - Team roster
- `GET /api/v2/players/:id` - Player/card details
- `GET /api/v2/battingcardratings/player/:id` - Batting scouting data
- `GET /api/v2/pitchingcardratings/player/:id` - Pitching scouting data
- `GET /api/v2/cardsets/:id` - Cardset details
- `POST /api/v2/games/submit` - Submit completed game

**Example Implementation**:
```python
from typing import Dict, Any, Optional
from .api_client import BaseApiClient

class PdApiClient(BaseApiClient):
    """PD League API client

    Handles communication with PD REST API for team/player/card data.
    Supports detailed scouting data with batting/pitching ratings.
    """

    async def get_team(self, team_id: int) -> Dict[str, Any]:
        """Fetch PD team details"""
        return await self._get(f"/api/v2/teams/{team_id}")

    async def get_team_roster(self, team_id: int) -> Dict[str, Any]:
        """Fetch PD team roster"""
        return await self._get(f"/api/v2/teams/{team_id}/roster")

    async def get_player(self, player_id: int) -> Dict[str, Any]:
        """Fetch PD player/card details

        Returns basic card info without scouting data.
        Use get_batting_card() and get_pitching_card() for detailed ratings.
        """
        return await self._get(f"/api/v2/players/{player_id}")

    async def get_batting_card(self, player_id: int) -> Optional[Dict[str, Any]]:
        """Fetch PD batting card scouting data

        Args:
            player_id: PD card ID

        Returns:
            Batting ratings (steal, bunting, hit ratings vs LHP/RHP) or None
        """
        try:
            return await self._get(f"/api/v2/battingcardratings/player/{player_id}")
        except ApiNotFoundError:
            # Not all cards have batting ratings (pitcher-only cards)
            return None

    async def get_pitching_card(self, player_id: int) -> Optional[Dict[str, Any]]:
        """Fetch PD pitching card scouting data

        Args:
            player_id: PD card ID

        Returns:
            Pitching ratings (balk, hold, ratings vs LHB/RHB) or None
        """
        try:
            return await self._get(f"/api/v2/pitchingcardratings/player/{player_id}")
        except ApiNotFoundError:
            # Not all cards have pitching ratings (position players)
            return None

    async def get_cardset(self, cardset_id: int) -> Dict[str, Any]:
        """Fetch PD cardset details

        Args:
            cardset_id: Cardset ID

        Returns:
            Cardset info (name, description, ranked_legal)
        """
        return await self._get(f"/api/v2/cardsets/{cardset_id}")

    async def submit_game_result(self, game_data: Dict[str, Any]) -> Dict[str, Any]:
        """Submit completed PD game to league system"""
        return await self._post("/api/v2/games/submit", game_data)
```

## Caching Strategy

**Location**: `app/data/cache.py` (not yet created)

**Purpose**: Reduce API calls by caching frequently accessed data.

**Caching Targets**:
- Player/card data (rarely changes)
- Team rosters (changes during roster management, not during games)
- Cardset information (static)
- **NOT** game state (always use database)

**Cache Backend Options**:

### Option 1: Redis (Recommended for Production)
```python
import redis.asyncio as redis
import json
from typing import Optional, Any

class RedisCache:
    """Redis-based cache for API data

    Provides async get/set operations with TTL support.
    """

    def __init__(self, redis_url: str = "redis://localhost:6379"):
        self.redis = redis.from_url(redis_url, decode_responses=True)

    async def get(self, key: str) -> Optional[Any]:
        """Get cached value

        Args:
            key: Cache key

        Returns:
            Cached value or None if not found/expired
        """
        value = await self.redis.get(key)
        if value:
            return json.loads(value)
        return None

    async def set(self, key: str, value: Any, ttl: int = 3600) -> None:
        """Set cache value with TTL

        Args:
            key: Cache key
            value: Value to cache (must be JSON-serializable)
            ttl: Time to live in seconds (default 1 hour)
        """
        await self.redis.set(key, json.dumps(value), ex=ttl)

    async def delete(self, key: str) -> None:
        """Delete cached value"""
        await self.redis.delete(key)

    async def clear_pattern(self, pattern: str) -> None:
        """Delete all keys matching pattern

        Args:
            pattern: Redis key pattern (e.g., "player:*")
        """
        keys = await self.redis.keys(pattern)
        if keys:
            await self.redis.delete(*keys)
```

### Option 2: In-Memory Cache (Development/Testing)
```python
import asyncio
import pendulum
from typing import Dict, Any, Optional, Tuple
from dataclasses import dataclass

@dataclass
class CacheEntry:
    """Cache entry with expiration"""
    value: Any
    expires_at: pendulum.DateTime

class MemoryCache:
    """In-memory cache for API data

    Simple dict-based cache with TTL support.
    Suitable for development/testing, not production.
    """

    def __init__(self):
        self._cache: Dict[str, CacheEntry] = {}
        self._lock = asyncio.Lock()

    async def get(self, key: str) -> Optional[Any]:
        """Get cached value if not expired"""
        async with self._lock:
            if key in self._cache:
                entry = self._cache[key]
                if pendulum.now('UTC') < entry.expires_at:
                    return entry.value
                else:
                    # Expired, remove it
                    del self._cache[key]
            return None

    async def set(self, key: str, value: Any, ttl: int = 3600) -> None:
        """Set cache value with TTL"""
        expires_at = pendulum.now('UTC').add(seconds=ttl)
        async with self._lock:
            self._cache[key] = CacheEntry(value=value, expires_at=expires_at)

    async def delete(self, key: str) -> None:
        """Delete cached value"""
        async with self._lock:
            self._cache.pop(key, None)

    async def clear(self) -> None:
        """Clear entire cache"""
        async with self._lock:
            self._cache.clear()
```

### Cached API Client Pattern
```python
from typing import Dict, Any, Optional
from .api_client import BaseApiClient
from .cache import RedisCache

class CachedApiClient:
    """Wrapper that adds caching to API client

    Caches player/team data to reduce API calls.
    """

    def __init__(self, api_client: BaseApiClient, cache: RedisCache):
        self.api_client = api_client
        self.cache = cache

    async def get_player(self, player_id: int, use_cache: bool = True) -> Dict[str, Any]:
        """Get player data with optional caching

        Args:
            player_id: Player ID
            use_cache: Whether to use cached data (default True)

        Returns:
            Player data from cache or API
        """
        cache_key = f"player:{player_id}"

        # Try cache first
        if use_cache:
            cached = await self.cache.get(cache_key)
            if cached:
                return cached

        # Cache miss, fetch from API
        data = await self.api_client.get_player(player_id)

        # Store in cache (1 hour TTL for player data)
        await self.cache.set(cache_key, data, ttl=3600)

        return data

    async def invalidate_player(self, player_id: int) -> None:
        """Invalidate cached player data

        Call this when player data changes (roster updates).
        """
        cache_key = f"player:{player_id}"
        await self.cache.delete(cache_key)
```

## Configuration Integration

API clients should read from league configs:

**Location**: `app/config/league_configs.py` (already exists)

```python
from app.config import get_league_config

# Get league-specific API base URL
config = get_league_config("sba")
api_url = config.get_api_base_url()  # "https://api.sba.manticorum.com"

# Or for PD
config = get_league_config("pd")
api_url = config.get_api_base_url()  # "https://api.pd.manticorum.com"
```

**Environment Variables** (`.env`):
```bash
# SBA League API
SBA_API_URL=https://api.sba.manticorum.com
SBA_API_KEY=your-api-key-here

# PD League API
PD_API_URL=https://api.pd.manticorum.com
PD_API_KEY=your-api-key-here

# Cache (optional)
REDIS_URL=redis://localhost:6379
CACHE_ENABLED=true
CACHE_DEFAULT_TTL=3600
```

## Usage Examples

### Fetching Team Roster
```python
from app.data import get_api_client
from app.models import SbaPlayer, PdPlayer

async def load_team_roster(league_id: str, team_id: int):
    """Load team roster from API"""

    # Get league-specific client
    api_client = get_api_client(league_id)

    async with api_client:
        # Fetch roster data
        roster_data = await api_client.get_team_roster(team_id)

        # Parse into player models
        players = []
        for player_data in roster_data['players']:
            if league_id == "sba":
                player = SbaPlayer.from_api_response(player_data)
            else:  # PD
                # Fetch detailed scouting data
                batting = await api_client.get_batting_card(player_data['id'])
                pitching = await api_client.get_pitching_card(player_data['id'])
                player = PdPlayer.from_api_response(player_data, batting, pitching)
            players.append(player)

    return players
```

### Submitting Game Results
```python
from app.data import get_api_client

async def submit_completed_game(game_id: str, league_id: str):
    """Submit completed game to league system"""

    # Fetch game data from database
    game_data = await db_ops.export_game_data(game_id)

    # Get league API client
    api_client = get_api_client(league_id)

    async with api_client:
        # Submit to league system
        result = await api_client.submit_game_result(game_data)

    return result
```

### Using Cache
```python
from app.data import get_cached_client

async def get_player_with_cache(league_id: str, player_id: int):
    """Get player data with caching"""

    # Get cached API client
    client = get_cached_client(league_id)

    async with client:
        # First call fetches from API and caches
        player_data = await client.get_player(player_id)

        # Second call returns from cache (fast)
        player_data = await client.get_player(player_id)

    return player_data
```

## Error Handling

All API client methods raise specific exceptions:

```python
from app.data import get_api_client, ApiClientError, ApiNotFoundError, ApiConnectionError

async def safe_api_call():
    """Example of proper error handling"""

    api_client = get_api_client("sba")

    try:
        async with api_client:
            player = await api_client.get_player(123)

    except ApiNotFoundError:
        # Player doesn't exist
        logger.warning(f"Player 123 not found")
        return None

    except ApiConnectionError:
        # Network error, retry later
        logger.error("API connection failed, will retry")
        raise

    except ApiClientError as e:
        # Other API error
        logger.error(f"API error: {e}")
        raise

    return player
```

## Testing Patterns

### Mocking API Clients
```python
from unittest.mock import AsyncMock, Mock
import pytest

@pytest.fixture
def mock_sba_client():
    """Mock SBA API client for testing"""
    client = Mock()

    # Mock get_player method
    client.get_player = AsyncMock(return_value={
        "id": 123,
        "name": "Mike Trout",
        "image": "https://example.com/trout.jpg",
        "pos_1": "CF",
        "wara": 8.5
    })

    # Mock context manager
    client.__aenter__ = AsyncMock(return_value=client)
    client.__aexit__ = AsyncMock(return_value=None)

    return client

async def test_game_with_mock_api(mock_sba_client):
    """Test game engine with mocked API"""

    # Use mock instead of real API
    async with mock_sba_client:
        player_data = await mock_sba_client.get_player(123)

    assert player_data['name'] == "Mike Trout"
```

### Testing Cache Behavior
```python
async def test_cache_hit():
    """Test cache returns cached value"""
    cache = MemoryCache()

    # Set value
    await cache.set("test:key", {"data": "value"}, ttl=60)

    # Get value (should be cached)
    result = await cache.get("test:key")
    assert result == {"data": "value"}

async def test_cache_expiration():
    """Test cache expires after TTL"""
    cache = MemoryCache()

    # Set value with 1 second TTL
    await cache.set("test:key", {"data": "value"}, ttl=1)

    # Wait for expiration
    await asyncio.sleep(2)

    # Should be expired
    result = await cache.get("test:key")
    assert result is None
```

## Common Tasks

### Adding a New API Endpoint

1. **Add method to appropriate client**:
```python
# In sba_client.py or pd_client.py
async def get_new_endpoint(self, param: int) -> Dict[str, Any]:
    """Fetch new endpoint data"""
    return await self._get(f"/new/endpoint/{param}")
```

2. **Update tests**:
```python
async def test_new_endpoint(sba_client):
    """Test new endpoint"""
    result = await sba_client.get_new_endpoint(123)
    assert result is not None
```

3. **Document in this file**: Add example to usage section

### Modifying Cache Behavior

1. **Adjust TTL for specific data type**:
```python
# Player data: 1 hour (changes rarely)
await cache.set(f"player:{player_id}", data, ttl=3600)

# Team roster: 10 minutes (may change during roster management)
await cache.set(f"roster:{team_id}", data, ttl=600)

# Cardset data: 24 hours (static)
await cache.set(f"cardset:{cardset_id}", data, ttl=86400)
```

2. **Add cache invalidation triggers**:
```python
async def on_roster_update(team_id: int):
    """Invalidate roster cache when roster changes"""
    await cache.delete(f"roster:{team_id}")
```

### Switching Cache Backends

Development (in-memory):
```python
from app.data.cache import MemoryCache

cache = MemoryCache()
```

Production (Redis):
```python
from app.data.cache import RedisCache
from app.config import settings

cache = RedisCache(settings.redis_url)
```

## Troubleshooting

### API Connection Issues

**Symptom**: `ApiConnectionError: Failed to connect to API`

**Checks**:
1. Verify API URL in config: `config.get_api_base_url()`
2. Check network connectivity: `curl https://api.sba.manticorum.com/health`
3. Verify API key is set: `echo $SBA_API_KEY`
4. Check firewall/network access

### Authentication Failures

**Symptom**: `ApiAuthenticationError: Authentication failed`

**Checks**:
1. Verify API key is correct
2. Check API key format (Bearer token vs other)
3. Confirm API key has required permissions
4. Check API key expiration

### Cache Not Working

**Symptom**: Every request hits API instead of cache

**Checks**:
1. Verify cache is initialized: `cache is not None`
2. Check Redis is running: `redis-cli ping` (should return "PONG")
3. Verify cache keys are consistent
4. Check TTL isn't too short

### Rate Limiting

**Symptom**: API returns 429 (Too Many Requests)

**Solutions**:
1. Implement exponential backoff retry logic
2. Reduce API call frequency
3. Increase cache TTL to reduce calls
4. Add rate limiting protection in client

## Performance Targets

- **API Response Time**: < 500ms for typical requests
- **Cache Hit Ratio**: > 80% for player/team data
- **Cache Lookup Time**: < 10ms (Redis), < 1ms (memory)
- **Concurrent Requests**: Support 10+ simultaneous API calls

## Security Considerations

- **API Keys**: Store in environment variables, never commit to git
- **HTTPS Only**: All API communication over encrypted connections
- **Input Validation**: Validate all API responses with Pydantic models
- **Error Messages**: Don't expose API keys or internal details in logs
- **Rate Limiting**: Respect API rate limits, implement backoff

## Dependencies

**Required**:
```txt
httpx>=0.25.0          # Async HTTP client
pydantic>=2.10.0       # Response validation (already installed)
```

**Optional**:
```txt
redis>=5.0.0           # Redis cache backend
aiohttp>=3.9.0         # Alternative HTTP client
```

## Implementation Priority

**Phase 1** (Week 7-8):
1. ✅ Create base API client with error handling
2. ✅ Implement SBA client for simple player data
3. ✅ Add basic integration tests
4. ⏳ Connect to game engine for roster loading

**Phase 2** (Week 9-10):
1. ⏳ Implement PD client with scouting data
2. ⏳ Add in-memory cache for development
3. ⏳ Add result submission endpoints

**Phase 3** (Post-MVP):
1. ⏳ Add Redis cache for production
2. ⏳ Implement advanced retry logic
3. ⏳ Add request/response logging
4. ⏳ Performance optimization

## References

- **PRD API Section**: `../../prd-web-scorecard-1.1.md` (lines 87-90, 1119-1126)
- **Player Models**: `../models/player_models.py` - SbaPlayer, PdPlayer classes
- **League Configs**: `../config/league_configs.py` - API URLs and settings
- **Backend Architecture**: `../CLAUDE.md` - Overall backend structure

---

**Status**: 🚧 **AWAITING IMPLEMENTATION**
**Current Phase**: Phase 2 - Week 7 (Runner Advancement & Play Resolution)
**Next Steps**: Implement base API client and SBA client for roster loading

**Note**: This directory will be populated during Phase 2 integration work when game engine needs to fetch real player data from league APIs.