|
|
|
|
@ -0,0 +1,356 @@
|
|
|
|
|
# Phase 0 — Render Pipeline Optimization: Project Plan
|
|
|
|
|
|
|
|
|
|
**Version:** 1.1
|
|
|
|
|
**Date:** 2026-03-13
|
|
|
|
|
**PRD Reference:** `docs/prd-evolution/02-architecture.md` § Card Render Pipeline Optimization, `13-implementation.md` § Phase 0
|
|
|
|
|
**Status:** Complete — deployed to dev (PR #94), client-side concurrent uploads merged via PR #28 (card-creation repo)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Overview
|
|
|
|
|
|
|
|
|
|
Phase 0 is independent of Card Evolution and benefits all existing card workflows immediately. The goal is to reduce per-card render time and full cardset uploads significantly by eliminating browser spawn overhead, CDN dependencies, and sequential processing.
|
|
|
|
|
|
|
|
|
|
**Bottlenecks addressed:**
|
|
|
|
|
1. New Chromium process spawned per render request (~1.0-1.5s overhead)
|
|
|
|
|
2. Google Fonts CDN fetched over network on every render (~0.3-0.5s) — no persistent cache since browser is destroyed after each render
|
|
|
|
|
3. Upload pipeline is fully sequential — one card at a time, blocking S3 upload via synchronous boto3
|
|
|
|
|
|
|
|
|
|
**Results:**
|
|
|
|
|
|
|
|
|
|
| Metric | Before | Target | Actual |
|
|
|
|
|
|--------|--------|--------|--------|
|
|
|
|
|
| Per-card render (fresh) | ~2.0s (benchmark avg) | <1.0s | **~0.98s avg** (range 0.63-1.44s, **~51% reduction**) |
|
|
|
|
|
| Per-card render (cached) | N/A | — | **~0.1s** |
|
|
|
|
|
| External dependencies during render | Google Fonts CDN | None | **None** |
|
|
|
|
|
| Chromium processes per 800-card run | 800 | 1 | **1** |
|
|
|
|
|
| 800-card upload (sequential, estimated) | ~27 min | ~8-13 min | ~13 min (estimated at 0.98s/card) |
|
|
|
|
|
| 800-card upload (concurrent 8x, estimated) | N/A | ~2-4 min | ~2-3 min (estimated) |
|
|
|
|
|
|
|
|
|
|
**Benchmark details (7 fresh renders on dev, 2026-03-13):**
|
|
|
|
|
|
|
|
|
|
| Player | Type | Time |
|
|
|
|
|
|--------|------|------|
|
|
|
|
|
| Michael Young (12726) | Batting | 0.96s |
|
|
|
|
|
| Darin Erstad (12729) | Batting | 0.78s |
|
|
|
|
|
| Wilson Valdez (12746) | Batting | 1.44s |
|
|
|
|
|
| Player 12750 | Batting | 0.76s |
|
|
|
|
|
| Jarrod Washburn (12880) | Pitching | 0.63s |
|
|
|
|
|
| Ryan Drese (12879) | Pitching | 1.25s |
|
|
|
|
|
| Player 12890 | Pitching | 1.07s |
|
|
|
|
|
|
|
|
|
|
**Average: 0.98s** — meets the <1s target. Occasional spikes to ~1.4s from Chromium GC pressure. Pitching cards tend to render slightly faster due to less template data.
|
|
|
|
|
|
|
|
|
|
**Optimization breakdown:**
|
|
|
|
|
- Persistent browser (WP-02): eliminated ~1.0s spawn overhead
|
|
|
|
|
- Variable font deduplication (WP-01 fix): eliminated ~163KB redundant base64 parsing, saved ~0.4s
|
|
|
|
|
- Remaining ~0.98s is Playwright page creation, HTML parsing, and PNG screenshot — not reducible without GPU acceleration or a different rendering approach
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Work Packages (6 WPs)
|
|
|
|
|
|
|
|
|
|
### WP-00: Baseline Benchmarks
|
|
|
|
|
|
|
|
|
|
**Repo:** `database` + `card-creation`
|
|
|
|
|
**Complexity:** XS
|
|
|
|
|
**Dependencies:** None
|
|
|
|
|
|
|
|
|
|
Capture before-metrics so we can measure improvement.
|
|
|
|
|
|
|
|
|
|
#### Tasks
|
|
|
|
|
1. Time 10 sequential card renders via the API (curl with timing)
|
|
|
|
|
2. Time a small batch S3 upload (e.g., 20 cards) via `pd-cards upload`
|
|
|
|
|
3. Record results in a benchmark log
|
|
|
|
|
|
|
|
|
|
#### Tests
|
|
|
|
|
- [ ] Benchmark script or documented curl commands exist and are repeatable
|
|
|
|
|
|
|
|
|
|
#### Acceptance Criteria
|
|
|
|
|
1. Baseline numbers recorded for per-card render time
|
|
|
|
|
2. Baseline numbers recorded for batch upload time
|
|
|
|
|
3. Methodology is repeatable for post-optimization comparison
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### WP-01: Self-Hosted Fonts
|
|
|
|
|
|
|
|
|
|
**Repo:** `database`
|
|
|
|
|
**Complexity:** S
|
|
|
|
|
**Dependencies:** None (can run in parallel with WP-02)
|
|
|
|
|
|
|
|
|
|
Replace Google Fonts CDN with locally embedded WOFF2 fonts. Eliminates ~0.3-0.5s network round-trip per render and removes external dependency.
|
|
|
|
|
|
|
|
|
|
#### Current State
|
|
|
|
|
- `storage/templates/player_card.html` lines 5-7: `<link>` tags to `fonts.googleapis.com`
|
|
|
|
|
- `storage/templates/style.html`: References `"Open Sans"` and `"Source Sans 3"` font-families
|
|
|
|
|
- Two fonts used: Open Sans (300, 400, 700) and Source Sans 3 (400, 700)
|
|
|
|
|
|
|
|
|
|
#### Implementation
|
|
|
|
|
1. Download WOFF2 files for both fonts (5 files total: Open Sans 300/400/700, Source Sans 3 400/700)
|
|
|
|
|
2. Base64-encode each WOFF2 file
|
|
|
|
|
3. Add `@font-face` declarations with base64 data URIs to `style.html`
|
|
|
|
|
4. Remove the three `<link>` tags from `player_card.html`
|
|
|
|
|
5. Visual diff: render the same card before/after and verify identical output
|
|
|
|
|
|
|
|
|
|
#### Files
|
|
|
|
|
- **Create:** `database/storage/fonts/` directory with raw WOFF2 files (source archive, not deployed)
|
|
|
|
|
- **Modify:** `database/storage/templates/style.html` — add `@font-face` declarations
|
|
|
|
|
- **Modify:** `database/storage/templates/player_card.html` — remove `<link>` tags (lines 5-7)
|
|
|
|
|
|
|
|
|
|
#### Tests
|
|
|
|
|
- [ ] Unit: `style.html` contains no `fonts.googleapis.com` references
|
|
|
|
|
- [ ] Unit: `player_card.html` contains no `<link>` to external font CDNs
|
|
|
|
|
- [ ] Unit: `@font-face` declarations present for all 5 font variants
|
|
|
|
|
- [ ] Visual: rendered card is pixel-identical to pre-change output (manual check)
|
|
|
|
|
|
|
|
|
|
#### Acceptance Criteria
|
|
|
|
|
1. No external network requests during card render
|
|
|
|
|
2. All 5 font weights render correctly
|
|
|
|
|
3. Card appearance unchanged
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### WP-02: Persistent Browser Instance
|
|
|
|
|
|
|
|
|
|
**Repo:** `database`
|
|
|
|
|
**Complexity:** M
|
|
|
|
|
**Dependencies:** None (can run in parallel with WP-01)
|
|
|
|
|
|
|
|
|
|
Replace per-request Chromium launch/teardown with a persistent browser that lives for the lifetime of the API process. Eliminates ~1.0-1.5s spawn overhead per render.
|
|
|
|
|
|
|
|
|
|
#### Current State
|
|
|
|
|
- `app/routers_v2/players.py` lines 801-826: `async with async_playwright() as p:` block creates and destroys a browser per request
|
|
|
|
|
- No browser reuse, no connection pooling
|
|
|
|
|
|
|
|
|
|
#### Implementation
|
|
|
|
|
1. Add module-level `_browser` and `_playwright` globals to `players.py`
|
|
|
|
|
2. Implement `get_browser()` — lazy-init with `is_connected()` auto-reconnect
|
|
|
|
|
3. Implement `shutdown_browser()` — clean teardown for API shutdown
|
|
|
|
|
4. Replace the `async with async_playwright()` block with page-per-request pattern:
|
|
|
|
|
```python
|
|
|
|
|
browser = await get_browser()
|
|
|
|
|
page = await browser.new_page(viewport={"width": 1280, "height": 720})
|
|
|
|
|
try:
|
|
|
|
|
await page.set_content(html_string)
|
|
|
|
|
await page.screenshot(path=file_path, type="png", clip={...})
|
|
|
|
|
finally:
|
|
|
|
|
await page.close()
|
|
|
|
|
```
|
|
|
|
|
5. Ensure page is always closed in `finally` block to prevent memory leaks
|
|
|
|
|
|
|
|
|
|
#### Files
|
|
|
|
|
- **Modify:** `database/app/routers_v2/players.py` — persistent browser, page-per-request
|
|
|
|
|
|
|
|
|
|
#### Tests
|
|
|
|
|
- [ ] Unit: `get_browser()` returns a connected browser
|
|
|
|
|
- [ ] Unit: `get_browser()` returns same instance on second call
|
|
|
|
|
- [ ] Unit: `get_browser()` relaunches if browser disconnected
|
|
|
|
|
- [ ] Integration: render 10 cards sequentially, no browser leaks (page count returns to 0 between renders)
|
|
|
|
|
- [ ] Integration: concurrent renders (4 simultaneous requests) complete without errors
|
|
|
|
|
- [ ] Integration: `shutdown_browser()` cleanly closes browser and playwright
|
|
|
|
|
|
|
|
|
|
#### Acceptance Criteria
|
|
|
|
|
1. Only 1 Chromium process running regardless of render count
|
|
|
|
|
2. Page count returns to 0 between renders (no leaks)
|
|
|
|
|
3. Auto-reconnect works if browser crashes
|
|
|
|
|
4. ~~Per-card render time drops to ~1.0-1.5s~~ **Actual: ~0.98s avg fresh render (from ~2.0s baseline) — target met**
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### WP-03: FastAPI Lifespan Hooks
|
|
|
|
|
|
|
|
|
|
**Repo:** `database`
|
|
|
|
|
**Complexity:** S
|
|
|
|
|
**Dependencies:** WP-02
|
|
|
|
|
|
|
|
|
|
Wire `get_browser()` and `shutdown_browser()` into FastAPI's lifespan so the browser warms up on startup and cleans up on shutdown.
|
|
|
|
|
|
|
|
|
|
#### Current State
|
|
|
|
|
- `app/main.py` line 54: plain `FastAPI(...)` constructor with no lifespan
|
|
|
|
|
- Only middleware is the DB session handler (lines 97-105)
|
|
|
|
|
|
|
|
|
|
#### Implementation
|
|
|
|
|
1. Add `@asynccontextmanager` lifespan function that calls `get_browser()` on startup and `shutdown_browser()` on shutdown
|
|
|
|
|
2. Pass `lifespan=lifespan` to `FastAPI()` constructor
|
|
|
|
|
3. Verify existing middleware is unaffected
|
|
|
|
|
|
|
|
|
|
#### Files
|
|
|
|
|
- **Modify:** `database/app/main.py` — add lifespan hook, pass to FastAPI constructor
|
|
|
|
|
- **Modify:** `database/app/routers_v2/players.py` — export `get_browser`/`shutdown_browser` (if not already importable)
|
|
|
|
|
|
|
|
|
|
#### Tests
|
|
|
|
|
- [ ] Integration: browser is connected immediately after API startup (before any render request)
|
|
|
|
|
- [ ] Integration: browser is closed after API shutdown (no orphan processes)
|
|
|
|
|
- [ ] Integration: existing DB middleware still functions correctly
|
|
|
|
|
- [ ] Integration: API health endpoint still responds
|
|
|
|
|
|
|
|
|
|
#### Acceptance Criteria
|
|
|
|
|
1. Browser pre-warmed on startup — first render request has no cold-start penalty
|
|
|
|
|
2. Clean shutdown — no orphan Chromium processes after API stop
|
|
|
|
|
3. No regression in existing API behavior
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### WP-04: Concurrent Upload Pipeline
|
|
|
|
|
|
|
|
|
|
**Repo:** `card-creation`
|
|
|
|
|
**Complexity:** M
|
|
|
|
|
**Dependencies:** WP-02 (persistent browser must be deployed for concurrent renders to work)
|
|
|
|
|
|
|
|
|
|
Replace the sequential upload loop with semaphore-bounded `asyncio.gather` for parallel card fetching, rendering, and S3 upload.
|
|
|
|
|
|
|
|
|
|
#### Current State
|
|
|
|
|
- `pd_cards/core/upload.py` `upload_cards_to_s3()` (lines 109-333): sequential `for x in all_players:` loop
|
|
|
|
|
- `fetch_card_image` timeout hardcoded to 6s (line 28)
|
|
|
|
|
- `upload_card_to_s3()` uses synchronous `boto3.put_object` — blocks the event loop
|
|
|
|
|
- Single `aiohttp.ClientSession` is reused (good)
|
|
|
|
|
|
|
|
|
|
#### Implementation
|
|
|
|
|
1. Wrap per-card processing in an `async def process_card(player)` coroutine
|
|
|
|
|
2. Add `asyncio.Semaphore(concurrency)` guard (default concurrency=8)
|
|
|
|
|
3. Replace sequential loop with `asyncio.gather(*[process_card(p) for p in all_players], return_exceptions=True)`
|
|
|
|
|
4. Offload synchronous `upload_card_to_s3()` to thread pool via `asyncio.get_event_loop().run_in_executor(None, upload_card_to_s3, ...)`
|
|
|
|
|
5. Increase `fetch_card_image` timeout from 6s to 10s
|
|
|
|
|
6. Add error handling: individual card failures logged but don't abort the batch
|
|
|
|
|
7. Add progress reporting: log completion count every N cards (not every start)
|
|
|
|
|
8. Add `--concurrency` CLI argument to `pd-cards upload` command
|
|
|
|
|
|
|
|
|
|
#### Files
|
|
|
|
|
- **Modify:** `pd_cards/core/upload.py` — concurrent pipeline, timeout increase
|
|
|
|
|
- **Modify:** `pd_cards/cli/upload.py` (or wherever CLI args are defined) — add `--concurrency` flag
|
|
|
|
|
|
|
|
|
|
#### Tests
|
|
|
|
|
- [ ] Unit: semaphore limits concurrent tasks to specified count
|
|
|
|
|
- [ ] Unit: individual card failure doesn't abort batch (return_exceptions=True)
|
|
|
|
|
- [ ] Unit: progress logging fires at correct intervals
|
|
|
|
|
- [ ] Integration: 20-card concurrent upload completes successfully
|
|
|
|
|
- [ ] Integration: S3 URLs are correct after concurrent upload
|
|
|
|
|
- [ ] Integration: `--concurrency 1` behaves like sequential (regression safety)
|
|
|
|
|
|
|
|
|
|
#### Acceptance Criteria
|
|
|
|
|
1. Default concurrency of 8 parallel card processes
|
|
|
|
|
2. Individual failures logged, don't abort batch
|
|
|
|
|
3. `fetch_card_image` timeout is 10s
|
|
|
|
|
4. 800-card upload estimated at ~3-4 minutes with 8x concurrency (with WP-01 + WP-02 deployed)
|
|
|
|
|
5. `--concurrency` flag available on CLI
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### WP-05: Legacy Upload Script Update
|
|
|
|
|
|
|
|
|
|
**Repo:** `card-creation`
|
|
|
|
|
**Complexity:** S
|
|
|
|
|
**Dependencies:** WP-04
|
|
|
|
|
|
|
|
|
|
Apply the same concurrency pattern to `check_cards_and_upload.py` for users who still use the legacy script.
|
|
|
|
|
|
|
|
|
|
#### Current State
|
|
|
|
|
- `check_cards_and_upload.py` lines 150-293: identical sequential pattern to `pd_cards/core/upload.py`
|
|
|
|
|
- Module-level boto3 client (line 27)
|
|
|
|
|
|
|
|
|
|
#### Implementation
|
|
|
|
|
1. Refactor the sequential loop to use `asyncio.gather` + `Semaphore` (same pattern as WP-04)
|
|
|
|
|
2. Offload synchronous S3 calls to thread pool
|
|
|
|
|
3. Increase fetch timeout to 10s
|
|
|
|
|
4. Add progress reporting
|
|
|
|
|
|
|
|
|
|
#### Files
|
|
|
|
|
- **Modify:** `check_cards_and_upload.py`
|
|
|
|
|
|
|
|
|
|
#### Tests
|
|
|
|
|
- [ ] Integration: legacy script uploads 10 cards concurrently without errors
|
|
|
|
|
- [ ] Integration: S3 URLs match expected format
|
|
|
|
|
|
|
|
|
|
#### Acceptance Criteria
|
|
|
|
|
1. Same concurrency behavior as WP-04
|
|
|
|
|
2. No regression in existing functionality
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## WP Summary
|
|
|
|
|
|
|
|
|
|
| WP | Title | Repo | Size | Dependencies | Tests |
|
|
|
|
|
|----|-------|------|------|-------------|-------|
|
|
|
|
|
| WP-00 | Baseline Benchmarks | both | XS | — | 1 |
|
|
|
|
|
| WP-01 | Self-Hosted Fonts | database | S | — | 4 |
|
|
|
|
|
| WP-02 | Persistent Browser Instance | database | M | — | 6 |
|
|
|
|
|
| WP-03 | FastAPI Lifespan Hooks | database | S | WP-02 | 4 |
|
|
|
|
|
| WP-04 | Concurrent Upload Pipeline | card-creation | M | WP-02 | 6 |
|
|
|
|
|
| WP-05 | Legacy Upload Script Update | card-creation | S | WP-04 | 2 |
|
|
|
|
|
|
|
|
|
|
**Total: 6 WPs, ~23 tests**
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Dependency Graph
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
WP-00 (benchmarks)
|
|
|
|
|
|
|
|
|
|
|
v
|
|
|
|
|
WP-01 (fonts) ──────┐
|
|
|
|
|
├──> WP-03 (lifespan) ──> Deploy to dev ──> WP-04 (concurrent upload)
|
|
|
|
|
WP-02 (browser) ────┘ |
|
|
|
|
|
v
|
|
|
|
|
WP-05 (legacy script)
|
|
|
|
|
|
|
|
|
|
|
v
|
|
|
|
|
Re-run benchmarks
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Parallelization:**
|
|
|
|
|
- WP-00, WP-01, WP-02 can all start immediately in parallel
|
|
|
|
|
- WP-03 needs WP-02
|
|
|
|
|
- WP-04 needs WP-02 deployed (persistent browser must be running server-side for concurrent fetches to work)
|
|
|
|
|
- WP-05 needs WP-04 (reuse the pattern)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Risks
|
|
|
|
|
|
|
|
|
|
| Risk | Likelihood | Impact | Mitigation |
|
|
|
|
|
|------|-----------|--------|------------|
|
|
|
|
|
| Base64-embedded fonts bloat template HTML | Medium | Low | WOFF2 files are small (~20-40KB each). Total ~150KB base64 added to template. Acceptable since template is loaded once into Playwright, not transmitted to clients. |
|
|
|
|
|
| Persistent browser memory leak | Medium | Medium | Always close pages in `finally` block. Monitor RSS after sustained renders. Add `is_connected()` check for crash recovery. |
|
|
|
|
|
| Concurrent renders overload API server | Low | High | Semaphore bounds concurrency. Start at 8, tune based on server RAM (~100MB per page). 8 pages = ~800MB, well within 16GB. |
|
|
|
|
|
| Synchronous boto3 blocks event loop under concurrency | Medium | Medium | Use `run_in_executor` to offload to thread pool. Consider `aioboto3` if thread pool proves insufficient. |
|
|
|
|
|
| Visual regression from font change | Low | High | Visual diff test before/after. Render same card with both approaches and compare pixel output. |
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Open Questions
|
|
|
|
|
|
|
|
|
|
None — Phase 0 is straightforward infrastructure optimization with no design decisions pending.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Follow-On: Local High-Concurrency Rendering (2026-03-14)
|
|
|
|
|
|
|
|
|
|
After Phase 0 was deployed, a follow-on improvement was implemented: **configurable API URL** for card rendering. This enables running the Paper Dynasty API server locally on the workstation and pointing upload scripts at `localhost` for dramatically higher concurrency.
|
|
|
|
|
|
|
|
|
|
### Changes
|
|
|
|
|
- `pd_cards/core/upload.py` — `upload_cards_to_s3()`, `refresh_card_images()`, `check_card_images()` accept `api_url` parameter (defaults to production)
|
|
|
|
|
- `pd_cards/commands/upload.py` — `--api-url` CLI option on `upload s3` command
|
|
|
|
|
- `check_cards_and_upload.py` — `PD_API_URL` env var override (legacy script)
|
|
|
|
|
|
|
|
|
|
### Expected Performance
|
|
|
|
|
|
|
|
|
|
| Scenario | Per-card | 800 cards |
|
|
|
|
|
|----------|----------|-----------|
|
|
|
|
|
| Remote server, 8x concurrency (current) | ~0.98s render + network | ~2-3 min |
|
|
|
|
|
| Local server, 32x concurrency | ~0.98s render, 32 parallel | ~30-45 sec |
|
|
|
|
|
|
|
|
|
|
### Usage
|
|
|
|
|
```bash
|
|
|
|
|
pd-cards upload s3 --cardset "2005 Live" --api-url http://localhost:8000/api --concurrency 32
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Notes
|
|
|
|
|
|
|
|
|
|
- Phase 0 is a prerequisite for Phase 4 (Animated Cosmetics) which needs the persistent browser for efficient multi-frame APNG capture
|
|
|
|
|
- The persistent browser also benefits Phase 2/3 variant rendering
|
|
|
|
|
- GPU acceleration was evaluated and rejected — see PRD `02-architecture.md` § Optimization 4
|
|
|
|
|
- Consider `aioboto3` as a future enhancement if `run_in_executor` thread pool becomes a bottleneck
|