cal/paper-dynasty-card-creation
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Phase 0 concurrent upload pipeline and benchmarks #27

Merged
cal merged 2 commits from feature/render-pipeline-optimization into main 2026-03-13 15:56:19 +00:00
Conversation 0 Commits 2 Files Changed 5 +777 -248
cal commented 2026-03-13 13:48:40 +00:00
Owner
Copy Link

Summary

Phase 0 render pipeline optimization — card-creation side (WP-00, WP-04, WP-05).

Companion to paper-dynasty-database PR #94 which handles the server-side changes (persistent browser, self-hosted fonts, lifespan hooks).

WP-00: Baseline Benchmarks

  • scripts/benchmark_render.sh — automated render timing script with --quick and --prod modes
  • scripts/benchmark_results.txt — baseline measurements (~2.0s avg per card)

WP-04: Concurrent Upload Pipeline (pd_cards/core/upload.py)

  • Replace sequential for x in all_players: loop with asyncio.gather + Semaphore(8)
  • Offload synchronous boto3 S3 calls to thread pool via run_in_executor
  • Increase fetch_card_image timeout from 6s → 10s
  • Add --concurrency / -j CLI flag (default: 8)
  • Progress reporting every 20 completions
  • Individual card failures logged but don't abort batch

WP-05: Legacy Upload Script (check_cards_and_upload.py)

  • Same concurrency pattern applied to the legacy script
  • CONCURRENCY = 8 constant, timeout increase, progress reporting

Results (with server-side Phase 0 deployed)

Metric Before After
Per-card render ~2.0s ~0.98s avg
800-card upload (sequential) ~27 min ~13 min (est)
800-card upload (8x concurrent) N/A ~2-3 min (est)

Supersedes

  • PR #25 (auto-generated WP-04)
  • PR #26 (auto-generated WP-05)

Test plan

  • Benchmark script runs and produces timing data
  • pd-cards upload s3 -c "2005 Live" --limit 10 -j 8 completes without errors
  • Legacy check_cards_and_upload.py runs concurrently without errors
  • --concurrency 1 behaves like sequential (regression safety)

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

## Summary Phase 0 render pipeline optimization — card-creation side (WP-00, WP-04, WP-05). Companion to paper-dynasty-database PR #94 which handles the server-side changes (persistent browser, self-hosted fonts, lifespan hooks). ### WP-00: Baseline Benchmarks - `scripts/benchmark_render.sh` — automated render timing script with `--quick` and `--prod` modes - `scripts/benchmark_results.txt` — baseline measurements (~2.0s avg per card) ### WP-04: Concurrent Upload Pipeline (`pd_cards/core/upload.py`) - Replace sequential `for x in all_players:` loop with `asyncio.gather` + `Semaphore(8)` - Offload synchronous boto3 S3 calls to thread pool via `run_in_executor` - Increase `fetch_card_image` timeout from 6s → 10s - Add `--concurrency / -j` CLI flag (default: 8) - Progress reporting every 20 completions - Individual card failures logged but don't abort batch ### WP-05: Legacy Upload Script (`check_cards_and_upload.py`) - Same concurrency pattern applied to the legacy script - `CONCURRENCY = 8` constant, timeout increase, progress reporting ## Results (with server-side Phase 0 deployed) | Metric | Before | After | |--------|--------|-------| | Per-card render | ~2.0s | ~0.98s avg | | 800-card upload (sequential) | ~27 min | ~13 min (est) | | 800-card upload (8x concurrent) | N/A | ~2-3 min (est) | ## Supersedes - PR #25 (auto-generated WP-04) - PR #26 (auto-generated WP-05) ## Test plan - [x] Benchmark script runs and produces timing data - [ ] `pd-cards upload s3 -c "2005 Live" --limit 10 -j 8` completes without errors - [ ] Legacy `check_cards_and_upload.py` runs concurrently without errors - [ ] `--concurrency 1` behaves like sequential (regression safety) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cal added 1 commit 2026-03-13 13:48:41 +00:00 
- Replace sequential upload loop with asyncio.gather + Semaphore(8) (WP-04)
- Offload synchronous boto3 S3 calls to thread pool executor
- Increase fetch_card_image timeout from 6s to 10s
- Add --concurrency/-j CLI flag to pd-cards upload
- Add progress reporting every 20 completions
- Individual card failures no longer abort batch
- Apply same concurrency pattern to legacy check_cards_and_upload.py (WP-05)
- Add benchmark script for render pipeline measurements (WP-00)

Target: 800-card upload from ~40 min to <5 min (with server-side
persistent browser deployed).

Refs: #87, #91, #92

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cal added 1 commit 2026-03-13 13:50:54 +00:00 
get_event_loop() is deprecated in Python 3.10+ when called inside
a running coroutine. get_running_loop() is the correct replacement.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cal merged commit 6e20a98226 into main 2026-03-13 15:56:19 +00:00
cal deleted branch feature/render-pipeline-optimization 2026-03-13 15:56:19 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
ai-merged

PR merged by AI ops agent

  
ai-pr-opened

   
ai-reviewed

   
ai-reviewing

   
ai-working
No Label
ai-merged
ai-pr-opened
ai-reviewed
ai-reviewing
ai-working
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cal/paper-dynasty-card-creation#27
No description provided.
Delete Branch "feature/render-pipeline-optimization"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?