WP-05: Legacy Upload Script Update #92

New Issue

cal · 2026-03-13T04:37:25Z

cal commented

2026-03-13 04:37:25 +00:00

Description

Apply the same concurrency pattern to check_cards_and_upload.py for users who still use the legacy script.

Repo: card-creation
Phase: 0 (Render Pipeline Optimization)
Dependencies: WP-04
Complexity: S

Current State

check_cards_and_upload.py lines 150-293: identical sequential pattern to pd_cards/core/upload.py
Module-level boto3 client (line 27)

Implementation

Refactor the sequential loop to use asyncio.gather + Semaphore (same pattern as WP-04)
Offload synchronous S3 calls to thread pool
Increase fetch timeout to 10s
Add progress reporting

Files

Modify: check_cards_and_upload.py

Tests

Integration: legacy script uploads 10 cards concurrently without errors
Integration: S3 URLs match expected format

Acceptance Criteria

Same concurrency behavior as WP-04
No regression in existing functionality

Plan reference: docs/prd-evolution/PHASE0_PROJECT_PLAN.md WP-05

## Description Apply the same concurrency pattern to `check_cards_and_upload.py` for users who still use the legacy script. **Repo:** `card-creation` **Phase:** 0 (Render Pipeline Optimization) **Dependencies:** WP-04 **Complexity:** S ## Current State - `check_cards_and_upload.py` lines 150-293: identical sequential pattern to `pd_cards/core/upload.py` - Module-level boto3 client (line 27) ## Implementation 1. Refactor the sequential loop to use `asyncio.gather` + `Semaphore` (same pattern as WP-04) 2. Offload synchronous S3 calls to thread pool 3. Increase fetch timeout to 10s 4. Add progress reporting ## Files - **Modify:** `check_cards_and_upload.py` ## Tests - [ ] Integration: legacy script uploads 10 cards concurrently without errors - [ ] Integration: S3 URLs match expected format ## Acceptance Criteria 1. Same concurrency behavior as WP-04 2. No regression in existing functionality **Plan reference:** `docs/prd-evolution/PHASE0_PROJECT_PLAN.md` WP-05

cal added this to the Card Evolution Phase 0 — Render Pipeline Optimization milestone 2026-03-13 04:37:38 +00:00

cal added the

evolution

phase-0

labels 2026-03-13 04:37:43 +00:00

cal referenced this issue

2026-03-13 04:38:00 +00:00

Card Evolution Phase 0 — Render Pipeline Optimization Master Tracker #93

Claude added the

ai-working

label 2026-03-13 13:01:39 +00:00

Claude referenced a pull request from cal/paper-dynasty-card-creation that will close this issue

2026-03-13 13:05:40 +00:00

feat: concurrent upload pipeline for legacy script (WP-05) (#92) #26

Claude commented

2026-03-13 13:05:49 +00:00

PR opened in card-creation repo: cal/paper-dynasty-card-creation#26

Approach: Replaced the sequential for x in all_players loop in main() with asyncio.gather + asyncio.Semaphore(CONCURRENCY). Added functools.partial to offload synchronous boto3 S3 PUT calls to thread pool via loop.run_in_executor. Added report_progress() with lock-protected counter, prints every 20 completions. Increased fetch timeout from 6s → 10s. Added CONCURRENCY = 8 constant. 1 file changed, 179 insertions / 124 deletions.

PR opened in `card-creation` repo: https://git.manticorum.com/cal/paper-dynasty-card-creation/pulls/26 **Approach:** Replaced the sequential `for x in all_players` loop in `main()` with `asyncio.gather` + `asyncio.Semaphore(CONCURRENCY)`. Added `functools.partial` to offload synchronous boto3 S3 PUT calls to thread pool via `loop.run_in_executor`. Added `report_progress()` with lock-protected counter, prints every 20 completions. Increased fetch timeout from 6s → 10s. Added `CONCURRENCY = 8` constant. 1 file changed, 179 insertions / 124 deletions.

Claude added

ai-pr-opened

and removed

ai-working

labels 2026-03-13 13:05:59 +00:00

cal commented

2026-03-16 16:03:43 +00:00

Completed. Legacy script updated alongside WP-04 in card-creation repo:

979f308 — feat: concurrent upload pipeline and benchmarks (Phase 0) (includes check_cards_and_upload.py refactor)
8bddf31 — feat: configurable API URL for local high-concurrency card rendering (adds PD_API_URL env var support)

All acceptance criteria met:

Same concurrency behavior as WP-04 (asyncio.gather + Semaphore, run_in_executor for boto3) ✅
No regression in existing functionality ✅

**Completed.** Legacy script updated alongside WP-04 in `card-creation` repo: - `979f308` — `feat: concurrent upload pipeline and benchmarks (Phase 0)` (includes `check_cards_and_upload.py` refactor) - `8bddf31` — `feat: configurable API URL for local high-concurrency card rendering` (adds `PD_API_URL` env var support) All acceptance criteria met: - Same concurrency behavior as WP-04 (`asyncio.gather` + `Semaphore`, `run_in_executor` for boto3) ✅ - No regression in existing functionality ✅

cal closed this issue

2026-03-16 16:03:48 +00:00

cal referenced this issue

2026-03-16 16:03:53 +00:00

Card Evolution Phase 0 — Render Pipeline Optimization Master Tracker #93

Sign in to join this conversation.