Skip to content

feat: add SyncManager for delta asset sync via GitHub Tree API#119

Open
raman78 wants to merge 1 commit intoSTOCD:mainfrom
raman78:feature/syncmanager
Open

feat: add SyncManager for delta asset sync via GitHub Tree API#119
raman78 wants to merge 1 commit intoSTOCD:mainfrom
raman78:feature/syncmanager

Conversation

@raman78
Copy link
Copy Markdown

@raman78 raman78 commented Mar 29, 2026

Problem

The current Downloader.download_image_list() approach re-downloads every asset that is missing from the local folder, but has no way to detect changed files (same filename, updated content). It also has no protection against Cloudflare 403 storms — a single block can cause hundreds of individual timeouts before giving up.

Solution

New src/syncmanager.py module with a SyncManager class that:

  1. Delta sync via GitHub Tree API — fetches the SHA1 + file size for every file in STOCD/SETS-Data, compares against local files, and downloads only what changed or is missing. The tree response is cached for 1 hour so repeated startups don't hit the API every time.

  2. Circuit breaker for 403 errors — after 3× HTTP 403 from GitHub or stowiki, the source is disabled for the rest of the session. Prevents Cloudflare stall storms where hundreds of requests queue up and time out one by one.

  3. Retry + stall timeout — transient network errors are retried once with a 10 s stall timeout per request.

  4. Ship image wiki fallback — if GitHub returns 404 for a ship image, falls back to stowiki Special:FilePath (same behaviour as Downloader, now with circuit breaker + retry).

Integration

  • SyncManager is instantiated in SETS.__init__ alongside Downloader (2 lines)
  • populate_cache() calls sync.run() before load_cargo_cache(), so cargo JSON files on disk are fresh before the 7-day TTL check runs — avoiding unnecessary stowiki cargo fetches on a healthy install
  • ImageManager.download_images() is unchanged: it continues to handle wiki-only groups (boff abilities, skill icons) not covered by the GitHub tree

No new dependencies

syncmanager.py only uses the standard library (hashlib, queue, threading) and requests (already a dependency via Downloader).

Uses GitHub's Git Tree API (SHA1 + file size) to detect changed or
missing files and downloads only what's needed — rather than re-fetching
everything on every startup.

Key improvements over the current download_image_list approach:
- Delta sync: skips files already present with matching SHA1 + size
- Circuit breaker: after 3× HTTP 403, the source (GitHub or wiki) is
  disabled for the session — prevents Cloudflare stall storms
- Retry + stall timeout: transient errors retried once; 10s stall = abort
- Tree cache: GitHub tree response cached for 1h, avoiding redundant API
  calls on repeated startups
- Ship images wiki fallback: if GitHub 404s a ship image, falls back to
  stowiki Special:FilePath (same behaviour as Downloader, now with retry)

Integration:
- SyncManager instantiated in SETS.__init__ alongside Downloader
- populate_cache() calls sync.run() before load_cargo_cache(), so cargo
  JSON files on disk are fresh before the 7-day TTL check runs
- ImageManager.download_images() is unchanged: it handles wiki-only
  groups (boff abilities, skill icons) not covered by GitHub Tree

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant