feat: add SyncManager for delta asset sync via GitHub Tree API#119
Open
raman78 wants to merge 1 commit intoSTOCD:mainfrom
Open
feat: add SyncManager for delta asset sync via GitHub Tree API#119raman78 wants to merge 1 commit intoSTOCD:mainfrom
raman78 wants to merge 1 commit intoSTOCD:mainfrom
Conversation
Uses GitHub's Git Tree API (SHA1 + file size) to detect changed or missing files and downloads only what's needed — rather than re-fetching everything on every startup. Key improvements over the current download_image_list approach: - Delta sync: skips files already present with matching SHA1 + size - Circuit breaker: after 3× HTTP 403, the source (GitHub or wiki) is disabled for the session — prevents Cloudflare stall storms - Retry + stall timeout: transient errors retried once; 10s stall = abort - Tree cache: GitHub tree response cached for 1h, avoiding redundant API calls on repeated startups - Ship images wiki fallback: if GitHub 404s a ship image, falls back to stowiki Special:FilePath (same behaviour as Downloader, now with retry) Integration: - SyncManager instantiated in SETS.__init__ alongside Downloader - populate_cache() calls sync.run() before load_cargo_cache(), so cargo JSON files on disk are fresh before the 7-day TTL check runs - ImageManager.download_images() is unchanged: it handles wiki-only groups (boff abilities, skill icons) not covered by GitHub Tree Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The current
Downloader.download_image_list()approach re-downloads every asset that is missing from the local folder, but has no way to detect changed files (same filename, updated content). It also has no protection against Cloudflare 403 storms — a single block can cause hundreds of individual timeouts before giving up.Solution
New
src/syncmanager.pymodule with aSyncManagerclass that:Delta sync via GitHub Tree API — fetches the SHA1 + file size for every file in
STOCD/SETS-Data, compares against local files, and downloads only what changed or is missing. The tree response is cached for 1 hour so repeated startups don't hit the API every time.Circuit breaker for 403 errors — after 3× HTTP 403 from GitHub or stowiki, the source is disabled for the rest of the session. Prevents Cloudflare stall storms where hundreds of requests queue up and time out one by one.
Retry + stall timeout — transient network errors are retried once with a 10 s stall timeout per request.
Ship image wiki fallback — if GitHub returns 404 for a ship image, falls back to
stowiki Special:FilePath(same behaviour asDownloader, now with circuit breaker + retry).Integration
SyncManageris instantiated inSETS.__init__alongsideDownloader(2 lines)populate_cache()callssync.run()beforeload_cargo_cache(), so cargo JSON files on disk are fresh before the 7-day TTL check runs — avoiding unnecessary stowiki cargo fetches on a healthy installImageManager.download_images()is unchanged: it continues to handle wiki-only groups (boff abilities, skill icons) not covered by the GitHub treeNo new dependencies
syncmanager.pyonly uses the standard library (hashlib,queue,threading) andrequests(already a dependency viaDownloader).