refactor(ci): extract mise/asdf caching to e2e-only cache#1124
refactor(ci): extract mise/asdf caching to e2e-only cache#1124
Conversation
Split the shared daily cache into a slim daily cache (for all jobs) and an e2e-specific cache (for machine-executor jobs that need tool binaries). This reduces cache restore size for non-e2e Docker jobs while preserving tool caching for e2e jobs. - Revert v2 cache key prefix back to v1 (stale v2 entries expire in 15d) - Remove ~/.asdf and ~/.local/share/mise from daily cache - Add v1-e2e-daily-cache with ~/.asdf and ~/.local/share/mise - Add restore_e2e_cache parameter to setup_environment (default: false) - Pass restore_e2e_cache: true from e2e job Refs: DT-5282 Assisted-By: claude-opus 4.6 via OpenCode
marnagy
left a comment
There was a problem hiding this comment.
Since we will be reverting the daily cache key, is it possible that there is OLD cache in CI that might get used between us releasing rc version of this repo and some people running their pipelines?
I'm not yet fully sure how the cache in CI works.
|
I tested this on an internal repo (detailed results) and ran into a few problems. All jobs run Docker images don't ship repo-specific tool versions. Every job runs The e2e cache is unreachable by e2e jobs
This PR removes it for all non-e2e jobs. That's a regression from even the pre-#1117 state. Numbers Total stored: 3.3 GiB (1.3 + 2.0) vs 3.2 GiB with #1117. Docker jobs go from 195s to 608s. E2e jobs get nothing due to the arch mismatch. I think #1117's single cache approach is simpler and works. If cache size near the 3 GiB limit becomes a concern, keying on |
What this PR does / why we need it
PR #1117 added
~/.local/share/miseto the shared daily cache, bloating it for every job (~500MB-1GB extra) even though only e2e jobs (machine executor) need cached tool installs. Non-e2e jobs run in Docker where tools are pre-installed.This PR splits the daily cache into two:
v1-daily-cache—~/.cache,~/.outreach/.cache— restored by all jobsv1-e2e-daily-cache—~/.asdf,~/.local/share/mise— restored only by e2e jobs (opt-in viarestore_e2e_cacheparameter)The
v2key prefix from #1117 is reverted tov1. No manual cache invalidation needed -- stalev2entries age out in 15 days since caches regenerate daily via{{ epoch }}.Jira ID
DT-5282
Notes for your reviewers
save_cachejob runs on a Docker executor but~/.asdfis populated via the pre-installed toolchain. Docker and machine executors share OS/arch, so cached binaries are portable.restore_e2e_cachedefaults tofalserather than being inferred frommachine: true, keeping the coupling loose.Rovo Dev code review: Rovo Dev has reviewed this pull request
Any suggestions or improvements have been posted as pull request comments.