fix: disable git auto-gc during dependency temp-dir clone+checkout#313
Closed
mchen-sentry wants to merge 1 commit intomainfrom
Closed
fix: disable git auto-gc during dependency temp-dir clone+checkout#313mchen-sentry wants to merge 1 commit intomainfrom
mchen-sentry wants to merge 1 commit intomainfrom
Conversation
We're seeing intermittent `shutil.Error` from `_checkout_dependency`'s copytree on getsentry/sentry CI now that runners ship git 2.54.0, whose geometric maintenance default writes more commit-graph chain fragments in the background. The fragments are written via create-then-rename and race with copytree's scandir-then-copy2 walk. #312 patched the related rmtree-on-cleanup race but doesn't help here since copytree doesn't go through TemporaryDirectory's cleanup. We disable auto-gc/maintenance/commit-graph writes on the two git calls that run inside the temp dir, which stops the background bookkeeping from ever touching files we're about to copy out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We're seeing intermittent acceptance failures on getsentry/sentry where
devservices upblows up withshutil.Errorfrom_checkout_dependency. Attempt 1 of run 25064611924 / job 73428191911:The same job logs
git version 2.54.0, and 2.54.0's release notes call out thatgit maintenancenow defaults to the geometric strategy. The new default writes moretmp_graph_*andcommit-graph-chain.lockfragments via the create-then-rename pattern git uses for atomic commit-graph chain updates. After ourgit checkoutreturns,gc.autofires in the background and starts writing those fragments whilecopytreeis still walking the temp tree:os.scandirenumerates a tmp file, thenshutil.copy2hitsENOENTbecause git has already renamed it.Fundamentally none of this background bookkeeping is useful inside a temp dir we're about to bulk-copy: any work either gets carried over verbatim during
copytreeor redone later when the cache is actually used. #312 patched a related race inTemporaryDirectory's implicit cleanup (thermtreewhen thewithblock exits), butcopytreedoesn't go through that cleanup machinery soignore_cleanup_errors=Truedoesn't apply here. We disable auto-gc on the two git invocations inside the temp dir (the partial clone and the checkout) by passing-c gc.auto=0 -c maintenance.auto=false -c fetch.writeCommitGraph=false. That stops the background gc from ever running against the temp dir, killing both thiscopytreerace and thermtreerace #312 was patching around.(Considered also adding
ignore=shutil.ignore_patterns('tmp_*', '*.lock')to thecopytreeas belt-and-suspenders, but disabling gc at the source is the cleaner fix; the patterns approach would still have a residual race for any non-tmp file that maintenance might touch.)