Skip to content

fix deadlock between in-progress builds & delete-crate/release#3248

Merged
GuillaumeGomez merged 1 commit intorust-lang:mainfrom
syphar:deadlock
Mar 15, 2026
Merged

fix deadlock between in-progress builds & delete-crate/release#3248
GuillaumeGomez merged 1 commit intorust-lang:mainfrom
syphar:deadlock

Conversation

@syphar
Copy link
Member

@syphar syphar commented Mar 15, 2026

I'm quite sure this is the reason why we sometimes stop consuming from the index. And it's when we get deletes while we have an in-progress build.

What happens today (simplified):

  • Builder txn: locks queue row (FOR UPDATE SKIP LOCKED) and keeps it locked while build runs.
  • Delete txn: locks releases/metadata rows, then tries DELETE FROM queue ... and waits on builder’s queue lock.
  • Builder’s build path (different DB txn/conn) may try touching releases and waits on delete txn.
  • Now each side waits on the other -> deadlock.

For now the simple solution is to make the delete wait for the build to be finished. I added a configurable timeout.

I want to postpone the better solution to when we have something event-based between crates.io & docs.rs, where I then could easily NACK some events, and handle them later.

@syphar syphar self-assigned this Mar 15, 2026
@syphar syphar requested a review from a team as a code owner March 15, 2026 09:22
@github-actions github-actions bot added the S-waiting-on-review Status: This pull request has been implemented and needs to be reviewed label Mar 15, 2026
@syphar

This comment was marked as outdated.

@GuillaumeGomez
Copy link
Member

Nice, thanks!

@GuillaumeGomez GuillaumeGomez enabled auto-merge (rebase) March 15, 2026 14:01
@GuillaumeGomez GuillaumeGomez merged commit 817010e into rust-lang:main Mar 15, 2026
15 checks passed
@syphar syphar deleted the deadlock branch March 15, 2026 14:39
@github-actions github-actions bot added S-waiting-on-deploy This PR is ready to be merged, but is waiting for an admin to have time to deploy it and removed S-waiting-on-review Status: This pull request has been implemented and needs to be reviewed labels Mar 15, 2026
@syphar syphar removed the S-waiting-on-deploy This PR is ready to be merged, but is waiting for an admin to have time to deploy it label Mar 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants