Skip to content

rust: add incremental builds and persistent worker pipelining#11

Open
walter-zeromatter wants to merge 1 commit intohermeticbuild:mainfrom
walter-zeromatter:user/wgray/persistent-worker
Open

rust: add incremental builds and persistent worker pipelining#11
walter-zeromatter wants to merge 1 commit intohermeticbuild:mainfrom
walter-zeromatter:user/wgray/persistent-worker

Conversation

@walter-zeromatter
Copy link
Copy Markdown

@walter-zeromatter walter-zeromatter commented Apr 4, 2026

Summary

  • Persistent worker pipelining: Adds a multiplex persistent worker mode to process_wrapper that enables Cargo-style pipelining — downstream crates can start compiling as soon as .rmeta metadata is emitted, without waiting for full codegen. This significantly reduces critical path build times. See DESIGN.md for details on how this works.
  • Incremental compilation: Adds optional incremental compilation support, allowing rustc to reuse prior compilation artifacts for faster rebuilds at the cost of higher up-front build cost

Key changes

  • rust/private/rustc.bzl: Wires up worker pipelining flags, metadata actions, and incremental build support
  • rust/private/incremental.bzl: New file for incremental compilation logic
  • rust/settings/settings.bzl: New settings (experimental_worker_pipelining, experimental_incremental)
  • util/process_wrapper/worker*.rs: Full multiplex worker implementation with pipelining, sandboxing, and cancellation support
  • test/unit/pipelined_compilation/: New tests for worker pipelining and strace-based sandbox compliance verification

Performance

On a 5-crate benchmark (Bazel 8.4.2, no multiplex sandboxing):

  • No pipelining: 20.7s (143 actions)
  • Hollow rlib (existing): 11.4s (251 actions)
  • Worker pipelining: 8.4s (115 actions) — 1.62× faster than no-pipeline, comparable to Cargo (8.2s)

Test plan

  • 10/10 pipelined_compilation tests pass
  • 51 process_wrapper tests pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant