Add worker-managed pipelined compilation and incremental build support#9
Draft
walter-zeromatter wants to merge 23 commits intohermeticbuild:mainfrom
Draft
Add worker-managed pipelined compilation and incremental build support#9walter-zeromatter wants to merge 23 commits intohermeticbuild:mainfrom
walter-zeromatter wants to merge 23 commits intohermeticbuild:mainfrom
Conversation
…elbuild#3829)" This reverts commit f198dde.
…() (bazelbuild#3816)" This reverts commit 9586468.
…#3) * 0 * Add rust analyzer test coverage
Replace the --rustc-quit-on-rmeta / .rmeta approach with Buck2-style hollow rlibs: the RustcMetadata action runs rustc to completion with -Zno-codegen, emitting a .rlib archive (named -hollow.rlib) that contains only metadata and optimized MIR. Key fixes: - Main Rustc action uses full rlib --extern deps so the SVH embedded in the full rlib matches what downstream binaries expect (avoiding E0460 with non-deterministic proc macros). - RUSTC_BOOTSTRAP=1 is set on both actions: it changes the crate hash, so inconsistent use would cause SVH mismatch even for deterministic crates. - -Ldependency= points to the _hollow/ subdirectory so the full rlib and hollow rlib never appear in the same search path (avoids E0463). - transitive_metadata_outputs are always included in the sandbox so rustc can resolve transitive -Ldependency= references. Adds an SVH mismatch test that demonstrates the problem with non-deterministic proc macros (uses a HashMap-based derive macro whose output varies by OS-seeded random seed).
walter-zeromatter
force-pushed
the
pr/worker-pipelining
branch
from
4de94b9 to
c5e532f
Compare
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
walter-zeromatter
force-pushed
the
pr/worker-pipelining
branch
from
c5e532f to
3bba328
Compare
dzbarsky
force-pushed
the
main
branch
2 times, most recently
from
d97ab75 to
e1ed0e4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements a multiplex persistent worker for rustc that enables true Cargo-style pipelined compilation:
worker.rs,worker_protocol.rs): JSON persistent worker protocol with singleplex and multiplex support, cancel handling, signal handlingworker_pipeline.rs): metadata action starts rustc, returns .rmeta early; full action retrieves running process and copies .rlibworker_sandbox.rs): sandbox-aware output materialization, writable output preparationincremental.bzl):-Cincrementalwith separate cache dirs for metadata/full actions, skips exec-platform and proc-macro targetsrustc.bzl: File objects for--out-dir,--sysroot,-Lviamap_eachexperimental_worker_pipelining,experimental_incrementalBenefits over hollow-rlib pipelining
-Zno-codegen/RUSTC_BOOTSTRAP=1requiredBug fixes applied during review
test/chained_direct_deps/mod1.rs_is_incremental_enabled/is_incremental_enabledinincremental.bzlprepare_outputs/prepare_outputs_sandboxedinto sharedprepare_outputs_implmaterialize_output_filebeforecanonicalizesyscallsRULES_RUST_WORKER_DEBUGenv varexperimental_worker_pipeliningdocstring from 65 lines to 25Test plan