build --release takes forever on latest rust release

[Qumeric]

I have a project of around 10k lines and 1000 dependency crates. I cannot share it because it's closed source.

I was working on it for a few months and previously compiled it with stable, beta and nightly rust.

I decided to update rust to latest version. I run cargo build --release and it stuck at

   Compiling reth-block-validator v0.1.0 (/path/...)
   Building [=====================> ] 1157/1161: my-crate-name

I tried to build my-crate-name solely (full project is workspace) but it stuck in a same way.

I tried to do it with macOS + lld, linux + mold, linux + ld. Linux were 3 different machines with 5x00X Ryzen cpus.

It always stuck (I waited at least 10 and up to 60 minutes). Both in latest (installed with rustup) stable, beta and nightly.

I tried different standard things like cargo clean but nothing helped.

The project builds without --release just fine and builds with --release in rust 1.74.1 (and some previous versions of nightly and beta). It takes ~4 minutes for clean --release build and ~1.5 minutes for clean debug build.

I may try git bisect on compiler versions later if I will have time for that. Although I am not completely sure if it's compiler issue, could be cargo issue as well?

[ehuss]

Thanks for the report!

I may try git bisect on compiler versions later if I will have time for that. Although I am not completely sure if it's compiler issue, could be cargo issue as well?

We have a tool called cargo-bisect-rustc specifically for bisecting these issues. There is a --timeout option for diagnosing slow/hung builds (documented here, though it currently does not work on macOS).

[Qumeric]

I finally run bisect tool, thanks for pointing on it.

Report:

********************************************************************************
Regression in 2e5a9dd6c9eaa42f0684b4b760bd68fc27cbe51b
********************************************************************************

Attempting to search unrolled perf builds
ERROR: couldn't find perf build comment
==================================================================================
= Please file this regression report on the rust-lang/rust GitHub repository     =
=        New issue: https://github.com/rust-lang/rust/issues/new                 =
=     Known issues: https://github.com/rust-lang/rust/issues                     =
= Copy and paste the text below into the issue report thread.  Thanks!           =
==================================================================================

searched nightlies: from nightly-2023-09-01 to nightly-2024-01-01
regressed nightly: nightly-2023-10-03
searched commit range: e0d7ed1...2e5a9dd
regressed commit: 2e5a9dd

bisected with cargo-bisect-rustc v0.6.8

Host triple: x86_64-unknown-linux-gnu
Reproduce with:
bash
cargo bisect-rustc --end=2024-01-01 --timeout 300 -- build --release

Also, A bit of feedback on this tool:
RESULT: nightly-2023-09-01, ===> No is a bit confusing, I am not sure what "No" means? Does it mean success or fail or it depends on what it's trying to check? I suggest changing it to something like "Timeout", "ICE", "Success" etc. (perhaps in addition to Yes/No).

I guess that "No" means "No failure" and "Yes" means "Failure" but I only understood it after the run.

[Qumeric]

Tried to build manually with nightly-2023-10-02 and can confirm it works.

Also tried the latest nightly and beta and it is still broken. Pretty sure the latest stable is also still broken but I did not check it.

[ehuss]

Thanks for investigating.

cc @nikic @InnovativeInventor, this seems to be caused by #102099. Is this kind of regression expected? Are there additional tasks for investigating why it appears to be getting stuck?

One possibility that comes to mind in this situation is that maybe it is doing significantly more inlining, and possibly running out of memory and hitting swap.