Allow linking against dylibs in LTO mode

[comex]

I would like to build a binary that links against some crates using LTO and others dynamically. (Why? Because I have a dynamic library that contains all of LLVM and is 30MB, slowing down linking. Note that this is not LLVM's own dynamic library, it's a Rust crate compiled to dylib that itself links LLVM statically. One might imagine more generic use cases, e.g. LTOing small dependencies and dynamically linking large ones.) Here's a model:

xa.rs:

#![crate_type = "bin"]
extern crate xb;
extern crate xc;
pub fn main() { xb::b(); xc::c(); }

xb.rs:

#![crate_type = "rlib"]
pub fn b() {}

xc.rs:

#![crate_type = "dylib"]
pub fn c() {}

This can be built successfully without LTO, albeit only if the dylib links libstd dynamically (otherwise you get "cannot satisfy dependencies so std only shows up once"):

rustc ../xc.rs -C prefer-dynamic -L. && rustc ../xb.rs -L. && rustc ../xa.rs -L.

The binary xa contains the code from xa.rs and xb.rs linked statically (but with a conventional linker rather than LTO), and dynamically links against libxc.dylib and libstd.

But if the last command includes -C lto, I just get:

error: could not find rlib for: `xc`

It would be nice if rustc supported this.

In principle support could also be added for combining LTO and non-LTO static linking, but that would require some method to identify which is desired, since both types refer to .rlib files.

[alexcrichton]

This seems like the most reasonable interpretation of "LTO for a dylib" to me, but there'd be a few caveats.

Right now when we create a dynamic Rust library, we keep track of what rlibs are included statically inside of it. That way if anything else needs to rlibs, we know that it get get all the symbols through the dylib. LTO, however, typically internalizes symbols that would otherwise be public, so this would no longer be true. Essentially there'd be no way to re-link against the dylib produced as we wouldn't be sure what symbols survived.

This may, however, be a good addition to my rdylib RFC? That would add a crate type for "a dylib which we can no longer link to", and this seems like the most reasonable interpretation of LTO if there are dylib dependencies as well.

[comex]

@alexcrichton I just re-read your comment and I'm a little confused - what you're talking about (producing dylibs) seems basically orthogonal to my use case (linking against them). With LTO enabled, you could link a cdylib against only rlibs, an executable against a combination of rlibs and rdylibs, or a cdylib against a combination of rlibs and rdylibs. So I don't see why this would be added to the rdylib RFC.

[alexcrichton]

Yeah perhaps not, it basically stays the same that you can LTO cdylibs (regardless of dependencies) and you can't LTO rdylibs.

[bjorn3]

Since #101403 this is now possible with -Zdylib-lto.