Renaming the project root dir invalidates compilation cache

[alubbe]

I was surprised to learn that renaming the project root dir invalidates the compilation cache. This is something that happens during every CI and deployment run, making it run quite a bit slower when none of the rust code was actually changed. Is this intentional / is there a way to keep the cache fresh when renaming the directory?

The behaviour can be reproduced as follows (stable 1.12.1):

/$ cargo new rrrrrr
     Created library `rrrrrr` project
/$ cd rrrrrr/
/rrrrrr$ cargo build
   Compiling rrrrrr v0.1.0 (file:///rrrrrr)                            <-- Code gets compiled
    Finished debug [unoptimized + debuginfo] target(s) in 0.11 secs 
/rrrrrr$ cargo build
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs     <-- Code does not re-compile
/rrrrrr$ cargo build
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs     <-- Code does not re-compile
/rrrrrr$ cd ..
/$ mv rrrrrr/ rrrrrr2/
/$ cd rrrrrr2/
/rrrrrr2$ cargo build
   Compiling rrrrrr v0.1.0 (file:///rrrrrr2)                           <-- Code re-compiles
    Finished debug [unoptimized + debuginfo] target(s) in 0.11 secs

[alexcrichton]

I believe Cargo stores absolute path names in various places of its crate cache, which is likely leading to this.

[alubbe]

That sounds like a likely source. In what cases wouldn't it be preferable to work with relative paths (to the directory containing Cargo.toml)?

[alexcrichton]

Ideally, yeah, although I don't think this would be easy to do at all in Cargo, the absolute paths are plumbed pretty far throughout the whole system. If possible I'd recommend getting the same directory working on CI (like Travis)

[KalitaAlexey]

Dear @alexcrichton,
Is incremental compilation data for the project sources stored inside the project dir?
I think it should use relative paths for these cases.
The cache also will be invalidated if you move the project dir to another place. I don't like it.
Is a fix hard enough?

[alexcrichton]

The problem here is that we store everything as an absolute path in memory, and then that gets hashed and/or serialized to disk. Changing the in-memory representation is likely to be difficult, so is changing how it's serialized/hashed. Unfortunately the fix here will probably be hard.

[alubbe]

Isn't it a matter of changing the places that pass the absolute path to the hashing function, so that they pass the relative path instead? Are there a lot of those?

[alexcrichton]

Patches are of course always welcome! I probably don't personally have time to work on this, but I can help out if needed.

[alubbe]

I might not have time this week, either, but I'd love to have a deeper understanding of what changes are involved and how much work it would be. I've done some searching around the repo to figure out where this hash is generated - if you could point in the right direction of the relevant files/code bits, that would super helpful!

[alexcrichton]

A PackageId ends up being hashed a bunch of places, which transitively contains a SourceId, containing a Url, representing a Path. That's probably absolute.

[alubbe]

I took a look at the source code and I believe these are the relevant places. I tested it locally by hard-coding fixed strings/paths at these places, but a real implementation would replace these absolute paths with relative ones that do not change when the whole directory is moved.
https://github.com/alubbe/cargo/blob/master/src/cargo/core/source.rs#L441
https://github.com/alubbe/cargo/blob/master/src/cargo/core/package.rs#L59
https://github.com/alubbe/cargo/blob/master/src/cargo/core/manifest.rs#L183

However, that only solves the first fingerprint. cargo will then check the fingerprint of src/lib.rs, but it expects to find it at the old, absolute path. Could someone help me find out where that happens and also, how to convert these all these absolute paths to relative ones?

[alexcrichton]

I believe Target::src_path is relative (or we could easily make it relative), Package::manifest_path AFAIK doesn't make its way to fingerprints, and then SourceId::url is the hard one to deal with.

Fingerprints being expected at absolute paths happens here -- https://github.com/alubbe/cargo/blob/f6500c6b6480c89492773bff6d41c4abf6c5ee15/src/cargo/ops/cargo_rustc/fingerprint.rs#L672-L687

Which... makes me think that this may almost not ever work?

[matklad]

In intelliJ there's also a problem of project local paths. It's solved in a hackish, but simple and transparent way: in memory, all paths are absolute. During serialization, any string which contains the absolute path to the project is replaced with %PROJECT_DIR%, and during deserialization it is restored back to the current project directory.

[alexcrichton]

@matklad that seems like a great solution!

[alubbe]

IIUC we can re-use that approach so that not only is the absolute path replaced before serialization, but also before hashing (in the places I mentioned before).
@matklad what are the chances you could contribute your solution as a PR to cargo?