Build stackage2nix in NixOS sandbox

[4e6]

Unable to build nix/stackage2nix on NixOS with nix.useSandbox enabled.

nix.useSandbox
If set, Nix will perform builds in a sandboxed environment that it will set up automatically for each build. This prevents impurities in builds by disallowing access to dependencies outside of the Nix store. This isn't enabled by default for performance. It doesn't affect derivation hashes, so changing this option will not trigger a rebuild of packages.

[4e6]

related #40

[4e6]

Currently, I see no ways of sandboxing the stackage2nix wrapper. See the issues below.

stackage2nix wrapper requires following dependencies to be fetched, see nix/lib.nix

• fpco/lts-haskell
• fpco/stackage-nightly
• commercialhaskell/all-cabal-hashes
• hackage-db at https://hackage.haskell.org/01-index.tar.gz

To be able to satisfy the sandbox requirements, all these dependencies should be prefetched before the build by the standard nix-prefetch-scripts.

Stackage config files

Only files are needed, so fetchgit can be used to fetch fpco/lts-haskell and fpco/stackage-nightly dependencies. The only issue with this approach is less convenient updates, because it would require updating revision and hash for both repos, instead of bumping single cacheVersion parameter.

all-cabal-hashes

To build the exact copy of stackage packages set, stackage2nix searches for a project definitions in all-cabal-hashes by a hash defined in stackage config (single version of the package may have different revisions). In order to do so, all-cabal-hashes should be fetched with git metadata. Due to NixOs/nixpkgs #8567 there is no reliable way to do this with fetchgit.
The solution might be to fetch zip archive of a particular version. AFAIK, Github is able to create such links but only for archives containing project files, without metadata.

hackage-db

An issue with hackage-db is that URL doesn't have a particular version to put in fetchurl script. I'm assuming that hackage-db could be recreated from all-cabal-hashes repo, but I'm not sure how. Other solution would be to fetch versioned db from some other place.

[kirelagin]

all-cabal-hashes stuff in callHackage: https://github.com/NixOS/nixpkgs/blob/43a62b66d0175b10fd3cc6f1fabdec9d205c171c/pkgs/development/haskell-modules/make-package-set.nix#L126

[4e6]

Regarding the non-determinism of all-cabal-hashes.

I've found this old comment on the original issue thread. The idea is to unpack the git objects and store them uncompressed bendlas/nixpkgs@4b9c24a
We should be able to do this unpacking as a postUnpack build step.

Downsides:

• will lead to increased size of git repository

Upsides:

• deterministic fetchgit
• (should be checked) we can access those objects through the libgit interface (no changes are needed for stackage2nix itself)

[zimbatm]

Do you really care about the git history or is it because the tool wants to query the current reference of the checkout?

For the latter, it could make sense to re-build a fake .git database with only the following files:

.git/HEAD -> ref: refs/heads/master
.git/refs/heads/master -> e843a2271a972b8cb6401e67f25d22c8f6fa68cb

[binarin]

@zimbatm It's the mapping from sha1 to a file content that is needed.

[zimbatm]

so the tool is not looking at the checked-out content but querying the git database directly instead?

if you go down the fetchgit + unpacked blobs maybe you can make it smaller by using a shallow copy of the database.

given the level of effort involved it could make sense to patch upstream as well

[binarin]

@zimbatm The full history is still needed, as we need all blobs reachable from the required commit.

I've discussed this with @4e6, and I think I'll just make a small tool that will create a canonical representation of git .pack file. So if everything (branches, tags) is properly pruned before that, the result will be a working git checkout that is also reproducible. I'll experiment with this approach here. If it'll work out, I try to do the same in the fetchgit itself.

[4e6]

I tried the approach referenced in my previous comment with the unpacking of git objects bendlas/nixpkgs@4b9c24a

This led to the increase of all-cabal-hashes checkout size from 1.6 Gb to 16 Gb, which is not acceptable.