zenroot 4.1.2.zfs.5: daemon-URI pointer cache + template metadata

Two ZFS-backend refinements that build on the v4.1.2.zfs.4 pointer-format groundwork.

What's new

Pointer cache extended to dockerd:// and podman://

enroot import dockerd://<image> and enroot import podman://<image> now produce the same magic-prefixed pointer file as docker:// imports and populate the layer-keyed template cache. Repeat imports hit the cache; enroot create becomes O(zfs clone) for daemon URIs too. Cache key is the daemon-reported image ID (${engine} inspect --format='{{.Id}}') — for registry-pulled daemon images this is identical in shape and meaning to the image-config-sha256 of a docker:// pull, so a daemon import of ubuntu:24.04 shares the cache with a registry import of the same reference for free.

Pointer schema gains an optional manifest-digest field — daemon-local images don't have a registry manifest digest. v1 magic line is unchanged; pointers from v4.1.2.zfs.4 still parse.

Template provenance as ZFS user properties

Each template dataset now carries:

Property	Set on
enroot:imported	every template (RFC3339 UTC)
enroot:last_used	every template (refreshed each clone)
enroot:uri	docker / dockerd / podman templates
enroot:manifest-digest	docker:// templates only
enroot:arch	docker / dockerd / podman templates

Templates are self-describing — zfs get -r enroot:uri tank/enroot/data/.templates shows what's cached without needing the (transient) pointer file. Properties replicate via zfs send -p, so they survive the zfs://<host>/<name> SSH transport.

Idle templates are unmounted

zfs clone <template>@pristine doesn't require the source to be mounted, so cached-but-unused templates now sit at zero mountpoints. Only active container clones (and the .templates / .ephemeral parents) appear in mount(8) output. Significant df / mount output cleanup if you have many cached images.

Inspection one-liner

zfs list -o name,enroot:uri,enroot:imported,enroot:last_used,used \
         -r -d 1 ${POOL}/.templates

Behavior changes

• ZFS backend default is unchanged — enroot import docker://, dockerd://, podman:// all produce a pointer file by default. Set ENROOT_ZFS_IMPORT_FORMAT=squashfs (env or config) for the legacy real-.sqsh artifact, or pass --format=squashfs per invocation.
• dir backend: no change.
• enroot:imported / enroot:uri / etc. are populated only on freshly-installed templates. Templates cached under v4.1.2.zfs.4 stay unstamped until they cycle through the warm/cold sweep or are manually destroyed and re-imported.

Action required for existing admins

None. Existing zfs allow set (delegated in v4.1.2.zfs.3 release notes) covers everything. New optional knob (default pointer):

#ENROOT_ZFS_IMPORT_FORMAT  pointer

Smoke results (spark-f2ff, OpenZFS 2.2.x, arm64)

Check	Result
enroot import dockerd://ubuntu:24.04	173-byte pointer; magic correct; no manifest-digest= line
Cache parity (dockerd:// hits docker:// template)	"Reusing cached template 3c220491de2e"
Cache-hit enroot create from dockerd:// pointer	0.564 s
Repeat-import idempotence	template count unchanged
Eviction recovery (destroy + create)	re-extracts from daemon, succeeds
enroot:imported set on fresh installs	2026-05-02T01:21:28Z ✓
Idle templates not mounted	confirmed
docker:// regression	unchanged; still emits manifest-digest=

Closes

• PR #15 — ZFS template metadata + unmount-when-idle.
• PR #16 — daemon-URI pointer cache.

Packages (aarch64 / arm64 only in this release)

Flavor	Package
Standard	enroot_4.1.2.zfs.5-1_arm64.deb
	enroot+caps_4.1.2.zfs.5-1_arm64.deb
Hardened	enroot-hardened_4.1.2.zfs.5-1_arm64.deb
	enroot-hardened+caps_4.1.2.zfs.5-1_arm64.deb

Plus enroot_4.1.2.zfs.5_aarch64.tar.xz (relocatable tarball) and enroot_4.1.2.zfs.5.orig.tar.xz (Debian source).

Build

git clone --recurse-submodules -b v4.1.2.zfs.5 https://github.com/zeroae/enroot.git
cd enroot
make
sudo make install setcap

For the hardened flavor: make deb PACKAGE=enroot-hardened.

See doc/zfs.md (new "Daemon URIs" subsection and "Inspecting cached templates" block) for full documentation.

zenroot 4.1.2.zfs.4: ZFS pointer-format import

Adds a stable, content-addressable cache key to the ZFS backend's import flow so repeat enroot import docker://<ref> invocations of the same image hit the existing template cache. Closes #13.

What's new

• enroot import docker://<ref> writes a 1-KiB pointer file (magic line enroot-zfs-image:v1) instead of a real squashfs when ENROOT_STORAGE_BACKEND=zfs. The pointer carries the docker manifest digest and the image-config-sha256 (the same stable cache key used by direct enroot create docker://<ref>). Skips mksquashfs entirely.
• enroot create magic-byte-sniffs the pointer, looks up the template by image-config-sha256, and clones — O(zfs clone) ≈ subseconds. Eviction-recovery re-pulls from the registry and validates the freshly-resolved config sha matches the pointer's claim (republished tag → clear "delete and re-import" error).
• --format=squashfs (CLI flag on enroot import) or ENROOT_ZFS_IMPORT_FORMAT=squashfs (env/config) preserves the legacy real-.sqsh behavior bit-for-bit.
• pyxis treats the import artifact as opaque (writes to per-uid runtime dir, deletes after enroot create) — the format change is invisible to existing pyxis configs.

Why it matters

Today's pyxis flow is enroot import docker://... → transient .sqsh → enroot create. Each import runs mksquashfs, whose output bytes vary across invocations even when the underlying image is identical (squashfs internal timestamps and per-build metadata leak through). The resulting .sqsh therefore has a different sha256 every time, and the template cache (keyed by sha256(.sqsh)) misses on every job. With this release, the cache key becomes the docker image config blob sha — stable across operators, invocations, and time.

Behavior changes

• ZFS backend: enroot import docker://<ref> produces a small pointer file by default. Set ENROOT_ZFS_IMPORT_FORMAT=squashfs (or pass --format=squashfs) for the legacy behavior — recommended on nodes that share .sqsh artifacts across nodes that don't share a ZFS pool.
• Default dir backend: unchanged.
• ZFS backend privileged-caller behavior: unchanged.
• Existing real .sqsh files keep working as before (sqsh-sha-keyed path is untouched).

Smoke results (from the merging branch)

Check	Result
Pointer import	260-byte file, magic-prefixed, template populated
Cache-hit enroot create	0.277s (issue's acceptance: subseconds)
Idempotent re-import (same image)	"Reusing cached template", no new template
Pyxis end-to-end (srun --container-image=docker://ubuntu:24.04 × 2)	Second create-step ≈ 0.1s; only one template per image (vs the issue's report of two for the old code)
Eviction recovery	Re-pulled and recovered, 2.3s
--format=squashfs opt-out	Real 62 MiB squashfs
Legacy real .sqsh on ZFS backend	Still works

Action required for existing admins

If you already deployed v4.1.2.zfs.1 through v4.1.2.zfs.3, your zfs allow set already covers everything needed — no new permissions required. New env knob to optionally set in /etc/enroot/enroot.conf:

ENROOT_ZFS_IMPORT_FORMAT  pointer

This is the default; the line is informational. Set to squashfs to opt out.

Packages (aarch64 / arm64 only in this release)

Flavor	Package
Standard	enroot_4.1.2.zfs.4-1_arm64.deb
	enroot+caps_4.1.2.zfs.4-1_arm64.deb
Hardened	enroot-hardened_4.1.2.zfs.4-1_arm64.deb
	enroot-hardened+caps_4.1.2.zfs.4-1_arm64.deb

Plus enroot_4.1.2.zfs.4_aarch64.tar.xz (relocatable tarball) and enroot_4.1.2.zfs.4.orig.tar.xz (Debian source).

Closes

• #13 — ZFS backend: stable template key for repeat enroot import docker:// invocations (pyxis workflow).

Build

git clone --recurse-submodules -b v4.1.2.zfs.4 https://github.com/zeroae/enroot.git
cd enroot
make
sudo make install setcap

For the hardened flavor: make deb PACKAGE=enroot-hardened.

See doc/zfs.md (new "Pointer-format import" section) for the full design and trust-boundary discussion.

zenroot 4.1.2.zfs.3: OpenZFS 2.2.x compat

Makes the ZFS backend work on Ubuntu 24.04 LTS (and any other distro pinned to OpenZFS 2.2.x userspace) by avoiding zfs clone -u, which was added in OpenZFS 2.3. Closes #11.

What's new

• src/storage_zfs.sh replaces zfs clone -u with zfs clone -o canmount=noauto everywhere in the create / ephemeral-clone path. Both achieve the same effect — skip the implicit ZFS-driven mount on creation — but canmount=noauto works on OpenZFS 2.0+. The enroot-zfs-mount helper handles the actual mount(2) afterward and is unaffected by canmount.
• doc/zfs.md zfs allow recipe updated: adds canmount (so zfs clone -o canmount=... succeeds for unprivileged callers) and userprop (so the template warm/cold lifecycle's zfs set enroot:last_used=... succeeds — this was a latent gap from v4.1.2.zfs.1).

Behavior changes

None at the user-facing level. ZFS backend continues to work on OpenZFS 2.3+; now also works on 2.2.x.

Action required for existing admins

If you already deployed v4.1.2.zfs.1 or v4.1.2.zfs.2, extend your delegation to include canmount and userprop:

zfs allow user canmount,userprop tank/enroot

The full recommended delegation is now:

zfs allow user create,mount,clone,destroy,snapshot,rename,promote,receive,readonly,hold,release,canmount,userprop tank/enroot

Packages (aarch64 / arm64 only in this release)

Flavor	Package
Standard	enroot_4.1.2.zfs.3-1_arm64.deb
	enroot+caps_4.1.2.zfs.3-1_arm64.deb
Hardened	enroot-hardened_4.1.2.zfs.3-1_arm64.deb
	enroot-hardened+caps_4.1.2.zfs.3-1_arm64.deb

Plus enroot_4.1.2.zfs.3_aarch64.tar.xz (relocatable tarball) and enroot_4.1.2.zfs.3.orig.tar.xz (Debian source).

Closes

• #11 — ZFS backend incompatible with OpenZFS 2.2.x userspace (Ubuntu 24.04 LTS).

Build

git clone --recurse-submodules -b v4.1.2.zfs.3 https://github.com/zeroae/enroot.git
cd enroot
make
sudo make install setcap

For the hardened flavor: make deb PACKAGE=enroot-hardened.

See doc/zfs.md for the full design and trust-boundary discussion.

zenroot 4.1.2.zfs.2: enroot-zfs-mount helper for pyxis/Slurm

Adds enroot-zfs-mount, a cap_sys_admin-elevated helper that lets unprivileged callers mount the ZFS datasets enroot creates. Closes #9 (pyxis/Slurm post-privilege-drop enroot create 10-minute timeout).

What's new

• bin/enroot-zfs-mount — small static-pie/musl helper that validates and performs mount(2) / umount2(2) of ZFS datasets owned by the calling user. Reads its config exclusively from /etc/enroot/enroot.conf (NOT from user-controlled config). Validation chain: dataset must be under ENROOT_DATA_PATH's parent dataset; the dataset's mountpoint property must be under ENROOT_DATA_PATH; idempotent if already mounted at the expected path.
• Shipped via enroot+caps and enroot-hardened+caps; the postinst grants cap_sys_admin+pe.
• src/storage_zfs.sh switched to zfs create -u / zfs clone -u / zfs receive -u (skip auto-mount) and calls the helper for the actual mount; destroy paths gain a helper unmount before zfs destroy.

With +caps installed, all unprivileged ZFS-backed flows (enroot create, enroot load, enroot start <image>, enroot remove) work end-to-end — including from inside slurmstepd post-privilege-drop, which is what pyxis needs.

Behavior changes

• enroot create / load / start <image> / remove now succeed for unprivileged callers when ENROOT_STORAGE_BACKEND=zfs and enroot+caps is installed. Previously they timed out or required root.
• Default dir backend behavior unchanged. ZFS backend's privileged-caller behavior unchanged.

Trust posture

The helper has cap_sys_admin. Validation bounds that trust to enroot's own operations:

• The dataset arg must be under ENROOT_DATA_PATH's parent (defends against tank/admin/secrets).
• The dataset's mountpoint property must be under ENROOT_DATA_PATH (defends against a self-owned dataset with mountpoint=/usr/local/bin).
• Already-mounted check refuses double-mounts.
• System-config-only: the helper reads /etc/enroot/enroot.conf, ignoring ~/.config/enroot/enroot.conf so a malicious caller can't redirect ENROOT_DATA_PATH.

Packages (aarch64 / arm64 only in this release)

Flavor	Package
Standard	enroot_4.1.2.zfs.2-1_arm64.deb
	enroot+caps_4.1.2.zfs.2-1_arm64.deb
Hardened	enroot-hardened_4.1.2.zfs.2-1_arm64.deb
	enroot-hardened+caps_4.1.2.zfs.2-1_arm64.deb

Plus enroot_4.1.2.zfs.2_aarch64.tar.xz (relocatable tarball) and enroot_4.1.2.zfs.2.orig.tar.xz (Debian source).

Closes

• #9 — ZFS backend: support pyxis (Slurm) workflows where enroot runs after privilege-drop to the job user.

Build

git clone --recurse-submodules -b v4.1.2.zfs.2 https://github.com/zeroae/enroot.git
cd enroot
make
sudo make install setcap

For the hardened flavor: make deb PACKAGE=enroot-hardened.

Admin setup for unprivileged ZFS

/etc/enroot/enroot.conf (system, root-owned) must set ENROOT_DATA_PATH and ENROOT_STORAGE_BACKEND:

ENROOT_STORAGE_BACKEND zfs
ENROOT_DATA_PATH       /var/lib/enroot

ZFS pool with delegations:

zfs create -o compression=zstd -o atime=off -o xattr=sa -o acltype=posixacl \
           -o mountpoint=/var/lib/enroot tank/enroot
zfs allow user create,mount,clone,destroy,snapshot,rename,promote,receive,readonly,hold,release tank/enroot

Install both enroot and enroot+caps. The helper picks up cap_sys_admin+pe via the +caps postinst.

See doc/zfs.md for the full design and trust-boundary discussion.

zenroot 4.1.2.zfs.1: ZFS storage backend

First release of the zenroot fork — adds an optional ZFS storage backend on top of upstream NVIDIA enroot 4.1.2.

Default dir backend is byte-for-byte unchanged; ZFS is opt-in via ENROOT_STORAGE_BACKEND=zfs.

What's new

When ENROOT_STORAGE_BACKEND=zfs:

• Clone-on-create — first enroot create foo.sqsh extracts to a shared template; subsequent creates from the same image are near-instant zfs clones. (Plan A — #1)
• Template warm/cold lifecycle — templates survive enroot remove for ENROOT_TEMPLATE_WARM_SECONDS (default 7 days), pressure-evicted LRU once ENROOT_TEMPLATE_PRESSURE_THRESHOLD of the templates dataset's quota is reached (default 80%). (Plan B — #5)
• .zfs image format — enroot create foo.zfs and enroot export --format=zfs for sneakernet/S3/USB ZFS-host-to-ZFS-host transfers. (Plan C — #7)
• zfs://[USER@]HOST/NAME URI transport — enroot load zfs://... to pull, enroot export NAME zfs://... to push, enroot import zfs://... to fetch as a .zfs or .sqsh file. SSH-based; both ends must run enroot with the ZFS backend. (Plan D — #8)
• Ephemeral start <image> ZFS path — enroot start foo.sqsh (no prior create) uses zfs clone to a throwaway dataset instead of squashfuse + overlayfs. PID semantics, signal forwarding, and enroot exec PID all work. Cleanup via the kernel's orphaned-process-group SIGHUP rule. (Plan E — #2)
• Docker layer load on ZFS — enroot load docker://... no longer requires ENROOT_NATIVE_OVERLAYFS=y on the ZFS backend; the merged image is materialized into a ZFS template (cached by image config sha256) and cloned for the user. (Plan F — #3)

See doc/zfs.md for the design and doc/plans/ for the implementation plans.

Tracking issues

• #4 — Plan G: per-layer ZFS clone chain for enroot load docker:// (opt-in, future).
• #6 — Tighten end-to-end verification of the template ENOSPC retry path.

Packages (aarch64 / arm64 only in this release)

Flavor	Package	Description
Standard	enroot_4.1.2.zfs.1-1_arm64.deb	Main utility, helper binaries, standard config. Open file descriptors inherited; Spectre v2/v4 mitigations disabled (HPC tradeoff, matches upstream's "standard" flavor).
	enroot+caps_4.1.2.zfs.1-1_arm64.deb	Grants extra capabilities to unprivileged users for image import/conversion. Install alongside enroot.
Hardened	enroot-hardened_4.1.2.zfs.1-1_arm64.deb	Same code, hardened build flags.
	enroot-hardened+caps_4.1.2.zfs.1-1_arm64.deb	+caps companion for the hardened flavor.

A relocatable tarball enroot_4.1.2.zfs.1_aarch64.tar.xz is also attached for non-Debian environments.

Build

git clone --recurse-submodules -b v4.1.2.zfs.1 https://github.com/zeroae/enroot.git
cd enroot
make
sudo make install

For the hardened flavor:

make deb PACKAGE=enroot-hardened

For non-aarch64 architectures, make deb produces a Debian package for the host architecture.

Admin setup for ZFS

Per doc/zfs.md, the rough recipe:

zfs create -o compression=zstd -o atime=off -o xattr=sa -o acltype=posixacl \
           -o mountpoint=/var/lib/enroot tank/enroot
zfs create tank/enroot/templates
zfs set quota=200G tank/enroot/templates
zfs allow user create,mount,clone,destroy,snapshot,rename,promote,receive tank/enroot

Then in /etc/enroot/enroot.conf:

ENROOT_DATA_PATH /var/lib/enroot/<user>
ENROOT_STORAGE_BACKEND zfs

Linux mount(2) caveat: unprivileged enroot create and enroot load require CAP_SYS_ADMIN for the kernel's mount(2) syscall, regardless of zfs allow delegation. Practical workaround: privileged create/load (sudo or systemd-User), unprivileged start/exec/remove.