✨ Show more descpritive errors in the CLI when the engine fails to start

[caniko]

What are you trying to do?

I use podman over docker, and I just can't get dagger to work on NixOS that is setup with podman:

{ pkgs, ... }:
{
  virtualisation.oci-containers.backend = "podman";

  virtualisation.podman = {
    enable = true;
    dockerCompat = true;
    dockerSocket.enable = true;
    defaultNetwork.settings.dns_enabled = true;
}

I am getting this error when I run dagger for the first time:

    ╰─✘ moby.buildkit.v1.Control/Info 0.0s ERROR
      ! connection error: desc = "error reading server preface: command [docker exec -i dagger-engine-v0.18.18
        buildctl dial-stdio] has exited with exit status 1, make sure the URL is valid, and Docker 18.09 or later is
        installed on the remote host: stderr=Error: dial unix /run/dagger/engine.sock: connect: no such file or
        directory\nUsage:\n  dial-stdio [flags]\n\nFlags:\n      --addr string   The address serving the grpc api
        (default \"unix:///run/dagger/engine.sock\")\n  -h, --help          help for dial-stdio\n      --timeout int
        The timeout in seconds for connecting to the grpc api (default 5)\n\n"

Why is this important to you?

I don't want to use docker just for dagger.io sake, I am fine with podman as is.

How are you currently working around this?

I don't have a workaround, and cannot use dagger on my PC for local development.

[kpenfound]

Thanks for opening the issue! As far as I know, the dockerCompat = true; option that you have set should have been enough to make it work out of the box. Lets see if anyone else has a workaround here

[sagikazarmark]

dockerCompat simply creates an alias to the docker command so that it uses podman AFAIK. It doesn't create an alias for the docker socket. Not sure what dagger mounts as its socket.

[caniko]

These are the logs from the engine container:

#!/bin/sh
set -e

cat $0

# cgroup v2: enable nesting
# see https://github.com/moby/moby/blob/38805f20f9bcc5e87869d6c79d432b166e1c88b4/hack/dind#L28
if [ -f /sys/fs/cgroup/cgroup.controllers ]; then
        # move the processes from the root group to the /init group,
        # otherwise writing subtree_control fails with EBUSY.
        # An error during moving non-existent process (i.e., "cat") is ignored.
        mkdir -p /sys/fs/cgroup/init
        xargs -rn1 < /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs || :
        # enable controllers
        sed -e 's/ / +/g' -e 's/^/+/' < /sys/fs/cgroup/cgroup.controllers \
                > /sys/fs/cgroup/cgroup.subtree_control
fi

# expect more open files due to per-client SQLite databases
# many systems default to 1024 which is far too low
ulimit -n 1048576 || echo "cannot increase open FDs with ulimit, ignoring"

exec /usr/local/bin/dagger-engine --config /etc/dagger/engine.toml "$@"
cannot increase open FDs with ulimit, ignoring
sh: error setting limit: Operation not permitted
time="2025-09-17T13:29:09Z" level=info msg="detected mtu 65520 via interface eth0"
time="2025-09-17T13:29:09Z" level=debug msg="engine name: 0d55753734c7"
time="2025-09-17T13:29:09Z" level=debug msg="creating engine GRPC server"
time="2025-09-17T13:29:09Z" level=debug msg="creating engine lockfile"
time="2025-09-17T13:29:09Z" level=debug msg="creating engine server"
dnsmasq[37]: started, version 2.91 cachesize 150
dnsmasq[37]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset no-nftset auth no-DNSSEC loop-detect inotify dumpfile
dnsmasq[37]: warning: interface dagger0 does not currently exist
dnsmasq[37]: using only locally-known addresses for dagger.local
dnsmasq[37]: reading /etc/dnsmasq-resolv.conf
dnsmasq[37]: using nameserver 169.254.1.1#53
dnsmasq[37]: using nameserver 192.168.178.1#53
dnsmasq[37]: using nameserver fd23:a32b:ca9b:0:d624:ddff:fe14:9793#53
dnsmasq[37]: using nameserver 2001:9e8:1632:1000:d624:ddff:fe14:9793#53
dnsmasq[37]: using only locally-known addresses for dagger.local
dnsmasq[37]: bad address at /etc/hosts line 11
time="2025-09-17T13:29:09Z" level=info msg="auto snapshotter: using overlayfs"
time="2025-09-17T13:29:09Z" level=warning msg="failed to release network namespace \"h1dc3d1rqzzu00bk42wqiijvc\" left over from previous run: plugin type=\"loopback\" failed (delete): unknown FS magic on \"/var/lib/dagger/net/cni/h1dc3d1rqzzu00bk42wqiijvc\": ef53"
dagger-engine: failed to create engine: failed to create network providers: CNI setup error: plugin type="bridge" failed (add): failed to list chains: running [/usr/local/sbin/iptables -t nat -S --wait]: exit status 3: iptables v1.8.11 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

[marcosnils]

seems to be hitting this issue: https://docs.dagger.io/reference/troubleshooting/#dagger-restarts-with-a-cni-setup-error.

Not the first time that a user trips with this. I think we should rename this issue and handle this error better so we can instruct the user how to get unblocked. This applies for any case where the engine container fails to start that is.

[caniko]

Yeah, I was going to suggest the same! Feel free to adapt the title and the initial body 👍🏼

Alternatively, you can close this issue here, and create a new one.

[caniko]

This was recently replaced on nixos, don't know about the other distros:

  boot.kernelModules = ["nf_nat"];

[chrislawso]

Did you find how to get dagger + podman to work on NixOS ?

[caniko]

The issue in this thread has two parts:

1. CNI/iptables error (likely not needed on recent kernels — nf_nat is included by default, but if you hit the can't initialize iptables table 'nat' error, add this):

boot.kernelModules = [ "nf_nat" ];

2. Socket compatibility — dockerCompat = true only aliases the docker CLI command to podman. For dagger to find the container socket, you also need dockerSocket.enable = true (which the OP already has). This creates a docker-compatible socket at /run/podman/podman.sock and symlinks it. If dagger still can't find it, explicitly set:

export DOCKER_HOST=unix:///run/podman/podman.sock

So the minimal NixOS config for dagger + podman:

{
  # Only needed if you hit the iptables/CNI error — recent kernels include nf_nat by default
  # boot.kernelModules = [ "nf_nat" ];

  virtualisation.podman = {
    enable = true;
    dockerCompat = true;
    dockerSocket.enable = true;
    defaultNetwork.settings.dns_enabled = true;
  };
}