Skip to content

Fix first run of elemental failing in Macbook podman#406

Open
dbw7 wants to merge 1 commit intoSUSE:mainfrom
dbw7:mac-repart-fix
Open

Fix first run of elemental failing in Macbook podman#406
dbw7 wants to merge 1 commit intoSUSE:mainfrom
dbw7:mac-repart-fix

Conversation

@dbw7
Copy link
Copy Markdown

@dbw7 dbw7 commented Apr 1, 2026

When you run elemental3 through podman for the first time on a Macbook, the image fails with the following error:

INFO[0032] Partitioning image '/config/image-2026-03-31T14-34-44.raw' 
DEBU[0032] Running cmd: 'PATH=/sbin:/usr/sbin:/usr/bin:/bin systemd-repart --json=pretty --definitions=/tmp/elemental-repart.d146009211 --dry-run=no --empty=create --size=35840M /config/image-2026-03-31T14-34-44.raw' 
DEBU[0036] "systemd-repart" command reported an error: exit status 1 
DEBU[0036] "systemd-repart" command output:             
DEBU[0036] "systemd-repart" stderr: No machine ID set, using randomized partition UUIDs.
Sized '/config/image-2026-03-31T14-34-44.raw' to 35G.
Applying changes to /config/image-2026-03-31T14-34-44.raw.
Failed to make loopback device of future partition 0: Device or resource busy 
ERRO[0036] Customizing image media failed               
ERRO[0036] Customizing installer media failed           
DEBU[0036] Cleaning up working directory                
2026/03/31 14:35:21 failed partitioning disk '/config/image-2026-03-31T14-34-44.raw' with systemd-repart: exit status 1

If you immediately run it a second time, the image build will succeed. If you run a 3rd time, it will fail, if you run it a 4th time it will succeed...

The issue is that systemd-repart needs multiple loopback devices during the partitioning process. During the first run, it does not have the 2 device nodes that it needs, but it seems like it leaves behind some artifact that allows systemd-repart to succeed on the second run.

My proposed solution is within the runSystemdRepart function, we check if we are inside of an arm64 container. If we are, we manually create some device nodes for systemd-repart to use. This seems to work in my tests where alternating success/fails no longer happen and it succeeds each time.

@dbw7 dbw7 requested a review from a team as a code owner April 1, 2026 03:08
@dbw7 dbw7 force-pushed the mac-repart-fix branch from 2ed2fcd to 9199db4 Compare April 1, 2026 03:11
Copy link
Copy Markdown
Contributor

@atanasdinov atanasdinov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any idea if this is going to work without --privileged parameter?

// the optional given flags. On success it parses systemd-repart output to get the generated partition UUIDs and update the
// given partitions list with them.
func runSystemdRepart(s *sys.System, target string, parts []Partition, flags ...string) error {
if os.Getenv("container") != "" && runtime.GOARCH == "arm64" {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's extract this snippet in a separate function, e.g. setupLoopDevices, and put a comment above it about what it's actually doing so it's clear when someone sees this 6 months from now.

@dbw7 dbw7 force-pushed the mac-repart-fix branch 2 times, most recently from a1ca269 to ad1530b Compare April 1, 2026 14:40
@dbw7 dbw7 force-pushed the mac-repart-fix branch 2 times, most recently from ff16681 to 93d3598 Compare April 1, 2026 15:03
@dbw7
Copy link
Copy Markdown
Author

dbw7 commented Apr 2, 2026

It does not work without --privileged but I don't think this is related to the change <- This was my user error, it does in fact work without the flag

Copy link
Copy Markdown
Contributor

@davidcassany davidcassany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand it correctly I think we should not hardcode the number of loop devices, as it should be equal or higher than the partitions number.

if os.Getenv("container") != "" && runtime.GOARCH == "arm64" {
for i := 0; i < 4; i++ {
devPath := fmt.Sprintf("/dev/loop%d", i)
_ = unix.Mknod(devPath, unix.S_IFBLK|0660, 7*256+i)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, what happens if the device already exists? errors out or overwrites it? Probably it is less intrusive if we just create the missing ones.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it silently fails without any issue

// This is not an issue in amd64 containers because enough loop device nodes are automatically available.
func setupLoopDeviceNodes() {
if os.Getenv("container") != "" && runtime.GOARCH == "arm64" {
for i := 0; i < 4; i++ {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 4? To my understanding this should be as big as the length of the partitions slice.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 is an arbitrary number that just allows it to work, based on the default settings (not sure how it works with further customization) I believe it only needs 1 device node

// the optional given flags. On success it parses systemd-repart output to get the generated partition UUIDs and update the
// given partitions list with them.
func runSystemdRepart(s *sys.System, target string, parts []Partition, flags ...string) error {
setupLoopDeviceNodes()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say we can simply pass the partitions number as an argument here.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that's entirely necessary as I don't believe they are tied or linked to each other. But I will hijack this comment to outline a comprehensive view of the flow and the issue:

On a Mac, if we do podman machine ssh and then run ls -la /dev/loop* this is the output we see:

root@localhost:~# ls -la /dev/loop*
crw-rw----. 1 root disk 10, 237 Mar 31 23:28 /dev/loop-control

Now if we enter into a an elemental3 container that we built like so:

podman run -it --privileged \
  --entrypoint /bin/sh \
  -v $PWD/examples/elemental/customize/linux-only/:/config \
  -v /run/podman/podman.sock:/var/run/docker.sock \
  local/elemental-image:v3.0.0-alpha.20251212-g93d3598

So that we have shell access, and then we run ls -la /dev/loop* this is what we see:

sh-5.3# ls -la /dev/loop*
crw-rw---- 1 root 6 10, 237 Apr  8 15:01 /dev/loop-control

So by default, no loop device nodes are created inside of the podman virtual machine on Macs

so now if within that same container we run elemental3 --debug customize --type raw --local
We will fail with the known issue:

DEBU[0057] Running cmd: 'PATH=/sbin:/usr/sbin:/usr/bin:/bin systemd-repart --json=pretty --definitions=/tmp/elemental-repart.d370830029 --dry-run=no --empty=create --size=8192M /config/image-2026-04-08T15-03-28.raw' 
DEBU[0062] "systemd-repart" command reported an error: exit status 1 
DEBU[0062] "systemd-repart" command output:             
DEBU[0062] "systemd-repart" stderr: No machine ID set, using randomized partition UUIDs.
Sized '/config/image-2026-04-08T15-03-28.raw' to 8G.
Applying changes to /config/image-2026-04-08T15-03-28.raw.
Failed to make loopback device of future partition 0: Device or resource busy 

After this failure, let's check the podman vm and container states:

sh-5.3# ls -la /dev/loop*
crw-rw---- 1 root 6 10, 237 Apr  8 15:01 /dev/loop-control

root@localhost:~# ls -la /dev/loop*
crw-rw----. 1 root disk 10, 237 Mar 31 23:28 /dev/loop-control
brw-rw----. 1 root disk  7,   0 Apr  8 11:04 /dev/loop0

What's interesting here is /dev/loop0 is present within the podman vm, but not within the container. However, if we start a new container using the same command as before:

sh-5.3# ls -la /dev/loop*
crw-rw---- 1 root 6 10, 237 Apr  8 15:06 /dev/loop-control
brw-rw---- 1 root 6  7,   0 Apr  8 15:06 /dev/loop0

We see that /dev/loop0 is present in the container and elemental will succeed and the /dev/loop0 will remain on the container:

DEBU[0075] systemd-repart output to parse:
[
        {
...
        },
        {
...
        }
] 
INFO[0097] Customize complete                           
DEBU[0097] Cleaning up working directory  
sh-5.3# ls -la /dev/loop*
crw-rw---- 1 root 6 10, 237 Apr  8 15:06 /dev/loop-control
brw-rw---- 1 root 6  7,   0 Apr  8 15:09 /dev/loop0

But it is no longer present in the podman vm:

root@localhost:~# ls -la /dev/loop*
crw-rw----. 1 root disk 10, 237 Mar 31 23:28 /dev/loop-control

However, if you rerun elemental within that same container, it will continue to succeed indefinitely as, even though the loop device node is cleaned up from the podman vm, it remains on the current container

So I believe what's happening is:

  1. There are no available loop device nodes in the podman vm by default (at least I think this is intentional default behavior but i could be wrong)
  2. We run elemental in a container, and what happens is that container inherits the current state of the podman vm
  3. When elemental runs systemd-repart, the loop device node is created in the podman vm, however, the podman container does not automatically inherit this, so it fails
  4. On the second run, the new container inherits the existing loop device node and succeeds, as a result, it is cleaned up from the podman vm, so the next new container will fail as there is no loop device node to inherit

So the proposed solution bypasses the need for the loop device to already pre-exist on the podman vm when the container is first ran, and also the need for it to be repopulated each time for each new container

This is why I don't think the loop device nodes are tied to the partitions or anything specific, I chose 4 as an arbitrary number, but it could be more or less, I think even 1 should work, the only condition that I haven't checked is if there are instances where systemd-repart might need more than 1 loop device node

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I had the impression a loop device was used in each partition. I guess we can leave it as is then.

@dbw7 dbw7 force-pushed the mac-repart-fix branch from 93d3598 to 0f128e8 Compare April 8, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants