vmstate parser fails on Azure nested virtualization (Firecracker v1.12.0, AMD EPYC)

[cchen7]

Environment

• Host: Azure Standard_D8ads_v6 (AMD EPYC 9V74, 8 vCPU)
• Host OS: Ubuntu 24.04.4, Kernel 6.17.0-1008-azure
• Nested virtualization: KVM-in-Hyper-V (Azure)
• Firecracker: v1.12.0
• ZeroBoot: commit 99d86c8

Problem

zeroboot test-exec and zeroboot bench fail with:

Error: cannot detect vmstate layout: IOAPIC base address 0xFEC00000 not found

Template creation (zeroboot template) succeeds normally — the issue is in vmstate.rs::detect_offset_shift() during snapshot restore.

Root Cause

The detect_offset_shift() function assumes a single global shift between reference offsets and actual offsets in the vmstate file. In our environment, the vmstate has two different shifts:

• IOAPIC region: shift = +4 (from reference 0x0591 to actual 0x058d)
• CPU registers (LAPIC, EFER, XSAVE, etc.): shift = -764

This happens because Firecracker's versionize format has variable-length sections between IOAPIC and the CPU state block. The current code finds IOAPIC at the correct offset but then validates by checking EFER at REF_EFER - shift, which points to the wrong location (offset 0x2AF1 instead of actual 0x2DF1).

Offset analysis

Field       Reference    Actual     Shift
IOAPIC      0x0591       0x058d     +4
LAPIC       0x2541       0x283d     -764
REGS        0x2955       0x2c51     -764
EFER        0x2AF5       0x2df1     -764
XCRS        0x2B75       0x2e71     -764
XSAVE       0x2D0D       0x3009     -764

Workaround

We patched vmstate.rs with a dual-anchor approach:

1. detect_offset_shift() — accepts IOAPIC shift even when EFER validation fails
2. New detect_cpu_shift() — independently locates EFER by searching for value 0xD01/0x501 and validating CR0 at a known relative offset (-40 bytes, with PG+PE bits set)
3. parse_vmstate() — uses ioapic_shift for IOAPIC redirect table and cpu_shift for all CPU register fields

After patching, fork time is 0.39ms p50 (1000 iterations) on Azure nested virtualization.

Suggested Fix

The detect_offset_shift() function should either:

1. Use multiple anchor points (IOAPIC + EFER independently) instead of assuming a single global shift, or
2. Search for each field type independently using known value patterns (IOAPIC base 0xFEC00000, EFER 0xD01/0x501, LAPIC version 0x50014, XSAVE FCW 0x037F)

Happy to submit a PR if you'd prefer.

[adammiribyan]

Yes, please!

[congwang-mk]

For anyone landing here because they need sandboxing on Azure (or other environments where /dev/kvm isn't available or nested virt is unreliable):

sandlock takes a non-VM approach to sandboxing — it uses Linux Landlock LSM + seccomp user notification instead of Firecracker/KVM. That means:

• Runs on any Linux 5.13+ kernel, no /dev/kvm required
• Works on Azure, ARM, burstable instances, and inside containers
• ~7ms startup overhead

It won't give you the same hardware-level isolation boundary as a microVM, but if your goal is sandboxing untrusted code or AI agent workloads, it covers network isolation, syscall filtering, resource limits (memory/CPU/processes), and filesystem COW out of the box.