Docs: RHEL 9 LVM Disk Visibility Fix After VM Clone or Snapshot Restore

[sethesterline]

Summary

Add a knowledge base article covering the RHEL 9 LVM devices file behavior that causes volume groups to be invisible after a VM is cloned or restored from a snapshot in VergeOS, and the steps required to fix it.

Type

Troubleshooting / How-to Guide

Background

RHEL 9 introduced a new LVM feature: the LVM devices file (/etc/lvm/devices/system.devices). Rather than scanning all block devices for physical volumes, RHEL 9 LVM tracks devices by their hardware/serial ID in this file. When a VM is cloned or restored from a snapshot in VergeOS, the restored VM's virtual disk receives a new device ID — and LVM's devices file no longer matches, causing vgchange -ay and lvdisplay to find nothing.

RHEL 8 and earlier do not use this feature, which is why restores/clones work fine on those versions.

Confirmed via support case with NASA/JPL (Case 00012019). Customer confirmed the fix resolves the issue on RHEL 9.7 (Plow).

Suggested Content

• Audience: VergeOS administrators running RHEL 9 VMs
• Prerequisites: Familiarity with RHEL rescue mode; access to VergeOS VM console

Key sections to cover:

Symptoms

• After restoring a VM from a snapshot, or cloning a VM, the RHEL 9 VM fails to boot (drops to emergency mode or hangs at boot)
• In rescue mode, vgchange -ay produces no output and lvdisplay shows nothing
• RHEL 8 VMs on the same system restore/clone without issue

Root Cause

• RHEL 9 uses /etc/lvm/devices/system.devices to track block devices by hardware ID
• VergeOS snapshot restore and clone operations assign new virtual disk IDs
• LVM's devices file no longer matches the restored disk's ID, so volume groups are not found

Fix Procedure (in rescue mode after restore/clone)

# 1. Confirm the disk is visible
lsblk
# Note the device name (e.g., /dev/sda) and the LVM partition (e.g., /dev/sda3)

# 2. Force LVM to re-scan all block devices (bypasses the devices file)
pvscan --cache
vgchange -ay

# 3. If step 2 still shows nothing, register the LVM partition explicitly
lvmdevices --adddev /dev/sda3   # replace with your actual LVM partition from lsblk
vgchange -ay

# 4. Verify volume groups are now visible
lvdisplay

# 5. Proceed with normal mount / chroot / dracut steps as documented in the import guide

Ongoing Consideration

• This fix must be re-applied each time the VM is restored from a snapshot or cloned, until LVM's devices file is updated inside the guest
• Optional long-term mitigation: inside the running RHEL 9 VM, run lvmdevices --update after any restore to refresh the devices file, or configure LVM to use global device scanning (use_devicesfile = 0 in /etc/lvm/lvm.conf)

Related Docs to Update

• How to Import a Linux VM (RHEL, CentOS, SUSE, Debian, Ubuntu) — Step 3.1 (lvdisplay) should add a note that RHEL 9 requires pvscan --cache and possibly lvmdevices --adddev before LVM volumes become visible
• VM Snapshots and Restores — Consider adding a note under "Restore to a Clone" and "Restore over Source" pointing to this article for RHEL 9 guests

Context

Confirmed via customer support case (NASA/JPL, Case 00012019). Customer validated the fix resolves the issue. Seth Esterline committed to logging this documentation need in his reply to the customer.