[SOLVED] Latest update (VE 7.3-3, 5.15.74-1-pve) broke my VMs

M3M1X

New Member
Jan 16, 2022
9
1
3
43
I have 2 VMs with PCI Passthrough and were working fine before the update. Now they won't start and I get the following error:
TASK ERROR: cannot prepare PCI pass-through, IOMMU not present

When I start the VM from command line, I get the following message:
cannot prepare PCI pass-through, IOMMU not present

My /etc/default/grub is:
Code:
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=efifb:eek:ff"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Initially, I had line 9 uncommented but reading some threads, people seem to suggest I should only use intel_iommu=on.

The output of dmesg | grep -i IOMMU is:
Code:
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
[    0.059894] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
[    0.059939] DMAR: IOMMU enabled
[    0.335104] iommu: Default domain type: Passthrough (set via kernel command line)

And here is one of the VM's conf file:
Code:
agent: 1
bios: ovmf
bootdisk: scsi0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: SSD:vm-102-disk-1,size=4M
hostpci0: 0000:01:00.0,pcie=1,x-vga=1
ide0: HDD:iso/virtio-win.iso,media=cdrom,size=402812K
ide2: none,media=cdrom
machine: pc-q35-5.1
memory: 18432
name: VM1
net0: virtio=0A:02:D6:F9:4D:2E,bridge=vmbr0,firewall=1,rate=2
numa: 0
onboot: 1
ostype: win10
scsi0: SSD:vm-102-disk-0,cache=writeback,size=500G
scsihw: virtio-scsi-pci
smbios1: uuid=3de8343d-a4a1-4ddc-8c50-6275fccae5a7
sockets: 1
startup: order=2
usb0: host=1-5,usb3=1
usb1: host=1-6,usb3=1
usb2: host=1-10
usb3: host=1-4,usb3=1
vmgenid: 28c3d350-fed9-4b81-944e-415d7d4a7fa2

Lastly, I checked my BIOS and the VT-d is enabled.

Any help is highly appreciated! And please let me know if you need any other details. Thanks!
 
Oh, and when I run this command find /sys/kernel/iommu_groups/ -type l, I get nothing.
 
Double check VT-d is enabled, because it can be reset by low battery or BIOS update. Check the output of cat /proc/cmdline if intel_iommu=on is present. Check which boot loader is used, because it might be that you changes had no effect: the initial 5.15 kernel had intel_iommu enabled by default but later versions reverted this.
 
  • Like
Reactions: M3M1X
Double check VT-d is enabled, because it can be reset by low battery or BIOS update. Check the output of cat /proc/cmdline if intel_iommu=on is present. Check which boot loader is used, because it might be that you changes had no effect: the initial 5.15 kernel had intel_iommu enabled by default but later versions reverted this.
Thank you leesteken.

VT-d is for sure enabled.

The output of cat /proc/cmdline is: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt

And the output of efibootmgr -v is:
Code:
BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0000,000B,000C,000D,0009,000A,0001,0002,0003
Boot0000* proxmox       HD(2,GPT,d2151dc7-636d-4ee6-9a56-15c1e9cc62a6,0x800,0x100000)/File(\EFI\PROXMOX\GRUBX64.EFI)
Boot0001* UEFI:CD/DVD Drive     BBS(129,,0x0)
Boot0002* UEFI:Removable Device BBS(130,,0x0)
Boot0003* UEFI:Network Device   BBS(131,,0x0)
Boot0009* WDC WD10EZEX-00BN5A0  BBS(HD,,0x0)..BO
Boot000A* SanDisk Ultra II 240GB        BBS(HD,,0x0)..BO
Boot000B* VK0480GDJXV   BBS(HD,,0x0)..BO
Boot000C* VK0480GDJXV   BBS(HD,,0x0)..BO
Boot000D* VK0480GDJXV   BBS(HD,,0x0)..BO

So, I am using grub. I enabled iommu in grup and performed update-grub
 
The last time the VMs worked was on the 5th of December. Here is a snippet from /var/log/syslog:
Code:
Dec  5 17:14:54 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.64-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  5 17:14:54 pve kernel: [    0.059894] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.64-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  5 17:14:54 pve kernel: [    0.324599] iommu: Default domain type: Translated
Dec  5 17:14:54 pve kernel: [    0.324599] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  5 17:14:54 pve kernel: [    0.375228] pci 0000:00:00.0: Adding to iommu group 0
Dec  5 17:14:54 pve kernel: [    0.375236] pci 0000:00:01.0: Adding to iommu group 1
Dec  5 17:14:54 pve kernel: [    0.375243] pci 0000:00:01.1: Adding to iommu group 2
Dec  5 17:14:54 pve kernel: [    0.375251] pci 0000:00:02.0: Adding to iommu group 3
Dec  5 17:14:54 pve kernel: [    0.375258] pci 0000:00:14.0: Adding to iommu group 4
Dec  5 17:14:54 pve kernel: [    0.375267] pci 0000:00:16.0: Adding to iommu group 5
Dec  5 17:14:54 pve kernel: [    0.375273] pci 0000:00:17.0: Adding to iommu group 6
Dec  5 17:14:54 pve kernel: [    0.375280] pci 0000:00:1b.0: Adding to iommu group 7
Dec  5 17:14:54 pve kernel: [    0.375288] pci 0000:00:1b.3: Adding to iommu group 8
Dec  5 17:14:54 pve kernel: [    0.375295] pci 0000:00:1c.0: Adding to iommu group 9
Dec  5 17:14:54 pve kernel: [    0.375303] pci 0000:00:1d.0: Adding to iommu group 10
Dec  5 17:14:54 pve kernel: [    0.375317] pci 0000:00:1f.0: Adding to iommu group 11
Dec  5 17:14:54 pve kernel: [    0.375322] pci 0000:00:1f.2: Adding to iommu group 11
Dec  5 17:14:54 pve kernel: [    0.375330] pci 0000:00:1f.3: Adding to iommu group 11
Dec  5 17:14:54 pve kernel: [    0.375336] pci 0000:00:1f.4: Adding to iommu group 11
Dec  5 17:14:54 pve kernel: [    0.375343] pci 0000:00:1f.6: Adding to iommu group 12
Dec  5 17:14:54 pve kernel: [    0.375351] pci 0000:01:00.0: Adding to iommu group 13
Dec  5 17:14:54 pve kernel: [    0.375358] pci 0000:01:00.1: Adding to iommu group 14
Dec  5 17:14:54 pve kernel: [    0.375366] pci 0000:02:00.0: Adding to iommu group 15
Dec  5 17:14:54 pve kernel: [    0.375373] pci 0000:02:00.1: Adding to iommu group 16
Dec  5 17:14:54 pve kernel: [    0.375379] pci 0000:04:00.0: Adding to iommu group 17
Dec  5 17:14:54 pve kernel: [    0.375386] pci 0000:05:00.0: Adding to iommu group 18
Dec  6 10:59:10 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.64-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  6 10:59:10 pve kernel: [    0.060461] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.64-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  6 10:59:10 pve kernel: [    0.339720] iommu: Default domain type: Translated
Dec  6 10:59:10 pve kernel: [    0.339720] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  6 18:11:34 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  6 18:11:34 pve kernel: [    0.059863] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  6 18:11:34 pve kernel: [    0.339355] iommu: Default domain type: Translated
Dec  6 18:11:34 pve kernel: [    0.339355] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  7 13:58:40 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  7 13:58:40 pve kernel: [    0.059886] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  7 13:58:40 pve kernel: [    0.339491] iommu: Default domain type: Translated
Dec  7 13:58:40 pve kernel: [    0.339491] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  7 14:01:15 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  7 14:01:15 pve kernel: [    0.059874] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  7 14:01:15 pve kernel: [    0.339097] iommu: Default domain type: Translated
Dec  7 14:01:15 pve kernel: [    0.339097] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  7 14:07:21 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  7 14:07:21 pve kernel: [    0.059719] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  7 14:07:21 pve kernel: [    0.335163] iommu: Default domain type: Translated
Dec  7 14:07:21 pve kernel: [    0.335163] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  7 14:12:56 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  7 14:12:56 pve kernel: [    0.060040] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  7 14:12:56 pve kernel: [    0.339517] iommu: Default domain type: Translated
Dec  7 14:12:56 pve kernel: [    0.339517] iommu: DMA domain TLB invalidation policy: lazy mode

So, it seems to have added the groups fine on the 5th when the kernel was vmlinuz-5.15.64-1-pve, but didn't do it on the 6th & 7th when the kernel was updated to vmlinuz-5.15.74-1-pve
 
OMG! This is embarrassing.. It turned out that VT-d was disabled afterall. In my BIOS I have 2 settings, VT-d and Intel Virtualization. I enabled them both, and now the VMs start.
 
  • Like
Reactions: leesteken
Don't worry this is more often the case than you would expect. Thank's for you report on the solutions. Did you find out why he VT-d and VT-x settings got disabled around the time of updating Proxmox?
 
  • Like
Reactions: M3M1X
Don't worry this is more often the case than you would expect. Thank's for you report on the solutions. Did you find out why he VT-d and VT-x settings got disabled around the time of updating Proxmox?
I have a feeling it was due to a power outage and the BIOS went to default settings... I know it shouldn't do that, but perhaps it is time to replace the CMOS battery!

Now I have to retrace my steps and see what changes I made to the system as one of the VMs is powering up but the screen is black and after a few minutes it shuts down on its own. Should be interesting to troubleshoot :D. Thanks leesteken, your input is very valuable and most appreciated!