[SOLVED] Latest update (VE 7.3-3, 5.15.74-1-pve) broke my VMs

M3M1X

New Member
Jan 16, 2022
9
1
3
42
I have 2 VMs with PCI Passthrough and were working fine before the update. Now they won't start and I get the following error:
TASK ERROR: cannot prepare PCI pass-through, IOMMU not present

When I start the VM from command line, I get the following message:
cannot prepare PCI pass-through, IOMMU not present

My /etc/default/grub is:
Code:
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=efifb:eek:ff"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Initially, I had line 9 uncommented but reading some threads, people seem to suggest I should only use intel_iommu=on.

The output of dmesg | grep -i IOMMU is:
Code:
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
[    0.059894] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
[    0.059939] DMAR: IOMMU enabled
[    0.335104] iommu: Default domain type: Passthrough (set via kernel command line)

And here is one of the VM's conf file:
Code:
agent: 1
bios: ovmf
bootdisk: scsi0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: SSD:vm-102-disk-1,size=4M
hostpci0: 0000:01:00.0,pcie=1,x-vga=1
ide0: HDD:iso/virtio-win.iso,media=cdrom,size=402812K
ide2: none,media=cdrom
machine: pc-q35-5.1
memory: 18432
name: VM1
net0: virtio=0A:02:D6:F9:4D:2E,bridge=vmbr0,firewall=1,rate=2
numa: 0
onboot: 1
ostype: win10
scsi0: SSD:vm-102-disk-0,cache=writeback,size=500G
scsihw: virtio-scsi-pci
smbios1: uuid=3de8343d-a4a1-4ddc-8c50-6275fccae5a7
sockets: 1
startup: order=2
usb0: host=1-5,usb3=1
usb1: host=1-6,usb3=1
usb2: host=1-10
usb3: host=1-4,usb3=1
vmgenid: 28c3d350-fed9-4b81-944e-415d7d4a7fa2

Lastly, I checked my BIOS and the VT-d is enabled.

Any help is highly appreciated! And please let me know if you need any other details. Thanks!
 
Oh, and when I run this command find /sys/kernel/iommu_groups/ -type l, I get nothing.
 
Double check VT-d is enabled, because it can be reset by low battery or BIOS update. Check the output of cat /proc/cmdline if intel_iommu=on is present. Check which boot loader is used, because it might be that you changes had no effect: the initial 5.15 kernel had intel_iommu enabled by default but later versions reverted this.
 
  • Like
Reactions: M3M1X
Double check VT-d is enabled, because it can be reset by low battery or BIOS update. Check the output of cat /proc/cmdline if intel_iommu=on is present. Check which boot loader is used, because it might be that you changes had no effect: the initial 5.15 kernel had intel_iommu enabled by default but later versions reverted this.
Thank you leesteken.

VT-d is for sure enabled.

The output of cat /proc/cmdline is: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt

And the output of efibootmgr -v is:
Code:
BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0000,000B,000C,000D,0009,000A,0001,0002,0003
Boot0000* proxmox       HD(2,GPT,d2151dc7-636d-4ee6-9a56-15c1e9cc62a6,0x800,0x100000)/File(\EFI\PROXMOX\GRUBX64.EFI)
Boot0001* UEFI:CD/DVD Drive     BBS(129,,0x0)
Boot0002* UEFI:Removable Device BBS(130,,0x0)
Boot0003* UEFI:Network Device   BBS(131,,0x0)
Boot0009* WDC WD10EZEX-00BN5A0  BBS(HD,,0x0)..BO
Boot000A* SanDisk Ultra II 240GB        BBS(HD,,0x0)..BO
Boot000B* VK0480GDJXV   BBS(HD,,0x0)..BO
Boot000C* VK0480GDJXV   BBS(HD,,0x0)..BO
Boot000D* VK0480GDJXV   BBS(HD,,0x0)..BO

So, I am using grub. I enabled iommu in grup and performed update-grub
 
The last time the VMs worked was on the 5th of December. Here is a snippet from /var/log/syslog:
Code:
Dec  5 17:14:54 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.64-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  5 17:14:54 pve kernel: [    0.059894] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.64-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  5 17:14:54 pve kernel: [    0.324599] iommu: Default domain type: Translated
Dec  5 17:14:54 pve kernel: [    0.324599] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  5 17:14:54 pve kernel: [    0.375228] pci 0000:00:00.0: Adding to iommu group 0
Dec  5 17:14:54 pve kernel: [    0.375236] pci 0000:00:01.0: Adding to iommu group 1
Dec  5 17:14:54 pve kernel: [    0.375243] pci 0000:00:01.1: Adding to iommu group 2
Dec  5 17:14:54 pve kernel: [    0.375251] pci 0000:00:02.0: Adding to iommu group 3
Dec  5 17:14:54 pve kernel: [    0.375258] pci 0000:00:14.0: Adding to iommu group 4
Dec  5 17:14:54 pve kernel: [    0.375267] pci 0000:00:16.0: Adding to iommu group 5
Dec  5 17:14:54 pve kernel: [    0.375273] pci 0000:00:17.0: Adding to iommu group 6
Dec  5 17:14:54 pve kernel: [    0.375280] pci 0000:00:1b.0: Adding to iommu group 7
Dec  5 17:14:54 pve kernel: [    0.375288] pci 0000:00:1b.3: Adding to iommu group 8
Dec  5 17:14:54 pve kernel: [    0.375295] pci 0000:00:1c.0: Adding to iommu group 9
Dec  5 17:14:54 pve kernel: [    0.375303] pci 0000:00:1d.0: Adding to iommu group 10
Dec  5 17:14:54 pve kernel: [    0.375317] pci 0000:00:1f.0: Adding to iommu group 11
Dec  5 17:14:54 pve kernel: [    0.375322] pci 0000:00:1f.2: Adding to iommu group 11
Dec  5 17:14:54 pve kernel: [    0.375330] pci 0000:00:1f.3: Adding to iommu group 11
Dec  5 17:14:54 pve kernel: [    0.375336] pci 0000:00:1f.4: Adding to iommu group 11
Dec  5 17:14:54 pve kernel: [    0.375343] pci 0000:00:1f.6: Adding to iommu group 12
Dec  5 17:14:54 pve kernel: [    0.375351] pci 0000:01:00.0: Adding to iommu group 13
Dec  5 17:14:54 pve kernel: [    0.375358] pci 0000:01:00.1: Adding to iommu group 14
Dec  5 17:14:54 pve kernel: [    0.375366] pci 0000:02:00.0: Adding to iommu group 15
Dec  5 17:14:54 pve kernel: [    0.375373] pci 0000:02:00.1: Adding to iommu group 16
Dec  5 17:14:54 pve kernel: [    0.375379] pci 0000:04:00.0: Adding to iommu group 17
Dec  5 17:14:54 pve kernel: [    0.375386] pci 0000:05:00.0: Adding to iommu group 18
Dec  6 10:59:10 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.64-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  6 10:59:10 pve kernel: [    0.060461] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.64-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  6 10:59:10 pve kernel: [    0.339720] iommu: Default domain type: Translated
Dec  6 10:59:10 pve kernel: [    0.339720] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  6 18:11:34 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  6 18:11:34 pve kernel: [    0.059863] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  6 18:11:34 pve kernel: [    0.339355] iommu: Default domain type: Translated
Dec  6 18:11:34 pve kernel: [    0.339355] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  7 13:58:40 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  7 13:58:40 pve kernel: [    0.059886] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  7 13:58:40 pve kernel: [    0.339491] iommu: Default domain type: Translated
Dec  7 13:58:40 pve kernel: [    0.339491] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  7 14:01:15 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  7 14:01:15 pve kernel: [    0.059874] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  7 14:01:15 pve kernel: [    0.339097] iommu: Default domain type: Translated
Dec  7 14:01:15 pve kernel: [    0.339097] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  7 14:07:21 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  7 14:07:21 pve kernel: [    0.059719] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  7 14:07:21 pve kernel: [    0.335163] iommu: Default domain type: Translated
Dec  7 14:07:21 pve kernel: [    0.335163] iommu: DMA domain TLB invalidation policy: lazy mode
Dec  7 14:12:56 pve kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction v
ideo=efifb:eek:ff
Dec  7 14:12:56 pve kernel: [    0.060040] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifun
ction video=efifb:eek:ff
Dec  7 14:12:56 pve kernel: [    0.339517] iommu: Default domain type: Translated
Dec  7 14:12:56 pve kernel: [    0.339517] iommu: DMA domain TLB invalidation policy: lazy mode

So, it seems to have added the groups fine on the 5th when the kernel was vmlinuz-5.15.64-1-pve, but didn't do it on the 6th & 7th when the kernel was updated to vmlinuz-5.15.74-1-pve
 
OMG! This is embarrassing.. It turned out that VT-d was disabled afterall. In my BIOS I have 2 settings, VT-d and Intel Virtualization. I enabled them both, and now the VMs start.
 
  • Like
Reactions: leesteken
Don't worry this is more often the case than you would expect. Thank's for you report on the solutions. Did you find out why he VT-d and VT-x settings got disabled around the time of updating Proxmox?
 
  • Like
Reactions: M3M1X
Don't worry this is more often the case than you would expect. Thank's for you report on the solutions. Did you find out why he VT-d and VT-x settings got disabled around the time of updating Proxmox?
I have a feeling it was due to a power outage and the BIOS went to default settings... I know it shouldn't do that, but perhaps it is time to replace the CMOS battery!

Now I have to retrace my steps and see what changes I made to the system as one of the VMs is powering up but the screen is black and after a few minutes it shuts down on its own. Should be interesting to troubleshoot :D. Thanks leesteken, your input is very valuable and most appreciated!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!