[SOLVED] Filesystem corruption after PCIe passthrough

[m0nsieurPsych0]

Hello,

I have a weird issue since on proxmox 7.0-14+1 with pcie passthrough that I did not have with proxmox 6.4.
Right now on proxmox 7.0-14+, I am able to pass through a Nvidia GPU without any issues, but when I pass a complete USB hub to the windows VM via pcie passthough I get data corruption and the system becomes read-only giving this error message:

Buffer I/O error on device dm-5, logical block 5277386
Aborting journal on device dm-5-8.
Buffer I/O error on device dm-5, logical block 3178496, lost sync page write
JBD2: Error -5 detected when updating journal superblock for dm-5-8.
EXT4-fs error (device dm-5): ext4_journal_check_start:83: comm pvesh: Detected aborted journal
EXT4-fs error (device dm-5): ext4_journal_check_start:83: comm rs:main Q:Reg: Detected aborted journal
Buffer I/O error on device dm-5, logical block 0, lost sync page write
EXT4-fs (dm-5): I/O error while writing superblock
EXT4-fs (dm-5): Previous I/O error to superblock detected
EXT4-fs (dm-5): Remounting Filesystem read-only
Buffer I/O error on device dm-5, logical block 0, lost sync page write

After that the console on the host is spammed with journalctl writting errors.

So here is the chain of events:

1- Vm is off I pass the USB pcie root
2- Turn vm on
3- Host System immediately spam the previous error and everything is dead
4- Hard reboot
5- Boot up and see proxmox login on host
6- Wait 10 seconds
7- Same error and journalctl message spam

The system is now corrupted and no amount of reboot and fsck does anything. I installed the system on two different ssd (different model) and had the same result everytime I tried the pcie passthrough.
Now at this point, I can boot without any issue in rescue mode, but I have no idea what to do to repair what's broken. I tried fsck on /dev/pve/root, but the system complains that the partition is mounted, so I booted a archlinux installation disk and ran it there. No error was found.

I have to admit that I'm a little bit confuse by the LVM setup so I might just be doing things wrong.

Last time, to «resolve» the issue, I just reinstalled the system, but this is hardly a solution.

So I humbly request the forums helps, if anyone has any idea it would be greatly appreciated.

Thanks!

Karl

 

[leesteken]

Sounds like the USB controller you want to passthrough to the VM is in the same IOMMU group as (one of) the drive controller(s) that are in use by the Proxmox host. You cannot shared devices in the same IOMMU group between VMs and/or the host. Proxmox therefore loses the drives when you start the VM.
However, I have not heard before that this breaks Proxmox permanently with filesystem corruption. Usually a reboot (and not starting the VM) fixes it. Also, IOMMU groups are not expected to change between PVE 6.4 and 7.0, as they are determined by the motherboard and BIOS. Passthrough of the USB controller worked fine on PVE 6.4?
If you do a reinstall (again), check your IOMMU groups before doing passthrough with this command: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done. Make sure to do this without using pcie_acs_override.

 

[m0nsieurPsych0]

Just ran the listing of the iommu devices on a fresh install and the device I'm trying to passthrough is :
IOMMU group 14 08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]

We can see that it's in the same group as a lot of other devices like the Sata controller that the os drive is installed on. So that might be why the data gets corrupted.

I will try to add some OS modification to split the iommu groups and come back with the results.

root@proxmox1:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 10 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU group 11 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU group 11 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 12 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU group 12 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU group 12 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU group 12 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU group 12 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU group 12 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU group 12 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
IOMMU group 12 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU group 13 00:19.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU group 13 00:19.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU group 13 00:19.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU group 13 00:19.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU group 13 00:19.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU group 13 00:19.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU group 13 00:19.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
IOMMU group 13 00:19.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU group 14 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller [1022:43ba] (rev 02)
IOMMU group 14 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller [1022:43b6] (rev 02)
IOMMU group 14 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge [1022:43b1] (rev 02)
IOMMU group 14 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 14 02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 14 02:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 14 02:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 14 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 14 02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 14 05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU group 14 07:00.0 Ethernet controller [0200]: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) [8086:105e] (rev 06)
IOMMU group 14 07:00.1 Ethernet controller [0200]: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) [8086:105e] (rev 06)
IOMMU group 14 08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]
IOMMU group 15 09:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03)
IOMMU group 16 0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1)
IOMMU group 16 0a:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU group 17 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU group 18 0b:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU group 19 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU group 20 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
IOMMU group 21 0c:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 22 0c:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
IOMMU group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]

 

[leesteken]

You can ignore bridges, but group 14 does indeed look like the "Ryzen motherboard chipset" group with all the devices that are on PCIe lanes not from the CPU. Proxmox has the pcie_acs_override patches included, but that breaks the VM/PCI isolation.

Have you tried the USB controller(0b:00.3 in group 19? That should pass through to a VM fine, in my experience.

I got the impression from your first post that this USB controller passthrough did work in PVE 6.4. Can you please confirm this? Or did I not understand correctly?

 

[m0nsieurPsych0]

I got the impression from your first post that this USB controller passthrough did work in PVE 6.4. Can you please confirm this? Or did I not understand correctly?

Yes that is correct. The thing is I installed and did modification to iommu groups more than a 1.5 year ago on proxmox 6.4 and thus don't remember exactly what I did. So I probably forgot a setting for this board when reinstalling proxmox 7. What I do remember is the OS not being corrupted after some tests.

Right now, I am following the doc here and guide there and will also try to passthrough the other USB port you suggested.

I will come back with the results.

 

[m0nsieurPsych0]

So, turns out I was using the option pcie_acs_override=downstream,multifunction in the previous config which makes the iommu group for the pcie device 08:00.0 be in a different group.

Could you expand on the potential problem with VM/PCI isolation breaking using this option ?

root@proxmox1:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 10 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU group 11 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU group 11 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 12 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU group 12 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU group 12 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU group 12 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU group 12 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU group 12 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU group 12 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
IOMMU group 12 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU group 13 00:19.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU group 13 00:19.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU group 13 00:19.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU group 13 00:19.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU group 13 00:19.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU group 13 00:19.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU group 13 00:19.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
IOMMU group 13 00:19.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU group 14 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller [1022:43ba] (rev 02)
IOMMU group 15 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller [1022:43b6] (rev 02)
IOMMU group 16 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge [1022:43b1] (rev 02)
IOMMU group 17 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 18 02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 19 02:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU group 20 02:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 21 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 22 02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 23 05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU group 24 07:00.0 Ethernet controller [0200]: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) [8086:105e] (rev 06)
IOMMU group 25 07:00.1 Ethernet controller [0200]: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) [8086:105e] (rev 06)
IOMMU group 26 08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]
IOMMU group 27 09:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03)
IOMMU group 28 0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1)
IOMMU group 29 0a:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU group 30 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU group 31 0b:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU group 32 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU group 33 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
IOMMU group 34 0c:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 35 0c:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
IOMMU group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]

 

[leesteken]

Could you expand on the potential problem with VM/PCI isolation breaking using this option ?

There's this and that. PCI(e) devices can do direct memory access (DMA) from one device to another and the host. The IOMMU keeps them in check, but only if the devices cannot access each other while bypassing Access Control Services (ACS). This information comes from the IOMMU groups (motherboard and BIOS).
If you override ACS, which breaks the groups, devices in separate groups, which then can be connected to different VM and/or the host, can communicate directly. This breaks isolation, allowing for information to be passed between VMs and possibly the host and is therefore a security issue. Whether this is a problem for your intended use, depends on you and your use case.

 

[m0nsieurPsych0]

Alright, so the file system is not corrupted. After some tests I understand better what happened.

When I first encountered the problem at boot, I deleted the VM disk file in rescue mode thinking it would prevent the vm from autostarting, but it did not. Although I'm not sure why, without further tests, I think the reason it still tried to start is because of the conf file of the vm situated in /etc/pve/qemu-server/.
In that conf file the flag « onboot: 1 » was still present and I think it tried to start the vm at boot even without a vm disk file consequently disconnecting the pcie device from the host. This theory would explain why it would boot fine, but crash again after about 10 seconds on the login prompt. I'm am not sure if this is a bug, further testing is necessary.

The problem is that this folder /dev/pve/ is empty when booting in rescue mode. From another post, I found out that the directory is populated with data from an sqlite database at /var/lib/pve-cluster/config.db.

Now the data we want is in a column of BLOB type called 'data'. The problem with BLOB type is that we can't modify them in place (at least I did not find a way with sqlite). So I have successfully created a little bash procedure to modify it:

# We set a variable for the vmid
vmid='111';

# Go into the directory /var/lib/pve-cluster/
cd /var/lib/pve-cluster/

# To list the content of the vm from the database
echo "SELECT data FROM tree WHERE name LIKE '$vmid.conf';" | sqlite3 -line config.db

# Then backup the config.db file  
cp config.db config.db.bak

# Modify config.db
#### you NEED to write everything exactly spaces and all ####

# We get the contents of the data from the tree table and remove the 'onboot: 1' flag and put it in the variable $d
d=$(echo "SELECT data FROM tree WHERE name LIKE '$vmid.conf';" | sqlite3 -line config.db | sed 's/ data = //' | grep -v 'onboot: 1');

# We update the content of the column data from the tree table
echo "UPDATE tree SET data = '$d' WHERE name LIKE '$vmid.conf';" | sqlite3 -line config.db

# Explanation of the command :
# The `sed 's/ data = //'` command remove the text ' data = ' from the output when we query the database.
# The `grep -v 'onboot: 1'` command output everything that is NOT 'onboot: 1'

#### IF IN A CLUSTER ####
1- Also modify the /etc/pve/nodes/<hostname>/qemu-server/<vmid>.conf on one other online node in the cluster and delete the line "onboot: 1"

2- Check if the data is modified on the config.db of the same online node from step 1 and all other online nodes as well as the config.db file.

3- If the data is present ANYWHERE on other nodes, when you boot up the host it will start the VM.

 

[leesteken]

Yes, the configuration (in /etc/pve but stored in a database) makes the VM start at boot and initiate PCI passthrough.
It might be easier to press e in the boot menu and change the kernel command line. Use amd_iommu=off to disable PCI passthrough but still boot normally. Proxmox should then fail to start the VM and also refuse to do passthrough.

 

[m0nsieurPsych0]

Well, this sound a lot easier!

One thing I still wonder though, is if we deleted the vm disk for a vm in rescue mode why does it try to passthrough the pcie device anyway when attempting to boot the vm?

I mean, there is certainly an order in which a vm verify if it can boot or not, no ? Shouldn't it check if a vm disk is present BEFORE trying to passthrough a pcie device ? I'm probably oversimplifying the process here, but I guess some kind of race condition is happening.

 

[neail]

This problem still exists in PVE8.


I have an intel machine (old — DH61ww). Which has 4 SATA ports, and all are controlled by the chipset (H61) and that is a PCIe interface.
— I accidentally passed the PCIE interface of the SATA channel of the H61 to a VM.
I was running proxmox from a drive which was 1 of the 4 SATA ports. Next Time I rebooted Proxmox, it didn't come online in the web GUI, and the HDD activity light was not blinking.
After a little troubleshoot on the networking side, I connected the PC to a display and finally found the issue,

The post by leesteken describes what to do, but I will give a step-by-step guide.

This can happen for an onboard network device as well, hopefully the solution will be the same.


Here is a guide what to do -

1. If the HDD light is not up, it is necessary to connect a display to further investigate, unless You have a KVM switch (KVM = Keyboard Video Mouse).
2. First boot after connecting display - make sure your command line giving something similar - remounting file system read-only

sorry for the bad image, it has aged. ​

​

3. Shut down the machine, removing the power, (also to be safe only keep the boot drive connected) and attach a Keyboard, startup the machine
4. When the computer is booting up and the blue screen appears, press E key from the Attached keyboard in the machine.
5. You should see the following screen.

6. Now press F2 to enter the Grub command line (thats a black screen. with grub >)
7. Inside the grub command line

- AMD users enter — amd_iommu=off​

- Intel useres enter - intel_iommu=off​

Press Enter.​

8. Now, the grub command line should return grub > on successful entry.
9. Type Exit >> Press F10 to exit to boot menu
10. Select the First option - of general startup.
11. The Proxmox should Automatically start naturally.
12. Don't forget to remove the PCIE device from the VM or CT.
13. Reboot proxmox to check.

Don't worry about the immou off, when you will restart it will be discarded and default startup will be performed. Unless you add it to the grub config file, it will be temporary for only that boot instance.

​