DMAR: IOMMU enabled / Error: cannot prepare PCI pass-through, IOMMU not present

[cik28]

Hi, i am trying to understund why the system behave this way to find out where the issue lays, system specifications and and steps taken in attempt to fix it bellow:

CPU: G4400T / I5 6500 (tried both)
RAM: 16GB (2x 8GB)
Motherboard: ASRock H110M-DGS R3.0 (BIOS: P7.4)
NetworkPVE: RTL8111 GbE (enp3s0 bridge vmbr0)
NetworkNAS: RTL8125 2.5GbE (enp2s0) <== connected via PCI-e x1 NIC
Boot SSD: 2x 128GB ZFS (local(ISOs), local-ZFS(TrueNAS Scale_VM...)) <== connected to PCH
VM HDD: 750GB ext4 2.5" (vm750GB-HDD(Other_VMs...)) <== connected to PCH
NAS HDD: 2x 4TB <== connected via ASMEDIA PCI-e x1 daughter board

I firstly went to ASRock webpage (https://www.asrock.com/mb/Intel/H110M-DGS R3.0/index.fr.asp#Memory) to gather information on the BIOS on this motherboard, it seems the P7.4 version is not listed and i have no clue why this motherboard is not able to flash the BIOS from a USB device (or a SATA device) with the Instant Flash BIOS tool, using DOS didn't rly help.

I went with the version i had installed since i do not have the tool to flash it externaly, looking at the BIOS layout and the user guide (https://download.asrock.com/Manual/H110M-DGS R3.0.pdf), i enabled the settings for proper Virtualization such as, VT-d (for device I/O isolation), Intel Virtualization, HW Prefetch, Adjacent Cache Line Prefetch, PCI-e ASPM, PCH PCI-e ASPM, DMI ASPM, PCH DMI ASPM, IOAPIC 24-119, SATA ALPM, ACPI HEPT Table, Suspend to RAM.

Fast Boot and CSM are disabled.

Installing Proxmox 8.1 following the guides (https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough) and cross-confirming infos with some Youtube tutorials went quite flawlessly, here are the modification i made:

Added the PVE no-subs repository
Removed PVE enterprise repositories
Reload, update and upgrade, reboot

root@lab:~# nano /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intremap=no_x2apic_output nox2apic acpi=off intel_iommu=on iommu=pt
(The 'intremap=no_x2apic_output nox2apic' part has been added after a first issue then i added 'acpi=off' when ACPI started to complain)

root@lab:~# nano /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd (seems not required but i added it as recomended)

root@lab:~# proxmox-boot-tool refresh
root@lab:~# update-initramfs -u -k all
root@lab:~# reboot

root@lab:~# lspci
root@lab:~# lsmod | grep vfio

root@lab:~# dmesg | grep -e DMAR -e IOMMU
root@lab:~# dmesg | grep -i iommu

root@lab:~# cat /proc/cmdline
root@lab:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done


The outputs seemed to confirm that IOMMU was enabled and working, i then proceeded to install TrueNAS 23.10.1 prior to pass-through any other device to the VM, RAM ballooning disabled and using vmbr as setup network, once installed i shutdown the VM to go back in Proxmox with the goal of passing the ASMEDIA controller (and the 2x 4TB HDD connected to it) to TrueNAS, then do the same with the RTL8125 NIC, here is where the issue appears, when passing a device the VM wont start and display 'Error: cannot prepare PCI pass-through, IOMMU not present' in the task log of Proxmox...

I'm going to keep digging but if anyone has a clue or a idea to share with me, that would be amazing !

Thanks a lot for taking the time reading this !

 

[leesteken]

Looks like both CPU and motherboard chipset support VT-d and intel_iommu=on is active. But devices are not assigned to IOMMU groups, which makes me think VT-d is not (fully) enabled in the motherboard BIOS settings (or it's disabled by the BIOS). Proxmox can't do anything about this. Double check your BIOS settings and maybe try ASRock support or forum?

 

[cik28]

Looks like both CPU and motherboard chipset support VT-d and intel_iommu=on is active. But devices are not assigned to IOMMU groups, which makes me think VT-d is not (fully) enabled in the motherboard BIOS settings (or it's disabled by the BIOS). Proxmox can't do anything about this. Double check your BIOS settings and maybe try ASRock support or forum?

Yeah thats the first things i had in mind, the BIOS is not flashable (as far i tested) with casual manners, i've tried to contact the customer support (its been few days im on it...) but they were unable to provide infos on "a legacy product"...

Thanks for confirming that the iommu is indeed enabled !

 

[leesteken]

Try updating the motherboard BIOS to the latest version (7.4 might have been a Beta or preview). Maybe use a smaller and older USB stick? Or use FreeDOS: http://wiki.freedos.org/wiki/index.php/How_to_Create_a_USB_Boot_Disk_Using_FreeDOS

 

[cik28]

Try updating the motherboard BIOS to the latest version (7.4 might have been a Beta or preview). Maybe use a smaller and older USB stick? Or use FreeDOS: http://wiki.freedos.org/wiki/index.php/How_to_Create_a_USB_Boot_Disk_Using_FreeDOS

Yeah i did try with 2 rly old USB thumbdrive (4GB in FAT32 and FAT) on a 2.0 and on a 3.0 port, also tried with a 20GB 2.5" old XBox HDD in FAT32 and FAT (plugged as a proper HDD with SATA) for the Instant Flash utility, then i did the same with Rufus to prepare these as FreeDOS bootable devices, the ASRock 'BIOS.exe' file didnt worked... I have not found much info on the weird version i have, P7.4 might be a enterprise/company BIOS? or mass deployed library, school whatnot version?

 

[cik28]

One thing just came in mind that i have not tested would be to try flashing a BIOS from a identical board but from a different revision, in hope this work with my R3.0 (late production batch i guess)

https://www.asrock.com/mb/Intel/H110M-DGS/index.asp#BIOS


Edit: No image file detected via the Instant Flash tool (Using the USB thumbdrive in FAT32 with the BIOS P7.70 file at the root of the drive)

Going to try in DOS with this BIOS in few minutes

 

[celemine1gig]

The latest BIOS has vt-d enabled, according to a quick check. See attached screenshot from BIOS 7.70.

The Board is using a standard AMI Aptio V BIOS, so of course this is flashable.
See the AMI AFU update utility: https://www.ami.com/bios-uefi-utilities/
You'll neeed:
- a FAT32 formatted USB thumb drive
- the "Instant Flash" BIOS package from Asrock (https://download.asrock.com/BIOS/1151/H110M-DGS(7.70)ROM.zip)
- the AMI AFU tool for EFI
- probably a precompiled EFI shell binary package

Then you should be good to go. The details you'll work out on your own with a little google-magic.

Edit:
For questions concerning BIOS topics, I can recomment the old win-raid forum: https://winraid.level1techs.com/c/bios-uefi-modding/7
Edit2:
As I know Asrock, you might even be able to make it work (i.e. flash) by just putting the BIOS file onto a FAT32 formatted thumb drive and then using the BIOS integrated flash tool in the setup.

 

[cik28]

The latest BIOS has vt-d enabled, according to a quick check. See attached screenshot from BIOS 7.70.

The Board is using a standard AMI Aptio V BIOS, so of course this is flashable.
See the AMI AFU update utility: https://www.ami.com/bios-uefi-utilities/
You'll neeed:
- a FAT32 formatted USB thumb drive
- the "Instant Flash" BIOS package from Asrock (https://download.asrock.com/BIOS/1151/H110M-DGS(7.70)ROM.zip)
- the AMI AFU tool for EFI
- probably a precompiled EFI shell binary package

Then you should be good to go. The details you'll work out on your own with a little google-magic.

Edit:
For questions concerning BIOS topics, I can recomment the old win-raid forum: https://winraid.level1techs.com/c/bios-uefi-modding/7
Edit2:
As I know Asrock, you might even be able to make it work (i.e. flash) by just putting the BIOS file onto a FAT32 formatted thumb drive and then using the BIOS integrated flash tool in the setup.

Thanks for your answer it gives me a good path to dive in but ngl after reading a bit the documentation im lost on what to do, it seems pretty tricky to fix if smth is done wrong and without proper knowledge.

I will keep digging, looking for some kind of tutorial on the 101 of BIOS flashing using the AFU, the link you provided to the level1 forums seems plentyfull of ressources !

In hope i can get this done soon enough to finish my lil' home setup haha

Edit: also tried to get back into BIOS to use the Instant Flash tool but the file seems to not be found by the tool, on Windows it shows as a .70 or .50 file type but i guess its just the version naming that is messing with the file extention shown to the user?

 

[cik28]

Update on the issue:

I ended up changing plateforme after trying to flash it and make it work without any interesting results, i can't rly spend more time on the LGA 1151 system...

I simplified a bit the layout, here are the new plateforme specifications:

CPU: Ryzen 5 PRO 4650G
RAM: 16GB (2x 8GB EUDIMM 3200MT/s)
Motherboard: Gigabyte B550M S2H
GPU: GTX 750 Ti (For Windows VM)
NetworkPVE: RTL8111 GbE (enp6s0 bridge vmbr0)
NetworkNAS: RTL8125 2.5GbE (enp5s0) <== connected via PCI-e x1 NIC
Boot SSD: 128GB ext4 2.5" (PVE + ISOs) <== connected to PCH
VM HDD: 750GB ext4 2.5" <== connected to PCH
NAS HDD: 2x 4TB <== connected via ASMEDIA PCI-e x1 daughter board

After following some tutorials and the documentation again, i can now passthrough the GPU to a Windows VM and it seems to work fine for now, the issue lays with TrueNas...

When installing TrueNas, all goes fine, when trying to passthrough the ASMedia controller or the 2.5GbE Nic, the PVE Host "freezes" and i have to force shutdown it via keeping the power button pressed.

Here are the output of the commands used to verify if everything is enabled properly and i think it is (i might have missed smth but i can't put my finger on what it is if so...)

root@local:~# nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

root@local:~# nano /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd (seems not required but i added it as recomended)

root@local:~# proxmox-boot-tool refresh
root@local:~# update-initramfs -u -k all
root@local:~# reboot

root@local:~# lsmod | grep vfio
root@local:~# lspci

root@local:~# dmesg | grep -e DMAR -e IOMMU
root@local:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

root@local:~# dmesg | grep -i iommu


I have no idea what is going on, tried to check on similar issues with no useful clues as far i could tell...

If anyone has an idea and can share a hint or smth, that would be greatly welcomed.

Thanks again for taking the time reading this post !

 

[celemine1gig]

Devices 2-6 are all on the same IOMMU group. See your own screenshots.
That means you cannot pass-through just one, but you can only handle them all at a time.
You are trying to pass only devices 4 (Asmedia SATA) and 5 (Realtek 2.5G Ethernet) to a VM. This is why the whole system freezes, as you simply cannot do that, on your hardware.
Try to read up more on the general topics of passing-through devices.