[SOLVED] iommu issues

rainbear · Wednesday at 08:40

cpu:Intel(R) Xeon(R) CPU E5-2698B v3
gpu:Radeon 5 320
pve version:8.3.0
kernel:6.8.12-4-pve
The BIOS options only have VMX and it's enabled
/etc/default/grub

Bash:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:off,vesafb:off pcie_acs_override=downstream"
GRUB_CMDLINE_LINUX=""

# If your computer has multiple operating systems installed, then you
# probably want to run os-prober. However, if your computer is a host
# for guest OSes installed via LVM or raw disk devices, running
# os-prober can cause damage to those guest OSes as it mounts
# filesystems to look for things.
#GRUB_DISABLE_OS_PROBER=false

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

/etc/kernel/cmdline

Code:

intel_iommu=on iommu=pt video=efifb:off,vesafb:off pcie_acs_override=downstream

/etc/modules

Code:

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.

coretemp
nct6775
vfio
vfio_iommu_type1
vfio_pci
#vfio_virqfd

/etc/modprobe.d/iommu_unsafe_interrupts.conf

Code:

options vfio_iommu_type1 allow_unsafe_interrupts=1

/etc/modprobe.d/pve-blacklist.conf

Code:

# This file contains a list of modules which are not supported by Proxmox VE

# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist amdgpu
blacklist radeon
blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi
blacklist i915

/etc/modprobe.d/vfio.conf

Code:

options vfio-pci ids=1002:6778,1002:aa98

dmesg | grep -e DMAR -e IOMMU

Code:

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.142983] DMAR: IOMMU enabled

dmesg | grep remapping

Code:

[    0.405446] x2apic: IRQ remapping doesn't support X2APIC mode

lspci -nn | grep VGA

Code:

02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos XT [Radeon HD 7470/8470 / R5 235/310 OEM] [1002:6778]

lspci -nnk

Code:

02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos XT [Radeon HD 7470/8470 / R5 235/310 OEM] [1002:6778]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Caicos XT [Radeon HD 7470/8470 / R5 235/310 OEM] [1462:3310]
        Kernel modules: radeon, amdgpu
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM] [1002:a...
        Subsystem: Micro-Star International Co., Ltd. [MSI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM] [1462:aa98]
        Kernel modules: snd_hda_intel

I've been struggling with this problem for quite some time and haven't figured out how to solve it.

cheiss · Wednesday at 09:20

Hi,

please provide the full output of

Code:

pvesh get /nodes/localhost/hardware/pci --pci-class-blacklist ""

pcie_acs_override=downstream and options vfio_iommu_type1 allow_unsafe_interrupts=1 are usually not needed on better hardware, which handles passthrough well. These options are workarounds and might or might not break things further down the road.

what kind of motherboard do you have? As PCIe passthrough is heavily depended on the firmware.

rainbear · Wednesday at 09:47

cheiss said:
Hi,

please provide the full output of

Code:

pvesh get /nodes/localhost/hardware/pci --pci-class-blacklist ""

pcie_acs_override=downstream and options vfio_iommu_type1 allow_unsafe_interrupts=1 are usually not needed on better hardware, which handles passthrough well. These options are workarounds and might or might not break things further down the road.

what kind of motherboard do you have? As PCIe passthrough is heavily depended on the firmware.

The command execution results are too lengthy to include directly in the post, so I've placed them in a file.

The motherboard is an X99 model. I'm not sure if this is a universal motherboard since I'm in China. The CPU socket is LGA1700, and the chipset is H610.

cheiss · Wednesday at 09:56

rainbear said:
The motherboard is an X99 model.

At least the chipset is usually good for these, but firmware is of course still heavily depended on the vendor.

rainbear said:
The BIOS options only have VMX and it's enabled

Is there no option for VT-d or similar?

The command results are what I expected, the PCIe devices are not grouped, as indicated by the iommugroup column being -1 for all devices.

Did you reboot after applying all changes?
Have you verified that e.g. the kernel commandline is applied using cat /proc/cmdline?

rainbear · Wednesday at 09:59

cheiss said:
At least the chipset is usually good for these, but firmware is of course still heavily depended on the vendor.

Is there no option for VT-d or similar?

The command results are what I expected, the PCIe devices are not grouped, as indicated by the iommugroup column being -1 for all devices.

Did you reboot after applying all changes?
Have you verified that e.g. the kernel commandline is applied using cat /proc/cmdline?

The system has been restarted. I succeeded once before, but for some reason it stopped working afterward.
The BIOS only offers VMX, not VT-d.

Code:

root@pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-4-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt video=efifb:off,vesafb:off pcie_acs_override=downstream
root@pve:~#

rainbear · Thursday at 04:08

Does anyone know how to fix this?

Impact · Thursday at 08:04

Would you mind providing dmidecode -t baseboard | grep -E "Manufacturer|Product Name" and lscpu | grep -E "Model name|Flags"?

rainbear · Thursday at 08:22

Impact said:
dmidecode -t baseboard | grep -E "Manufacturer|Product Name"

Code:

        Manufacturer: JGINYUE
        Product Name: X99I Server

Impact said:
lscpu | grep -E "Model name|Flags"

Code:

Model name:                           Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00GHz
BIOS Model name:                      Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00GHz  CPU @ 2.0GHz
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb pti intel_ppin ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts vnmi

celemine1gig · Friday at 00:10

The board itself (hardware-wise) would be relatively good, however, the BIOS is really bad. And no support from the "manufacturer" at all.
I have that same board and contacted them with a question. No answer at all.

So, you basically have two choices:
- Scrap that POS and get something more stable and compatible.
- Learn how to work with a SPI-programmer and/or AMI BIOS tools.

I am still asking myself, why oh why I once thought that getting this board would be a good idea.

uzumo · Friday at 02:46

The BIOS only offers VMX, not VT-d.

It's not that the BIOS doesn't have VT-d.
However, since the BIOS was unofficial or had operational issues, I discarded that motherboard.

rainbear · Friday at 03:36

uzumo said:
It's not that the BIOS doesn't have VT-d.
However, since the BIOS was unofficial or had operational issues, I discarded that motherboard.

View attachment 90734

celemine1gig said:
The board itself (hardware-wise) would be relatively good, however, the BIOS is really bad. And no support from the "manufacturer" at all.

I found the address for the motherboard BIOS update.
https://www.jginyue.com/index/Article/show/cat_id/48/id/71

uzumo · Friday at 03:46

No. The BIOS for that motherboard is not an official version; it is merely using an evaluation BIOS.

rainbear · Friday at 03:50

uzumo said:
No. The BIOS for that motherboard is not an official version; it is merely using an evaluation BIOS.

What should I do?

uzumo · Friday at 04:47

Even with such an unstable BIOS, the VT-d menu is present, so you should use that.
*That manufacturer's motherboard uses a BIOS intended for testing by the motherboard manufacturer, not one that should be used by the general public, and this does not necessarily function properly.

The VT-d menu exists within the hierarchical structure shown in the image, even in the initial x99i server BIOS X990A006.bin.

If it still doesn't work, you'll have no choice but to replace it.
*Buying the epc612d4i should be fine, but you should buy the latest consumer motherboard/CPU/RAM instead.

After changing the BIOS settings, verify the following
*The display of “DMAR: IOMMU enabled” does not confirm that the IOMMU is truly enabled.

https://vfio.blogspot.com/2016/09/intel-iommu-enabled-it-doesnt-mean-what.html?m=1

celemine1gig · Sunday at 21:31

In the meantime I spent about a week, in order to get a BIOS onto that board (X99I Server), that would make it behave, the way it should for my use case.
That means enabled and working ECC and also enabled and working Serial Console Redirection. Although the physical connection for the latter needs to be made manually (soldering) on the NCP5532D SIO.
So, for anyone considering that board and not being able to directly modify the board: STAY AWAY from that thing!
Much too expensive for the crap you actually get.

P.S.: I never bothered running Proxmox with the stock BIOS, as this was unbearably bad. So I cannot judge about your actual issue. All I can say is: In theory VT-d should work just fine on this. It now does for me. So if it doesn't for you, then you can say thanks to Jinguye for this "great" product.

rainbear · 2025-09-22T03:17:26+0200

celemine1gig said:
与此同时，我花了大约一周的时间，将 BIOS 安装到该主板（X99I 服务器）上，使其按照我的用例的方式运行。
这意味着 ECC 和串行控制台重定向均已启用并正常工作。尽管后者的物理连接需要在 NCP5532D SIO 上手动（焊接）完成。
因此，对于任何考虑使用该电路板但又无法直接修改该电路板的人来说：远离该东西！
对于你实际得到的垃圾来说，这太贵了。

附言：我从来没想过用原厂BIOS运行Proxmox，因为那简直糟糕透了。所以我无法判断你的实际问题。我只能说：理论上VT-d应该可以正常工作。我现在确实可以正常工作。所以，如果你不行，那就感谢Jinguye，感谢它带来了这款“优秀”的产品。

celemine1gig said:
AMI BIOS 工具

I used this tool to enable the VT-d option by default. After flashing the modified BIOS file onto the motherboard, it worked normally.

uzumo · 2025-09-22T03:32:28+0200

If resolved, please mark as SOLVED.

rainbear · 2025-09-22T03:36:33+0200

uzumo said:
If resolved, please mark as SOLVED.

I don't see where to set it.

uzumo · 2025-09-22T03:56:59+0200

Click “Edit thread” at the top of the thread.
Select “Solved” from the prefix and save.

[SOLVED] iommu issues

New Member

Proxmox Staff Member

New Member

Attachments

Proxmox Staff Member

New Member

New Member

Renowned Member

New Member

Renowned Member

Active Member

New Member

Active Member

New Member

Active Member

Renowned Member

New Member

Active Member

New Member

Active Member

We value your privacy