PCI Passthrough: No IOMMU detected

[akihomura]

Hello, I am trying to passthrough my SATA controller by following the Wiki page, but it still not work and show a "No IOMMU detected" notice in the web.

I have edit the grub command line and enable the kernel module:

root@pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.4.65-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on

The VT-D option on my BIOS is enable, and dmesg is shown that it's working:

root@pve:~# dmesg | grep "Virtualization Technology for Directed I/O"
[ 1.276695] DMAR: Intel(R) Virtualization Technology for Directed I/O

IOMMU group is valid: (my device is 04:00.0)

root@pve:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1c.2
/sys/kernel/iommu_groups/15/devices/0000:04:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:1c.0
/sys/kernel/iommu_groups/13/devices/0000:02:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.2
/sys/kernel/iommu_groups/11/devices/0000:00:1f.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.3
/sys/kernel/iommu_groups/1/devices/0000:00:03.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.3
/sys/kernel/iommu_groups/16/devices/0000:05:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.1
/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:1b.0
/sys/kernel/iommu_groups/12/devices/0000:01:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:14.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4

But it still not work, full dmesg is attached, please help me.

 

[leesteken]

IOMMU appears to be enabled (otherwise, you would have no groups), but maybe the VFIO drivers are not loaded? Did you do a update-initramfs -u and restart?
Please show us:
- The contents of /etc/modules
- The contents of files that you added to /etc/modprobe.d/
- Your VM configuration (/etc/pve/qemu-server/<vmid>.conf)
- The journalctl entries from around the time of starting the VM (look for "start VM <vmid>" up to the error messages)

 

[akihomura]

I am using the 5.4 kernel so the /etc/modules file is empty, but module is loaded.

root@pve:~# lsmod | grep vfio
vfio_pci 53248 1
vfio_virqfd 16384 1 vfio_pci
vfio_iommu_type1 32768 1
vfio 32768 5 vfio_iommu_type1,vfio_pci
irqbypass 16384 15 vfio_pci,kvm

files in /etc/modprobe.d/

root@pve:~# cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb

root@pve:~# cat /etc/modprobe.d/pve-passthrough.conf
# options vfio-pci ids=1b4b:9215
# options vfio-iommu-type1 allow_unsafe_interrupts=1

I have try to uncomment upper two line, but does not working.

VM Config:

root@pve:~# cat /etc/pve/qemu-server/101.conf
agent: 1
bios: ovmf
bootdisk: scsi0
cores: 2
efidisk0: local-lvm:vm-101-disk-1,size=4M
hostpci0: 04:00
ide2: local:iso/debian-10.5.0-amd64-netinst.iso,media=cdrom
machine: q35
memory: 6656
name: NAS
net0: virtio=0E:A2:1F:EC:B4:9A,bridge=vmbr1,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-101-disk-0,size=128G
scsi1: /dev/disk/by-id/ata-ST2000LM015-2E8174_WDZPEA3C,size=1953514584K
scsihw: virtio-scsi-pci
smbios1: uuid=e57c5b3d-79b8-4fa3-858b-6487c776e2bf
sockets: 1
startup: order=2
vcpus: 2
vga: qxl,memory=128
vmgenid: c46e7c1a-05a5-4c24-82f8-7605bdda70b5

Error log:

Nov 19 09:42:52 pve QEMU[1103]: kvm: vfio-pci: Cannot read device rom at 0000:04:00.0
Nov 19 09:42:52 pve QEMU[1103]: Device option ROM contents are probably invalid (check dmesg).
Nov 19 09:42:52 pve QEMU[1103]: Skip option ROM probe with rombar=0, or load from file with romfile=

I disable the rombar option and the error disappear, but the device is not recognized in the VM.

 

[leesteken]

You need to run update-initramfs -u and reboot to effectuate changes to files in /etc/modprobe.d/.

Maybe you need to bind the vfio-pci driver to the SATA controller before the system's SATA driver. You can do this via the kernel command line options vfio-pci.ids=1b4b:9215 (to make sure the SATA driver is not used), and use lspci -k to check that the SATA driver did not touch the SATA controller.
Mayube you need to dump the ROM of the SATA controller (without it being touched by the driver) to a file and use that in the pcihost0 romfile= setting.
PS: this post is about someone having success with this controller, maybe it helps you?

 

[akihomura]

Yes, I have update initramfs after edit files in /etc/modprobe.d/

I think the driver is correctly bind to the SATA controller:

04:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev ff)
Kernel driver in use: vfio-pci
Kernel modules: ahci

I also change machine type to pc-q35-3.1 like the post, but still not work.

There is a error on my VM's dmesg, I think this is the problem:

[ 0.926814] pci 0000:06:10.0: [1b4b:9215] type 7f class 0xffffff
[ 0.926996] pci 0000:06:10.0: unknown header type 7f, ignoring device

And how can I get the ROM of SATA controller?

 

[leesteken]

How did you go from "No IOMMU detected" to passthough working with ROM issue inside VM?

Have you tried hostpci0: 04:00.0 (because it is not a multifunction device, but I don't think it will help)?

Dumping the ROM to a file should be similar to doing it for GPUs, and do it without using the device or loading the SATA driver (fresh reboot):
echo 1 > '/sys/bus/pci/devices/0000:04:00.0/rom'
cat '/sys/bus/pci/devices/0000:04:00.0/rom' >sata04.rom
Put the sata04.rom file in the /usr/share/kvm/ folder and add ,romfile=sata04.rom to the hostpci0 line.

 

[akihomura]

The "No IOMMU detected" message is still there, so I think it might be display problem.

I have try the "hostpci0: 04:00.0" option, but it doesn't works.

 

[dcsapak]

can you post the output of

pvesh get /nodes/zita/hardware/pci --output-format-pretty

?

caution, the output can be very big

 

[akihomura]

--output-format-pretty is invalid format option, so I use the --output-format json-pretty.

 

[dcsapak]

--output-format-pretty is invalid format option, so I use the --output-format json-pretty.

sorry, that was what i actually meant

so the message is just a false positive (since your first device has no iommu group) but this has nothing to do with it not working
is this pci device uefi capable? did you try with seabios instead of ovmf already?

 

[akihomura]

so the message is just a false positive (since your first device has no iommu group) but this has nothing to do with it not working

So my IOMMU is not working correctly?

is this pci device uefi capable? did you try with seabios instead of ovmf already?

Yes, I could use this device in host with UEFI.
In seabios this device is not working like in ovmf.

 

[dcsapak]

hi, sry for the late answer

So my IOMMU is not working correctly?

is seems to be working correctly, just our detection for the gui notice is wrong

In seabios this device is not working like in ovmf.

mhmm.. then i have really no idea what can be the problem...