PCI/PCIe Passthrough of Intel Virtual Function Network Card

[tobby]

Hi

I am trying to use the PCIe-Passthrough function of Proxmox 5.

root@t-hyper:~# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-7 (running version: 5.3-7/e8ed1e22)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-19
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-35
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-1
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-33
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-44
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

So I followed this article: https://pve.proxmox.com/wiki/Pci_passthrough

I added this to my kernel boot parameters (/etc/default/grub):

intel_iommu=on pcie_acs_override=downstream

and this to my /etc/sysfs.conf:

bus/pci/devices/0000:00:1c.4/0000:05:00.0/sriov_drivers_autoprobe = 0
bus/pci/devices/0000:00:1c.4/0000:05:00.1/sriov_drivers_autoprobe = 0
bus/pci/devices/0000:00:1c.4/0000:05:00.0/sriov_numvfs = 10
bus/pci/devices/0000:00:1c.4/0000:05:00.1/sriov_numvfs = 2

and this to /etc/modules:

        vfio
        vfio_iommu_type1
        vfio_pci
        vfio_virqfd

IOMMU-isolation seems to work:

root@t-hyper:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:05:10.4
/sys/kernel/iommu_groups/7/devices/0000:00:1c.4
/sys/kernel/iommu_groups/25/devices/0000:05:10.5
/sys/kernel/iommu_groups/15/devices/0000:05:10.0
/sys/kernel/iommu_groups/5/devices/0000:00:1c.0
/sys/kernel/iommu_groups/23/devices/0000:05:10.1
/sys/kernel/iommu_groups/13/devices/0000:05:00.1
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.1
/sys/kernel/iommu_groups/21/devices/0000:05:11.4
/sys/kernel/iommu_groups/11/devices/0000:03:00.0
/sys/kernel/iommu_groups/11/devices/0000:04:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/18/devices/0000:05:10.6
/sys/kernel/iommu_groups/8/devices/0000:00:1d.0
/sys/kernel/iommu_groups/26/devices/0000:05:10.7
/sys/kernel/iommu_groups/16/devices/0000:05:10.2
/sys/kernel/iommu_groups/6/devices/0000:00:1c.2
/sys/kernel/iommu_groups/24/devices/0000:05:10.3
/sys/kernel/iommu_groups/14/devices/0000:07:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:17.0
/sys/kernel/iommu_groups/22/devices/0000:05:11.6
/sys/kernel/iommu_groups/12/devices/0000:05:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:14.2
/sys/kernel/iommu_groups/2/devices/0000:00:14.0
/sys/kernel/iommu_groups/20/devices/0000:05:11.2
/sys/kernel/iommu_groups/10/devices/0000:01:00.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:05:11.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.2
/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.4

My virtual function network cards are also working and lspci shows them:

root@t-hyper:~# lspci
00:00.0 Host bridge: Intel Corporation Device 5918 (rev 05)
00:01.0 PCI bridge: Intel Corporation Skylake PCIe Controller (x16) (rev 05)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H Thermal subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:16.1 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #2 (rev 31)
00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #1 (rev f1)
00:1c.2 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #3 (rev f1)
00:1c.4 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #5 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
05:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
05:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
05:10.0 Ethernet controller: Intel Corporation X550 Virtual Function
05:10.1 Ethernet controller: Intel Corporation X550 Virtual Function
05:10.2 Ethernet controller: Intel Corporation X550 Virtual Function
05:10.3 Ethernet controller: Intel Corporation X550 Virtual Function
05:10.4 Ethernet controller: Intel Corporation X550 Virtual Function
05:10.5 Ethernet controller: Intel Corporation X550 Virtual Function
05:10.6 Ethernet controller: Intel Corporation X550 Virtual Function
05:10.7 Ethernet controller: Intel Corporation X550 Virtual Function
05:11.0 Ethernet controller: Intel Corporation X550 Virtual Function
05:11.2 Ethernet controller: Intel Corporation X550 Virtual Function
05:11.4 Ethernet controller: Intel Corporation X550 Virtual Function
05:11.6 Ethernet controller: Intel Corporation X550 Virtual Function
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a808

Now the problem:
If I add this to my VM

hostpci0: 05:00.1,pcie=1

I can passthrough the whole second port of my network card and it is working.

But If I add this:

hostpci0: 05:10.0,pcie=1

then I can't start the VM:

kvm: -device vfio-pci,host=05:10.0,id=hostpci0,bus=pci.0,addr=0x10: vfio error: 0000:05:10.0: failed to open /dev/vfio/15: No such file or directory
TASK ERROR: start failed: command '/usr/bin/kvm -id 104 -name template-ubuntu-1804 -chardev 'socket,id=qmp,path=/var/run/qemu-server/104.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/104.pid -daemonize -smbios 'type=1,family=Proxmox-VMs,manufacturer=Tobby,product=Ubuntu-VM,uuid=3dd793d5-74bd-45ac-82a6-addbc93476c7,version=3.1/18.04' -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/104.vnc,x509,password -cpu 'Skylake-Client-IBRS,+pcid,+spec-ctrl,+ssbd,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,vendor=GenuineIntel' -m 2048 -k de -device 'vmgenid,guid=9740b6bb-5b12-4322-83c6-b09aaf758f6e' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b' -device 'vfio-pci,host=05:10.0,id=hostpci0,bus=pci.0,addr=0x10' -chardev 'spicevmc,id=usbredirchardev0,name=usbredir' -device 'usb-redir,chardev=usbredirchardev0,id=usbredirdev0,bus=ehci.0' -chardev 'spicevmc,id=usbredirchardev1,name=usbredir' -device 'usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0' -chardev 'spicevmc,id=usbredirchardev2,name=usbredir' -device 'usb-redir,chardev=usbredirchardev2,id=usbredirdev2,bus=ehci.0' -chardev 'spicevmc,id=usbredirchardev3,name=usbredir' -device 'usb-redir,chardev=usbredirchardev3,id=usbredirdev3,bus=ehci.0' -chardev 'socket,id=serial0,path=/var/run/qemu-server/104.serial0,server,nowait' -device 'isa-serial,chardev=serial0' -device 'qxl-vga,id=vga,vgamem_mb=32,ram_size_mb=128,vram_size_mb=64,bus=pcie.0,addr=0x1' -chardev 'socket,path=/var/run/qemu-server/104.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -spice 'tls-port=61009,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on' -device 'virtio-serial,id=spice,bus=pci.0,addr=0x9' -chardev 'spicevmc,id=vdagent,name=vdagent' -device 'virtserialport,chardev=vdagent,name=com.redhat.spice.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -device 'i6300esb,bus=pci.0,addr=0x4' -watchdog-action reset -iscsi 'initiator-name=iqn.1993-08.org.debian:01:6b9367822e39' -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' -drive 'file=/dev/zvol/rpool/data/vm-104-disk-0,if=none,id=drive-scsi0,cache=writeback,rerror=report,werror=report,discard=on,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,rotation_rate=1,bootindex=100' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -drive 'if=none,id=drive-sata0,media=cdrom,rerror=report,werror=report,aio=threads' -device 'ide-cd,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=200' -machine 'type=q35'' failed: exit code 1

Any ideas why I can passthrough the whole card/port but not the virtual functions?

 

[dcsapak]

why did you add the acs override patch ?

intel_iommu=on pcie_acs_override=downstream

?

have you verified that vt-d is active ? e.g. via dmesg?

please try withouth the acs override and check if you have vt-d enabled in bios

 

[tobby]

why did you add the acs override patch ?

Good question. As far as I remember I had problems without it a long long time ago and since then I always add it. I just removed it but no change.

And like I said: Passing through the whole card works, so VT-d seems to work. But passing through the virtual functions of the network card does not work...

 

[dcsapak]

mhmm weird, i did not have a problem with a x520 (i think)
can you try to bind the virtual function to the vfio-pci driver with module paramters (see https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_pci_passthrough on how to do that)