Problem with NIC passthough to Windows Guest

[turin231]

I have been trying implement a PCI passthrough for a 25g NIC to a Windows Server 2016 guest but it does not seem to work.

The Host server is a Dell Poweredge with a 1st gen Epyc CPU. I have tried the same passthrough with CentOS 8 guests on a different server with the same hardware (same CPU same NIC) without issues.

Yet here the VMs will not boot up with the passthrough in place:

TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name analsyisVM -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=e79e8830-310a-4c02-8588-e5dd9e48e6ef' -smp '120,sockets=1,cores=120,maxcpus=120' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/100.vnc,password -no-hpet -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,enforce' -m 223510 -device 'vmgenid,guid=01720a3f-afef-4d86-a0b0-c91574555808' -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0000:e1:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,rombar=0,multifunction=on' -device 'vfio-pci,host=0000:e1:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'VGA,id=vga,bus=pcie.0,addr=0x1' -chardev 'socket,path=/var/run/qemu-server/100.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:f6d9f03858f0' -drive 'file=/mnt/pve/ISOs/template/iso/virtio-win-0.1.171.iso,if=none,id=drive-ide0,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/Images/vm-100-disk-0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35+pve1' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeout

Kernel modules are loaded as described in the wiki

The output if dmesg | grep -e DMAR -e IOMMU -e AMD-Vi is this:



And the Correct Driver seems to be loaded:


Any ideas about what the problem is? Does maybe the a Windows guest need some additional preparation?

 

[dcsapak]

can you show us your vm config? (qm config ID)? also the complete dmesg output please

 

[turin231]

can you show us your vm config? (qm config ID)? also the complete dmesg output please

VM config:

agent: 1
bootdisk: scsi0
cores: 120
ide0: ISOs:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K
machine: q35
memory: 223510
name: analsyisVM
net0: virtio=9A:E8:C0:45:B4:7D,bridge=vmbr1
numa: 0
onboot: 1
ostype: win10
scsi0: Images:vm-100-disk-0,discard=on,size=2000G
scsihw: virtio-scsi-pci
smbios1: uuid=e79e8830-310a-4c02-8588-e5dd9e48e6ef
sockets: 1
vmgenid: 01720a3f-afef-4d86-a0b0-c91574555808

I am adding the passthrough from the web interface like so:




The full dmesg:https://pastebin.com/s0q9YUe5

 

[dcsapak]

ok can you please configure it and start it again and post the error message? the pci ids do not match from your first post
also i need the dmesg output from after you tried to start the vm (i cannot detect where you have it started)

 

[dcsapak]

oh also you seem to use an old version, i'd recommend upgrading

 

[turin231]

oh also you seem to use an old version, i'd recommend upgrading

I did do an upgrade and the error message now is more specific.

kvm: -device vfio-pci,host=0000:e1:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,rombar=0: vfio 0000:e1:00.0: hardware reports invalid configuration, MSIX PBA outside of specified BAR
TASK ERROR: start failed: QEMU exited with code

Dmesg output: https://pastebin.com/6zNvSKrV

Configuration of the passthrough:





Sorry for the mix up before. There are more than one NICs on the machine.

The VM configuration is the same as in the previous post but without the net0:
virtio=9A:E8:C0:45:B4:7D,bridge=vmbr1
and added the PCI device.

 

[dcsapak]

so you did not blacklist the module responsible for the network device, this can be seen here:

[ 14.857710] bnxt_en 0000:e1:00.0 enp225s0f0np0: NIC Link is Up, 25000 Mbps full duplex, Flow control: ON - receive & transmit
[ 14.857713] bnxt_en 0000:e1:00.0 enp225s0f0np0: FEC autoneg off encodings: None
[ 15.199995] bnxt_en 0000:e1:00.1 enp225s0f1np1: NIC Link is Up, 25000 Mbps full duplex, Flow control: ON - receive & transmit
[ 15.199998] bnxt_en 0000:e1:00.1 enp225s0f1np1: FEC autoneg off encodings: None

maybe do that, so the host does not initialize the nic before passing it through to the vm

 

[showiproute]

Not sure if this would work or not but your could enable a Virtual Function (SRIOV) of the network interfaces and pass that to your VM?
AFAIK you can enable SRIOV within BIOS.

Then you do not need to blacklist the whole NIC for Proxmox but can use a VF of it

 

[turin231]

Not sure if this would work or not but your could enable a Virtual Function (SRIOV) of the network interfaces and pass that to your VM?
AFAIK you can enable SRIOV within BIOS.

Then you do not need to blacklist the whole NIC for Proxmox but can use a VF of it

SRIOV is already enabled globally. I will try to blacklist the module.

@dcsapak The module to blacklist would be bnxt_en correct?

 

[showiproute]

If you disable the module the kernel will do not load the drivers for your NIC meaning that both PCIe NICs will not be used from Proxmox.
Keep that in mind if you have bridges or are using the NICs in a different context.

Why did you not use a virtual function for your VM?

 

[turin231]

If you disable the module the kernel will do not load the drivers for your NIC meaning that both PCIe NICs will not be used from Proxmox.
Keep that in mind if you have bridges or are using the NICs in a different context.

Why did you not use a virtual function for your VM?

I do but the performance is really subpar even with Vrtio. On other machines this was only solved with a passthrough.

The good thing is that the only other bridge to be used if the passthrough works is a 1g one that uses a different module thus not really an issue to blacklist the driver.

 

[showiproute]

I do but the performance is really subpar even with Vrtio. On other machines this was only solved with a passthrough.

Usually if you passthrough a VF you do not need a VirtIO as you would passthrough a PCIe device from Proxmox perspective.

 

[turin231]

so you did not blacklist the module responsible for the network device, this can be seen here:


maybe do that, so the host does not initialize the nic before passing it through to the vm

Same result with the driver blacklisted:

kvm: -device vfio-pci,host=0000:e1:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,rombar=0,multifunction=on: vfio 0000:e1:00.0: hardware reports invalid configuration, MSIX PBA outside of specified BAR
TASK ERROR: start failed: QEMU exited with code 1

dmesg output: https://pastebin.com/6ve8bdLH

blacklist contents:


The vfio-cpi driver seems to be loaded

 

[showiproute]

@turin231 :

I am pasting my PCIe passthrough of a VF:






If I remeber right "All functions" would pass the whole NIC to the VM and not only one of the two interfaces.

 

[turin231]

@turin231 :

I am pasting my PCIe passthrough of a VF:

View attachment 25869




If I remeber right "All functions" would pass the whole NIC to the VM and not only one of the two interfaces.

That is what i want. To pass both interfaces and team them up in the VM.

 

[showiproute]

You might give it a try to pass throught each NIC separate without the "all function" checked?

 

[turin231]

You might give it a try to pass throught each NIC separate without the "all function" checked?

I could try but on other similar machines when i did that, the passthrough would fail as it would try to passthrough the same NIC twice. Only passing once with all functionality worked in passing both interfaces. I am really not sure what is up here.

any ideas @dcsapak maybe?

 

[showiproute]

Not sure how it works for you but my X550-T2 are having two individual PCIe :




The same would work for my other server where I also do have BMC NICs.

 

[turin231]

Not sure how it works for you but my X550-T2 are having two individual PCIe :

View attachment 25910


The same would work for my other server where I also do have BMC NICs.

Its looks the same but it would fail when i tried it in other machines...but as i said i will try it again ans see on this one.

 

[turin231]

ok i finally found the issue. Turns that it was the actual NIC. For what ever reason the specific NIC would only work bridged and not using pass through. When i switched to the other pci NIC the passthrough worked. No module blacklist needed either. So i will just passthrough the working NIC on the widnows VM and use the other one bridged to the other VMs.