Problem with NIC passthough to Windows Guest

turin231

New Member
Apr 20, 2020
19
1
1
37
I have been trying implement a PCI passthrough for a 25g NIC to a Windows Server 2016 guest but it does not seem to work.

The Host server is a Dell Poweredge with a 1st gen Epyc CPU. I have tried the same passthrough with CentOS 8 guests on a different server with the same hardware (same CPU same NIC) without issues.

Yet here the VMs will not boot up with the passthrough in place:
TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name analsyisVM -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=e79e8830-310a-4c02-8588-e5dd9e48e6ef' -smp '120,sockets=1,cores=120,maxcpus=120' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/100.vnc,password -no-hpet -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,enforce' -m 223510 -device 'vmgenid,guid=01720a3f-afef-4d86-a0b0-c91574555808' -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0000:e1:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,rombar=0,multifunction=on' -device 'vfio-pci,host=0000:e1:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'VGA,id=vga,bus=pcie.0,addr=0x1' -chardev 'socket,path=/var/run/qemu-server/100.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:f6d9f03858f0' -drive 'file=/mnt/pve/ISOs/template/iso/virtio-win-0.1.171.iso,if=none,id=drive-ide0,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/Images/vm-100-disk-0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35+pve1' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeout

Kernel modules are loaded as described in the wiki

The output if dmesg | grep -e DMAR -e IOMMU -e AMD-Vi is this:

Screenshot from 2021-04-26 00-21-33.png

And the Correct Driver seems to be loaded:
Screenshot from 2021-04-26 00-20-23.png

Any ideas about what the problem is? Does maybe the a Windows guest need some additional preparation?
 
Last edited:

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
5,632
580
133
32
Vienna
can you show us your vm config? (qm config ID)? also the complete dmesg output please
 

turin231

New Member
Apr 20, 2020
19
1
1
37
can you show us your vm config? (qm config ID)? also the complete dmesg output please
VM config:


agent: 1
bootdisk: scsi0
cores: 120
ide0: ISOs:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K
machine: q35
memory: 223510
name: analsyisVM
net0: virtio=9A:E8:C0:45:B4:7D,bridge=vmbr1
numa: 0
onboot: 1
ostype: win10
scsi0: Images:vm-100-disk-0,discard=on,size=2000G
scsihw: virtio-scsi-pci
smbios1: uuid=e79e8830-310a-4c02-8588-e5dd9e48e6ef
sockets: 1
vmgenid: 01720a3f-afef-4d86-a0b0-c91574555808

I am adding the passthrough from the web interface like so:

Screenshot from 2021-05-03 18-25-36.png
Screenshot from 2021-05-03 18-25-46.png

The full dmesg:https://pastebin.com/s0q9YUe5
 
Last edited:

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
5,632
580
133
32
Vienna
ok can you please configure it and start it again and post the error message? the pci ids do not match from your first post
also i need the dmesg output from after you tried to start the vm (i cannot detect where you have it started)
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
5,632
580
133
32
Vienna
oh also you seem to use an old version, i'd recommend upgrading
 

turin231

New Member
Apr 20, 2020
19
1
1
37
oh also you seem to use an old version, i'd recommend upgrading


I did do an upgrade and the error message now is more specific.


kvm: -device vfio-pci,host=0000:e1:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,rombar=0: vfio 0000:e1:00.0: hardware reports invalid configuration, MSIX PBA outside of specified BAR
TASK ERROR: start failed: QEMU exited with code

Dmesg output: https://pastebin.com/6zNvSKrV

Configuration of the passthrough:

Screenshot from 2021-05-04 20-38-17.png

Screenshot from 2021-05-04 20-38-11.png

Sorry for the mix up before. There are more than one NICs on the machine.

The VM configuration is the same as in the previous post but without the net0:
virtio=9A:E8:C0:45:B4:7D,bridge=vmbr1
and added the PCI device.
 
Last edited:

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
5,632
580
133
32
Vienna
so you did not blacklist the module responsible for the network device, this can be seen here:
[ 14.857710] bnxt_en 0000:e1:00.0 enp225s0f0np0: NIC Link is Up, 25000 Mbps full duplex, Flow control: ON - receive & transmit
[ 14.857713] bnxt_en 0000:e1:00.0 enp225s0f0np0: FEC autoneg off encodings: None
[ 15.199995] bnxt_en 0000:e1:00.1 enp225s0f1np1: NIC Link is Up, 25000 Mbps full duplex, Flow control: ON - receive & transmit
[ 15.199998] bnxt_en 0000:e1:00.1 enp225s0f1np1: FEC autoneg off encodings: None

maybe do that, so the host does not initialize the nic before passing it through to the vm
 

showiproute

Member
Mar 11, 2020
212
7
18
Austria
Not sure if this would work or not but your could enable a Virtual Function (SRIOV) of the network interfaces and pass that to your VM?
AFAIK you can enable SRIOV within BIOS.

Then you do not need to blacklist the whole NIC for Proxmox but can use a VF of it
 

turin231

New Member
Apr 20, 2020
19
1
1
37
Not sure if this would work or not but your could enable a Virtual Function (SRIOV) of the network interfaces and pass that to your VM?
AFAIK you can enable SRIOV within BIOS.

Then you do not need to blacklist the whole NIC for Proxmox but can use a VF of it

SRIOV is already enabled globally. I will try to blacklist the module.

@dcsapak The module to blacklist would be bnxt_en correct?
 

showiproute

Member
Mar 11, 2020
212
7
18
Austria
If you disable the module the kernel will do not load the drivers for your NIC meaning that both PCIe NICs will not be used from Proxmox.
Keep that in mind if you have bridges or are using the NICs in a different context.

Why did you not use a virtual function for your VM?
 

turin231

New Member
Apr 20, 2020
19
1
1
37
If you disable the module the kernel will do not load the drivers for your NIC meaning that both PCIe NICs will not be used from Proxmox.
Keep that in mind if you have bridges or are using the NICs in a different context.

Why did you not use a virtual function for your VM?
I do but the performance is really subpar even with Vrtio. On other machines this was only solved with a passthrough.

The good thing is that the only other bridge to be used if the passthrough works is a 1g one that uses a different module thus not really an issue to blacklist the driver.
 
Last edited:

showiproute

Member
Mar 11, 2020
212
7
18
Austria
I do but the performance is really subpar even with Vrtio. On other machines this was only solved with a passthrough.
Usually if you passthrough a VF you do not need a VirtIO as you would passthrough a PCIe device from Proxmox perspective.
 

turin231

New Member
Apr 20, 2020
19
1
1
37
so you did not blacklist the module responsible for the network device, this can be seen here:


maybe do that, so the host does not initialize the nic before passing it through to the vm

Same result with the driver blacklisted:


kvm: -device vfio-pci,host=0000:e1:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,rombar=0,multifunction=on: vfio 0000:e1:00.0: hardware reports invalid configuration, MSIX PBA outside of specified BAR
TASK ERROR: start failed: QEMU exited with code 1

dmesg output: https://pastebin.com/6ve8bdLH

blacklist contents:

Screenshot from 2021-05-07 21-41-11.png
The vfio-cpi driver seems to be loaded

Screenshot from 2021-05-07 21-36-25.png
 

turin231

New Member
Apr 20, 2020
19
1
1
37
You might give it a try to pass throught each NIC separate without the "all function" checked?
I could try but on other similar machines when i did that, the passthrough would fail as it would try to passthrough the same NIC twice. Only passing once with all functionality worked in passing both interfaces. I am really not sure what is up here.

any ideas @dcsapak maybe?
 
Last edited:

showiproute

Member
Mar 11, 2020
212
7
18
Austria
Not sure how it works for you but my X550-T2 are having two individual PCIe :

Unbenannt.PNG


The same would work for my other server where I also do have BMC NICs.
 

turin231

New Member
Apr 20, 2020
19
1
1
37
ok i finally found the issue. Turns that it was the actual NIC. For what ever reason the specific NIC would only work bridged and not using pass through. When i switched to the other pci NIC the passthrough worked. No module blacklist needed either. So i will just passthrough the working NIC on the widnows VM and use the other one bridged to the other VMs.
 
  • Like
Reactions: showiproute

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!