DELL PowerEdge R710 ethernet passthrough issues

J

jaceqp

Well-Known Member
May 28, 2018
98
7
48
44
Hello there.
I'm currently testing out virtualisation using PROXMOX on Dell R710 servers
Config:
CPU: 2x Xeon X5670
RAM: 32GB DDR3 ECC
HDD/RAID: Perc H700 + 4x600GB SAS 2,5" (hardware RAID10)

I've installed 2 test VMs (Win2003 Server and Windows 2016 Standard).
Since server has 4x onboard Ethernet ports I'd like to try out some pci passthrough...

Now. I've made some basic passthrough setup (according to PROXMOX Passthrough WIKI - including GRUB modification, adding vfio, vmXXX.conf edit etc...)

My ethernet devices are shown below:
Code: 
root@test:~# lspci |grep Ethernet
01:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

IOMMU GROUPS:

Code: 
root@test:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:fe:02.5
/sys/kernel/iommu_groups/17/devices/0000:fe:02.3
/sys/kernel/iommu_groups/17/devices/0000:fe:02.1
/sys/kernel/iommu_groups/17/devices/0000:fe:02.4
/sys/kernel/iommu_groups/17/devices/0000:fe:02.2
/sys/kernel/iommu_groups/17/devices/0000:fe:02.0
/sys/kernel/iommu_groups/7/devices/0000:00:09.0
/sys/kernel/iommu_groups/25/devices/0000:ff:04.0
/sys/kernel/iommu_groups/25/devices/0000:ff:04.3
/sys/kernel/iommu_groups/25/devices/0000:ff:04.1
/sys/kernel/iommu_groups/25/devices/0000:ff:04.2
/sys/kernel/iommu_groups/15/devices/0000:03:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:06.0
/sys/kernel/iommu_groups/23/devices/0000:ff:02.3
/sys/kernel/iommu_groups/23/devices/0000:ff:02.1
/sys/kernel/iommu_groups/23/devices/0000:ff:02.4
/sys/kernel/iommu_groups/23/devices/0000:ff:02.2
/sys/kernel/iommu_groups/23/devices/0000:ff:02.0
/sys/kernel/iommu_groups/23/devices/0000:ff:02.5
/sys/kernel/iommu_groups/13/devices/0000:01:00.1
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:04.0
/sys/kernel/iommu_groups/21/devices/0000:fe:06.1
/sys/kernel/iommu_groups/21/devices/0000:fe:06.2
/sys/kernel/iommu_groups/21/devices/0000:fe:06.0
/sys/kernel/iommu_groups/21/devices/0000:fe:06.3
/sys/kernel/iommu_groups/11/devices/0000:00:1e.0
/sys/kernel/iommu_groups/11/devices/0000:08:03.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/18/devices/0000:fe:03.4
/sys/kernel/iommu_groups/18/devices/0000:fe:03.2
/sys/kernel/iommu_groups/18/devices/0000:fe:03.0
/sys/kernel/iommu_groups/18/devices/0000:fe:03.1
/sys/kernel/iommu_groups/8/devices/0000:00:14.2
/sys/kernel/iommu_groups/8/devices/0000:00:14.0
/sys/kernel/iommu_groups/8/devices/0000:00:14.1
/sys/kernel/iommu_groups/26/devices/0000:ff:05.3
/sys/kernel/iommu_groups/26/devices/0000:ff:05.1
/sys/kernel/iommu_groups/26/devices/0000:ff:05.2
/sys/kernel/iommu_groups/26/devices/0000:ff:05.0
/sys/kernel/iommu_groups/16/devices/0000:fe:00.0
/sys/kernel/iommu_groups/16/devices/0000:fe:00.1
/sys/kernel/iommu_groups/6/devices/0000:00:07.0
/sys/kernel/iommu_groups/24/devices/0000:ff:03.4
/sys/kernel/iommu_groups/24/devices/0000:ff:03.2
/sys/kernel/iommu_groups/24/devices/0000:ff:03.0
/sys/kernel/iommu_groups/24/devices/0000:ff:03.1
/sys/kernel/iommu_groups/14/devices/0000:02:00.1
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:05.0
/sys/kernel/iommu_groups/22/devices/0000:ff:00.0
/sys/kernel/iommu_groups/22/devices/0000:ff:00.1
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/2/devices/0000:00:03.0
/sys/kernel/iommu_groups/20/devices/0000:fe:05.3
/sys/kernel/iommu_groups/20/devices/0000:fe:05.1
/sys/kernel/iommu_groups/20/devices/0000:fe:05.2
/sys/kernel/iommu_groups/20/devices/0000:fe:05.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.7
/sys/kernel/iommu_groups/10/devices/0000:00:1d.1
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:fe:04.2
/sys/kernel/iommu_groups/19/devices/0000:fe:04.0
/sys/kernel/iommu_groups/19/devices/0000:fe:04.3
/sys/kernel/iommu_groups/19/devices/0000:fe:04.1
/sys/kernel/iommu_groups/9/devices/0000:00:1a.0
/sys/kernel/iommu_groups/9/devices/0000:00:1a.7
/sys/kernel/iommu_groups/9/devices/0000:00:1a.1
/sys/kernel/iommu_groups/27/devices/0000:ff:06.2
/sys/kernel/iommu_groups/27/devices/0000:ff:06.0
/sys/kernel/iommu_groups/27/devices/0000:ff:06.3
/sys/kernel/iommu_groups/27/devices/0000:ff:06.1

I've tried to add a single NIC passthrough to Win2016 VM. Unfortunately, after modifying my
/etc/pve/qemu-server/<vmid>.conf file I cannot start the VM anymore...
Code: 
boot: cd
bootdisk: scsi0
cores: 2
hostpci0: 02:00.1,pcie=1
ide0: local:iso/virtio-win-0.1.141.iso,media=cdrom,size=309208K
ide2: local:iso/SW_DVD9_Win_Server_STD_CORE_2016_64Bit_blah_blah.ISO,media=cdrom
machine: q35
memory: 8132
name: Win2016
net0: virtio=62:3D:0E:DA:01:34,bridge=vmbr0
numa: 0
ostype: win10
scsi0: local-lvm:vm-101-disk-1,discard=on,size=60G
scsihw: virtio-scsi-pci
smbios1: uuid=a729f0f0-faa2-4167-9347-ea826d1ba735
sockets: 1

Startup errors shown below:
Code: 
kvm: -device vfio-pci,host=02:00.1,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio error: 0000:02:00.1: failed to setup container for group 14: failed to set iommu for container: Operation not permitted

So... If anyone has a suggestion, I'd be grateful... thx!
 
Hi,
you can only passthrough the hole iommu_group and not only one device of it.
iommu_group 14 include 2 nics and only one.
jaceqp said:
02:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 02:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Click to expand...
 
Good call (I guess)...
I've adjusted VM.conf file with:
Code: 
machine: q35
hostpci0: 02:00,pcie=1
Still same error, but then - adding:
Code: 
 [root@TEST ~]# cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts
Y
...allowed to successfully launch Win2016 VM... I saw two new NIC adapters on hardware list, however short after plugging ethernet cable to one of them the warning LED started to blink on servers self-diagnostics, and LCD panel showed some "PCIe fatal error...".
I guess I have to revert those recent changes - unsafe interrupts sure are.... unsafe in this scenario. Basically I couldn't remote access the VM host and VM guests anymore.

I'm thinking if removing those ports from Container ethernet list would solve the issue...

BTW: 02:00.0 and 02:00.1 NIC devices are obviously unused - vmbr0 natively created by PROXMOX is currently working on 01:00.0 NIC.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom