PCIe Bifurcation and PCIe address change

[silverstone]

I successfully managed to setup PCIe Bifurcation on my Supermicro X9DRi-LN4F+, since I am using a NVME Expander (ASUS Hyper M.2 X16 V2) for 2 x NVME drives in striped configuration (planning to destroy & recreate as mirror for safety reasons !). The NVME Expander is *not* passed through any guest, but it used by the Proxmox VE host directly.

However I am finding some problems to let PCIe passthrough since this is a virtualized Desktop system and, trying to pass through NVIDIA GPUs / USB 3.0 controllers, just don't work because at every boot the PCI address is different: sometimes it's 07:00, sometimes it's 09:00, sometimes even 0e:01, all for the same card. Therefore the guest VM refuses to boot every single each time, since (at least one) PCIe device is not available.

Current version with all updates applied:

pveversion -v

proxmox-ve: 6.2-1 (running kernel: 5.4.44-1-pve)
pve-manager: 6.2-6 (running version: 6.2-6/ee1d7754)
pve-kernel-5.4: 6.2-3
pve-kernel-helper: 6.2-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-3
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-7
pve-cluster: 6.1-8
pve-container: 3.1-8
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-3
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

Is there a kernel command line to tell Linux to keep consistent PCIe addresses? I believe the problem is that, probably due to a delay, the 2 unused 4x slots provided by the expander, sometimes are detected and sometimes are not, causing all successive PCIe addresses to "shift down".

Let me know if I can provide some additional information.

Thank you for your help.

 

[Stefan_R]

I'm fairly certain that you cannot force PCIe addresses, and you cannot change them after boot. The easiest solution would probably be to run a script at startup that changes your VM configuration for you.

As a starting point, you can configure your VMs to use different addresses like so:

qm set <vmid> -hostpciX 00:YY.Z,pcie=1,rombar=1

(the pcie/rombar/... paramters are optional of course, but will be removed from the config if not set in the command)

Extract the addresses via 'lspci' and insert them into the command up top. Then run it at boot, say by making an '@reboot' crontab entry.

 

[silverstone]

That's unfortunate.

I solved this the "easy" way by bifurcating the PCIe slot to x8x8 instead of x4x4x4x4.

This way, since 2 devices are installed (in M.2 slot #1 and #3, so that each falls in a separate x8 slot), no "empty" places are left. The "trick" appears to work fine across several reboots, although it's still too early to draw any conclusion.

Your approach would likely also work, just need to do sothing like this to get the PCI addresses (maybe there is an easier way):

lspci | grep NVIDIA | sed  -e 's#\(..\)\:\(..\).\(.\).*#\1:\2\.\3#g'

Then just compare the "Major" address and determine which device goes to which VM.