Persistent PCI addresses

Hi. Where can I learn more about how Proxmox keeps PCI addresses stable between reboots?

I believe Libvirt keeps PCI device addressing stable by keeping details in their XML config files. This is also know as 'Stable machine ABIs'.

I've seen some forum posts about using the Proxmox args: configuration to pass -device settings to the VM in order to change the PCI address of a device.

I'm most interested to know if it's possible to start a Proxmox VM where the earlier PCI address mappings are reset or discarded.

I'd like to change the order of PCI addresses in a VM so the ethernet interface names change.

Thanks.
 
do you mean the pci addresses inside the guests? if yes, the devices have a static map of type <-> pci address, e.g the vga device will always have the same pci address
same for virtio disks, sata controllers, etc.
 
Hi, yes the PCI addresses in the guest - and how they get mapped to PCI addresses in the host.

For example:

Intel x710 NIC has 4 ports - each gets a PCI address in the host and each get mapped to PCI address in the guest.

Now you go and add SR-IOV interfaces on the host, which introduces new PCI addresses in the guest.

I attached two screenshots of different guest VMs with Intel NIC ports.

If your theory is that it's always determanistic based on hardware, why the difference in the way the 'Virtual Function' PCI addresses are assigned?

server 1, lspci in a guest

Screenshot 2022-03-10 at 14.40.53.jpg


server 2, lspci in a guest

Screenshot 2022-03-10 at 14.42.27.jpg
 
can you post both vm configs? (qm config ID)
 
ok from your output i can see that
hostpci0 -> 03:00.0
hostpci1 -> 04:00.0
hostpci2 -> 05:00.0
hostpci3 -> 06:00.0
hostpci4 -> 01:00.0
hostpci5 -> 02:00.0

on both hosts...
that seems weird though (i'd expect hostpci0 to be 01:00.0 and so on)

can you post the following info of both hosts:
Code:
pveversion -v
cat /cat /usr/share/qemu-server/pve-q35-4.0.cfg

also the output of this command for both vms:
Code:
qm showcmd ID --pretty
?

edit: after a bit of trying it out, i can see the following:
the pci adresses are static, but the linux kernel/ovmf does enumerate them in the order of appearances, so the hostpci4 and hostpci5 (and the ones after that) get put before the ones from hostpci0-4 so they get the lower numbers in the guest
but from boot to boot of the same vm with the same config, the enumeration does not change, so they are 'persistent'
 
Last edited:
host1
proxmox-ve: 7.1-1 (running kernel: 5.13.19-3-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)

host2
proxmox-ve: 6.4-1 (running kernel: 5.4.162-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)

attached the showcmd output as files
 

Attachments

  • guest1.txt
    2.8 KB · Views: 1
  • guest2.txt
    2.7 KB · Views: 1
Hi

from boot to boot of the same vm with the same config, the enumeration does not change, so they are 'persistent'

Yeah, I think that's by design, I've read that Microsoft Windows requires this to be static from boot to boot.

I've read a bit about how libvirt does this with their XML config file system.

I'm wondering how Proxmox does this? Where is this 'database' kept?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!