Persistent PCI addresses

[Joe Botha]

Hi. Where can I learn more about how Proxmox keeps PCI addresses stable between reboots?

I believe Libvirt keeps PCI device addressing stable by keeping details in their XML config files. This is also know as 'Stable machine ABIs'.

I've seen some forum posts about using the Proxmox args: configuration to pass -device settings to the VM in order to change the PCI address of a device.

I'm most interested to know if it's possible to start a Proxmox VM where the earlier PCI address mappings are reset or discarded.

I'd like to change the order of PCI addresses in a VM so the ethernet interface names change.

Thanks.

 

[dcsapak]

do you mean the pci addresses inside the guests? if yes, the devices have a static map of type <-> pci address, e.g the vga device will always have the same pci address
same for virtio disks, sata controllers, etc.

 

[Joe Botha]

Hi, yes the PCI addresses in the guest - and how they get mapped to PCI addresses in the host.

For example:

Intel x710 NIC has 4 ports - each gets a PCI address in the host and each get mapped to PCI address in the guest.

Now you go and add SR-IOV interfaces on the host, which introduces new PCI addresses in the guest.

I attached two screenshots of different guest VMs with Intel NIC ports.

If your theory is that it's always determanistic based on hardware, why the difference in the way the 'Virtual Function' PCI addresses are assigned?

server 1, lspci in a guest




server 2, lspci in a guest

 

[dcsapak]

can you post both vm configs? (qm config ID)

 

[Joe Botha]

Two screenshots, host at the top with qm config output,
bottom, the VM running on the host, with lspci output

How does the VM get the order of the PCI addresses of the ports?

Is there nothing inside the VM which stores the state of the PCI address layout?

 

[Joe Botha]

Maybe also useful, the layout of the PCI addresses on the two hosts.

I can't spot any logic about how host addresses get mapped to guest addresses - do you know?

 

[dcsapak]

ok from your output i can see that
hostpci0 -> 03:00.0
hostpci1 -> 04:00.0
hostpci2 -> 05:00.0
hostpci3 -> 06:00.0
hostpci4 -> 01:00.0
hostpci5 -> 02:00.0

on both hosts...
that seems weird though (i'd expect hostpci0 to be 01:00.0 and so on)

can you post the following info of both hosts:

pveversion -v
cat /cat /usr/share/qemu-server/pve-q35-4.0.cfg

also the output of this command for both vms:

qm showcmd ID --pretty

?

edit: after a bit of trying it out, i can see the following:
the pci adresses are static, but the linux kernel/ovmf does enumerate them in the order of appearances, so the hostpci4 and hostpci5 (and the ones after that) get put before the ones from hostpci0-4 so they get the lower numbers in the guest
but from boot to boot of the same vm with the same config, the enumeration does not change, so they are 'persistent'

 

[Joe Botha]

host1
proxmox-ve: 7.1-1 (running kernel: 5.13.19-3-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)

host2
proxmox-ve: 6.4-1 (running kernel: 5.4.162-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)

attached the showcmd output as files

 

[Joe Botha]

Hi

from boot to boot of the same vm with the same config, the enumeration does not change, so they are 'persistent'

Yeah, I think that's by design, I've read that Microsoft Windows requires this to be static from boot to boot.

I've read a bit about how libvirt does this with their XML config file system.

I'm wondering how Proxmox does this? Where is this 'database' kept?

 

[dcsapak]

I'm wondering how Proxmox does this? Where is this 'database' kept?

it's a static map:

for i440fx:
https://git.proxmox.com/?p=qemu-ser...39e8354913e25520767;hb=refs/heads/master#l127

and for q35 (some things get generated by code in a deterministic way):
https://git.proxmox.com/?p=qemu-ser...39e8354913e25520767;hb=refs/heads/master#l291