Controlling PCI Slot order in Proxmox 5.4

[justing6]

I'm trying to solve a very specific problem to get a Palo Alto VM-series firewall online on Proxmox (not supported by Palo Alto but should work fine in KVM). The VM itself works fine, but I'm trying to give it three specific network interfaces: One virtio interface for the dedicated management interface (net0, registered as eth0 in the VM), and then a PCI-Passthrough'd Intel 82576 PCI Express card with two NICs.

This issue is the Palo Alto VM will always take the NIC with the lowest PCI-ID to be the mangement interface, and using hostpci0 to passthrough the card gives the two NICs on the card IDs of 00:01:00 and 00:01:00.1, while the virtio int has a PCI-ID of 00:06:12.

If I use hostpci3 instead of hostpci0, the Intel 82576 card's PCI-ID becomes 00:04:00 and 00:4.00.1, but is still lower than the virtio interface. I tried editing /usr/share/perl5/PVE/QemuServer/PCI.pm, /usr/share/perl5/PVE/QemuServer.pm , and /usr/share/qemu-server/pve-q35.cfg to allow for up to hostpci7 to be valid, which worked and put the PCI-ID at 00:08:00 and 00:08:00.1, but also pushed the virtio nic to a PCI-ID of 00:0a:12.

Is there a way I can manipulate some config files to raise the PCI-ID (slot number) of the passed-through Intel 82576 and/or lower the PCI-ID of the virtio net0? I obviously would prefer not to burn one of my two dedicated interfaces for management traffic.



Thank you in advance!

 

[dcsapak]

mhmm i see no easy way with our config.. what you could do is to use the 'args' part of the config to manually add a device with the desired address (see 'man qm.conf' and search for 'args')

 

[justing6]

Thank you, that sounds like exactly what I need! As this is a specific issue affecting only this one guest machine, editing the base config is not a requirement.

If I set my Intel 82576 to hostpci3 which gives it an ID of 00:04:00 and I can manually add a single virtio net0 at 00:02:00 for example it would fix my issue.

I looked into the man page for qm.conf and was poking around other forums and ran into a case where some people used a specific nic driver (e1000e) to virtualize synology on Proxmox, with their args looking like this:

args: -device ich9-usb-ehci1,id=usb,multifunction=on,bus=pci.0,addr=0xa -drive file=/var/lib/vz/images/103/vm-103-USB103b.raw,format=raw,if=none,id=drive-usb-disk2,cache=writeback -device usb-storage,bus=usb.0,port=2,drive=drive-usb-disk2,id=usb-disk2,bootindex=1,removable=off -netdev type=tap,id=net0,ifname=tap103i0 -device e1000e,mac=BA:E2:8A:3B:AC:3E,netdev=net0,bus=pci.0,addr=0x12,id=net0

Now if I want to add a virtio net instead of the e1000e in this example, I'm assuming I'll have to directly reference the image file of virtio, as well as possibly using ich9-pcie instead of ich0-usb (I'm not sure on this, I'll have to do more research) and finally manipulating 'bus' and 'addr' to suit my needs. If you have any free time some advice would be appreciated, but I'll keep searching around to see what I can find now that you've pointed me in the right direction!

 

[justing6]

After taking a second look at that and realizing most of it was other manually added devices that aren't relevant to me, I tried this args line:

args: -netdev type=tap,id=net0,ifname=tap103i0 -device virtio-net-pci,mac=BA:E2:8A:3B:AC:3E,netdev=net0,bus=pci.0,addr=0x2,id=net0

This created a virtio device with an address of 0000:0a:02.0, I'll keep messing with that address variable to see if I can change that and I still am not sure how to attach the tap interface to a vmbr, but I think I'm getting there.

 

[dcsapak]

yeah i guess you have to put it onto the pcie bus instead if you use q35 and pcie (bus=pcie.0)
but i guess in that case you could change the addresses in the q35 cfg and put the pcie root ports (used for hostpci in pcie mode,q35) after the pci bridges...

 

[justing6]

That did it! I made a change that didn't do exactly what I thought it would, but it worked nonetheless. I removed hostpci3 in the q35 cfg to open up its pci mapping (launching the VM failed when I left it in) and moved it into the -args file nic:

args: -netdev type=tap,id=net0,ifname=tap103i0 -device virtio-net-pci,mac=BA:E2:8A:3B:AC:3E,netdev=net0,bus=pcie.0,addr=1c.2,id=net0

The result of this was that the PA-VM doesn't recognize a PCI-ID for this interface at all, resulting in it being the "lowest" ID and becoming eth0 and therefore management. Here is the output from the firewall with two normal virtio net adapter (net1 and net2) added with the normal "net1: virtio=" method:

Interface_name       Base-OS_port       Base-OS_MAC             PCI-ID         Driver
mgt                     eth0          ba:e2:8a:3b:ac:3e                      virtio_net
Ethernet1/1             eth1          6c:b3:11:1b:e6:76       0000:07:00.0          igb
Ethernet1/2             eth2          6c:b3:11:1b:e6:77       0000:07:00.1          igb
Ethernet1/3             eth3          9a:56:e9:77:97:28       0000:09:13.0   virtio_net
Ethernet1/4             eth4          c6:ca:66:e2:9a:c5       0000:09:14.0   virtio_net

I'm not sure why that custom added nic automatically egresses through vmbr0 without any reference to vmbr0 on the command, but thankfully that's exactly what I wanted it to do anyway.

Thanks for the help!

 

[trch]

Unfortunately, the bridged virtio NIC that I'm attempting to use has the highest ID, rather than the lowest device ID. I'm new to proxmox and kvm, so this is rather foreign to me and I don't understand the syntax or even why everything listed on the "args" line is necessary.

Could anyone provide an example of an "args: " line that uses virtio NIC connected to a bridge, with another device passed through? What controls the PCI-ID that's assigned to devices within the VM? Is there another place that this can be changed? The proxmox docs say "for advanced users" but provides no further explanation.

An explanation of the values being used in the following would be helpful:

args: -netdev type=tap,id=net0,ifname=tap103i0 -device virtio-net-pci,mac=AA:BB:CC:DD:EE:FF,netdev=net0,bus=pcie.0,addr=1c.2,id=net0

 

[trch]

I figured out that args: simply passes parameters to kvm and did some searching.

First, I removed the net0: device definition from qm.conf. After reading some documentation, I decided to omit the bus= parameter from the device; this change causes it to attach to the first available bus. I incremented the device address starting from 00.0 until I didn't get an error that the ID was already in use while starting the VM. Based on what justing6 says about the NIC attaching to the bridge without a configuration, I don't know if br=vmbr0 actually does anything but the configuration works, so I decided to leave it in place for now.

Here are the relevant lines from qm.conf

args: -device virtio-net-pci,netdev=net0,mac=AA:BB:CC:DD:EE:FF,addr=02.0 -netdev tap,id=net0,br=vmbr0
hostpci0: 03:00,pcie=1

 

[flexyz]

Hi

I am also trying to get a PA-VM running on my Proxmox v7.x

If I just add one vmbr I can get management networking up and running. But when I add the SFP+ Nic with PCI-Passthroug things stop workring.

trch: Can I see your full qemu-server conf file?​

Thanks
Felix

 

[ZZer00]

Bit of a thread necro here, but I wanted to provide a really easy solution to this

You can just login and run a single command that will switch the mgmt interfaces

set system setting mgmt-interface-swap enable yes

https://docs.paloaltonetworks.com/v...-to-swap-the-management-interface-on-kvm.html