I've been using PCI passthrough on a Proxmox 3.4 host, and it has always worked flawlessly. Tried to upgrade to Proxmox 4.0, and there were numerous problems, so I rolled back and installed a fresh 4.0 separately to test.
Symptom 1: guests boot extremely slow
On 4.0, if a guest uses PCI passthrough, the boot process becomes very slow. A pfSense VM that boots under 2 minutes without passthrough. If passthrough is used, then it takes more than 3 minutes just to load the kernel. After kernel is loaded, the rest of the boot process seems to execute normally.
Similar symptoms occur if I pass through the SAS HBA card to my FreeNAS VM.
Symptom 2: IOMMU group includes multiple PCI devices
On 3.4 I have passed through one port on a Intel Pro/1000 VT quad port NIC to a FreeNAS VM and use the adjacent port on the host. On 4.0 when I pass through the same port, the adjacent port disappears from the host. Looks like it's because the two ports are in the same IOMMU group (there was no concept of IOMMU groups with kernel 2.6).
Discussions online seems to point to the 4.2 kernel for the slow boot issue (bbs.archlinux.org/viewtopic.php?id=203240). Running Debian's own 3.16 kernel on the new Proxmox 4.0 installation does indeed fix the slow boot, although the IOMMU group issue remains (the post linked above also refers to a patched kernel that fixes IOMMU grouping).
Code:
~# pveversion -v
proxmox-ve: 4.0-16 (running kernel: 4.2.2-1-pve)
pve-manager: 4.0-50 (running version: 4.0-50/d3a6b7e5)
pve-kernel-4.2.2-1-pve: 4.2.2-16
lvm2: 2.02.116-pve1
corosync-pve: 2.3.5-1
libqb0: 0.17.2-1
pve-cluster: 4.0-23
qemu-server: 4.0-31
pve-firmware: 1.1-7
libpve-common-perl: 4.0-32
libpve-access-control: 4.0-9
libpve-storage-perl: 4.0-27
pve-libspice-server1: 0.12.5-1
vncterm: 1.2-1
pve-qemu-kvm: 2.4-10
pve-container: 1.0-10
pve-firewall: 2.0-12
pve-ha-manager: 1.0-10
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.3-1
lxcfs: 0.9-pve2
cgmanager: 0.37-pve2
criu: 1.6.0-1
zfsutils: 0.6.5-pve4~jessieSymptom 1: guests boot extremely slow
On 4.0, if a guest uses PCI passthrough, the boot process becomes very slow. A pfSense VM that boots under 2 minutes without passthrough. If passthrough is used, then it takes more than 3 minutes just to load the kernel. After kernel is loaded, the rest of the boot process seems to execute normally.
Similar symptoms occur if I pass through the SAS HBA card to my FreeNAS VM.
Symptom 2: IOMMU group includes multiple PCI devices
On 3.4 I have passed through one port on a Intel Pro/1000 VT quad port NIC to a FreeNAS VM and use the adjacent port on the host. On 4.0 when I pass through the same port, the adjacent port disappears from the host. Looks like it's because the two ports are in the same IOMMU group (there was no concept of IOMMU groups with kernel 2.6).
Code:
# lspci
<snip>
05:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
06:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
06:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
07:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
07:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
08:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
08:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
# find /sys/kernel/iommu_groups/ -type l
<snip>
/sys/kernel/iommu_groups/13/devices/0000:05:00.0
/sys/kernel/iommu_groups/14/devices/0000:06:02.0
/sys/kernel/iommu_groups/14/devices/0000:07:00.0
/sys/kernel/iommu_groups/14/devices/0000:07:00.1
/sys/kernel/iommu_groups/15/devices/0000:06:04.0
/sys/kernel/iommu_groups/15/devices/0000:08:00.0
/sys/kernel/iommu_groups/15/devices/0000:08:00.1Discussions online seems to point to the 4.2 kernel for the slow boot issue (bbs.archlinux.org/viewtopic.php?id=203240). Running Debian's own 3.16 kernel on the new Proxmox 4.0 installation does indeed fix the slow boot, although the IOMMU group issue remains (the post linked above also refers to a patched kernel that fixes IOMMU grouping).
