ATTO ExpressSAS H680 – PCI Passthrough
- Thread starter saltarob
- Start date
Check your IOMMU groups. Devices in the same IOMMU groups cannot be (safely and securely) shared between VMs and/or the Proxmox host. Maybe the Proxmox host lost its network and disks? Move the PCI(e) device to another PCI(e) slot to see if it moves into a different group.
Thank you so much for your quick reply. Looks like everything is fine with the IOMMU groups, isn't it?
I had to shorten the table because it exceeded the max character limit. But each device has it's unique group number.
Rich (BB code):
root@pvemacpro:~# pvesh get /nodes/pvemacpro/hardware/pci --pci-class-blacklist ""
┌──────────┬────────┬──────────────┬────────────┬────────┬─────────────────────────────────────────────────────────────────┬──────┬──────────────────┬
│ class │ device │ id │ iommugroup │ vendor │ device_name │ mdev │ subsystem_device │
╞══════════╪════════╪══════════════╪════════════╪════════╪═════════════════════════════════════════════════════════════════╪══════╪══════════════════╪
│ 0x010601 │ 0x3a22 │ 0000:00:1f.2 │ 44 │ 0x8086 │ 82801JI (ICH10 Family) SATA AHCI Controller │ │ 0x7270 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x010700 │ 0x0042 │ 0000:04:00.0 │ 49 │ 0x117c │ ExpressSAS 6Gb/s SAS/SATA HBA │ │ 0x0042 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x010802 │ 0xa804 │ 0000:03:00.0 │ 48 │ 0x144d │ NVMe SSD Controller SM961/PM961/SM963 │ │ 0xa801 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x020000 │ 0x10f6 │ 0000:0b:00.0 │ 55 │ 0x8086 │ 82574L Gigabit Network Connection │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x020000 │ 0x10f6 │ 0000:0c:00.0 │ 56 │ 0x8086 │ 82574L Gigabit Network Connection │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x028000 │ 0x432b │ 0000:0f:00.0 │ 58 │ 0x14e4 │ BCM4322 802.11a/b/g/n Wireless LAN Controller │ │ 0x008e │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x030000 │ 0x731f │ 0000:07:00.0 │ 52 │ 0x1002 │ Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] │ │ 0x0b36 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x040000 │ 0xa139 │ 0000:08:00.0 │ 54 │ 0xbdbd │ Intensity Pro 4K │ │ 0xa139 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x040300 │ 0x3a3e │ 0000:00:1b.0 │ 35 │ 0x8086 │ 82801JI (ICH10 Family) HD Audio Controller │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x040300 │ 0xab38 │ 0000:07:00.1 │ 53 │ 0x1002 │ Navi 10 HDMI Audio │ │ 0xab38 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x3406 │ 0000:00:00.0 │ 0 │ 0x8086 │ 5520 I/O Hub to ESI Port │ │ 0x7270 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x343a │ 0000:00:0d.0 │ 4 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x343b │ 0000:00:0d.1 │ 5 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x343c │ 0000:00:0d.2 │ 6 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x343d │ 0000:00:0d.3 │ 7 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x3418 │ 0000:00:0d.4 │ 8 │ 0x8086 │ 7500/5520/5500/X58 Physical Layer Port 0 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x3419 │ 0000:00:0d.5 │ 9 │ 0x8086 │ 7500/5520/5500 Physical Layer Port 1 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341a │ 0000:00:0d.6 │ 10 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341b │ 0000:00:0d.7 │ 11 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341c │ 0000:00:0e.0 │ 12 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341d │ 0000:00:0e.1 │ 13 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341e │ 0000:00:0e.2 │ 14 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341f │ 0000:00:0e.3 │ 15 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x3439 │ 0000:00:0e.4 │ 16 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060100 │ 0x3a18 │ 0000:00:1f.0 │ 44 │ 0x8086 │ 82801JIB (ICH10) LPC Interface Controller │ │ 0x7270 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3408 │ 0000:00:01.0 │ 1 │ 0x8086 │ 5520/5500/X58 I/O Hub PCI Express Root Port 1 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x340a │ 0000:00:03.0 │ 2 │ 0x8086 │ 5520/5500/X58 I/O Hub PCI Express Root Port 3 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x340e │ 0000:00:07.0 │ 3 │ 0x8086 │ 5520/5500/X58 I/O Hub PCI Express Root Port 7 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a40 │ 0000:00:1c.0 │ 36 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Root Port 1 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a42 │ 0000:00:1c.1 │ 37 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Port 2 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a44 │ 0000:00:1c.2 │ 38 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Root Port 3 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a46 │ 0000:00:1c.3 │ 39 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Root Port 4 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a48 │ 0000:00:1c.4 │ 40 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Root Port 5 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a4a │ 0000:00:1c.5 │ 41 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Root Port 6 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x8061 │ 0000:01:00.0 │ 45 │ 0x111d │ PES12T3G2 PCI Express Gen2 Switch │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x8061 │ 0000:02:02.0 │ 46 │ 0x111d │ PES12T3G2 PCI Express Gen2 Switch │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x8061 │ 0000:02:04.0 │ 47 │ 0x111d │ PES12T3G2 PCI Express Gen2 Switch │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x1478 │ 0000:05:00.0 │ 50 │ 0x1002 │ Navi 10 XL Upstream Port of PCI Express Switch │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x1479 │ 0000:06:00.0 │ 51 │ 0x1002 │ Navi 10 XL Downstream Port of PCI Express Switch │ │ 0x1479 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x823e │ 0000:0d:00.0 │ 57 │ 0x104c │ XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060401 │ 0x244e │ 0000:00:1e.0 │ 43 │ 0x8086 │ 82801 PCI Bridge │ │ 0x7270 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080000 │ 0x3425 │ 0000:00:10.0 │ 18 │ 0x8086 │ 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080000 │ 0x3426 │ 0000:00:10.1 │ 18 │ 0x8086 │ 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 │ │ 0x0000 │
root@pvemacpro:~#
Last edited:
What is the output of
cat /proc/cmdline? If you are using pcie_acs_override, please be aware that it does not actually change the groups, it only lies about them.If not, then maybe the device does not reset properly and takes the whole PCIe bus and the Proxmox host with it when it crashes? Do you know of anyone who had success with that device and passthrough?
Code:
root@pvemacpro:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.2.16-8-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init pcie_acs_override=downstream,multifunction pci=nommconf
root@pvemacpro:~#Oh it's there:
Code:
pcie_acs_overrideNo I am new to this topic and I couldn't find anything in my internet and forum search.
Last edited:
Remove
pcie_acs_override=downstream,multifunction, make the change active and you'll be able to see the real IOMMU groups.Try putting the device in another PCI(e) slot to see if it moves to an IOMMU group that is not shared with other essential devices.
If this not a PCIe but a PCI device, then chances that it will work are slim, in my experience.
Unfortunately the ExpressSAS shares the IOMMU group with 5 other devices. Will put it in another PCI slot now.
The card should be PCIe: https://www.atto.com/software/files/techpdfs/TechSheet_ESASH608.680.H644.pdf
The card should be PCIe: https://www.atto.com/software/files/techpdfs/TechSheet_ESASH608.680.H644.pdf
Rich (BB code):
root@pvemacpro:~# pvesh get /nodes/pvemacpro/hardware/pci --pci-class-blacklist ""
┌──────────┬────────┬──────────────┬────────────┬────────┬─────────────────────────────────────────────────────────────────┬──────┬──────────────────┬
│ class │ device │ id │ iommugroup │ vendor │ device_name │ mdev │ subsystem_device │
╞══════════╪════════╪══════════════╪════════════╪════════╪═════════════════════════════════════════════════════════════════╪══════╪══════════════════╪
│ 0x010601 │ 0x3a22 │ 0000:00:1f.2 │ 39 │ 0x8086 │ 82801JI (ICH10 Family) SATA AHCI Controller │ │ 0x7270 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x010700 │ 0x0042 │ 0000:04:00.0 │ 1 │ 0x117c │ ExpressSAS 6Gb/s SAS/SATA HBA │ │ 0x0042 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x010802 │ 0xa804 │ 0000:03:00.0 │ 1 │ 0x144d │ NVMe SSD Controller SM961/PM961/SM963 │ │ 0xa801 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x020000 │ 0x10f6 │ 0000:0b:00.0 │ 36 │ 0x8086 │ 82574L Gigabit Network Connection │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x020000 │ 0x10f6 │ 0000:0c:00.0 │ 36 │ 0x8086 │ 82574L Gigabit Network Connection │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x028000 │ 0x432b │ 0000:0f:00.0 │ 36 │ 0x14e4 │ BCM4322 802.11a/b/g/n Wireless LAN Controller │ │ 0x008e │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x030000 │ 0x731f │ 0000:07:00.0 │ 2 │ 0x1002 │ Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] │ │ 0x0b36 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x040000 │ 0xa139 │ 0000:08:00.0 │ 3 │ 0xbdbd │ Intensity Pro 4K │ │ 0xa139 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x040300 │ 0x3a3e │ 0000:00:1b.0 │ 35 │ 0x8086 │ 82801JI (ICH10 Family) HD Audio Controller │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x040300 │ 0xab38 │ 0000:07:00.1 │ 2 │ 0x1002 │ Navi 10 HDMI Audio │ │ 0xab38 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x3406 │ 0000:00:00.0 │ 0 │ 0x8086 │ 5520 I/O Hub to ESI Port │ │ 0x7270 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x343a │ 0000:00:0d.0 │ 4 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x343b │ 0000:00:0d.1 │ 5 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x343c │ 0000:00:0d.2 │ 6 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x343d │ 0000:00:0d.3 │ 7 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x3418 │ 0000:00:0d.4 │ 8 │ 0x8086 │ 7500/5520/5500/X58 Physical Layer Port 0 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x3419 │ 0000:00:0d.5 │ 9 │ 0x8086 │ 7500/5520/5500 Physical Layer Port 1 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341a │ 0000:00:0d.6 │ 10 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341b │ 0000:00:0d.7 │ 11 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341c │ 0000:00:0e.0 │ 12 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341d │ 0000:00:0e.1 │ 13 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341e │ 0000:00:0e.2 │ 14 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x341f │ 0000:00:0e.3 │ 15 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060000 │ 0x3439 │ 0000:00:0e.4 │ 16 │ 0x8086 │ │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060100 │ 0x3a18 │ 0000:00:1f.0 │ 39 │ 0x8086 │ 82801JIB (ICH10) LPC Interface Controller │ │ 0x7270 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3408 │ 0000:00:01.0 │ 1 │ 0x8086 │ 5520/5500/X58 I/O Hub PCI Express Root Port 1 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x340a │ 0000:00:03.0 │ 2 │ 0x8086 │ 5520/5500/X58 I/O Hub PCI Express Root Port 3 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x340e │ 0000:00:07.0 │ 3 │ 0x8086 │ 5520/5500/X58 I/O Hub PCI Express Root Port 7 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a40 │ 0000:00:1c.0 │ 36 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Root Port 1 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a42 │ 0000:00:1c.1 │ 36 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Port 2 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a44 │ 0000:00:1c.2 │ 36 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Root Port 3 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a46 │ 0000:00:1c.3 │ 36 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Root Port 4 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a48 │ 0000:00:1c.4 │ 36 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Root Port 5 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x3a4a │ 0000:00:1c.5 │ 36 │ 0x8086 │ 82801JI (ICH10 Family) PCI Express Root Port 6 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x8061 │ 0000:01:00.0 │ 1 │ 0x111d │ PES12T3G2 PCI Express Gen2 Switch │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x8061 │ 0000:02:02.0 │ 1 │ 0x111d │ PES12T3G2 PCI Express Gen2 Switch │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x8061 │ 0000:02:04.0 │ 1 │ 0x111d │ PES12T3G2 PCI Express Gen2 Switch │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x1478 │ 0000:05:00.0 │ 2 │ 0x1002 │ Navi 10 XL Upstream Port of PCI Express Switch │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x1479 │ 0000:06:00.0 │ 2 │ 0x1002 │ Navi 10 XL Downstream Port of PCI Express Switch │ │ 0x1479 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060400 │ 0x823e │ 0000:0d:00.0 │ 36 │ 0x104c │ XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x060401 │ 0x244e │ 0000:00:1e.0 │ 38 │ 0x8086 │ 82801 PCI Bridge │ │ 0x7270 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080000 │ 0x3425 │ 0000:00:10.0 │ 18 │ 0x8086 │ 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080000 │ 0x3426 │ 0000:00:10.1 │ 18 │ 0x8086 │ 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080000 │ 0x3427 │ 0000:00:11.0 │ 19 │ 0x8086 │ 7500/5520/5500 Physical and Link Layer Registers Port 1 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080000 │ 0x3428 │ 0000:00:11.1 │ 19 │ 0x8086 │ 7500/5520/5500 Routing & Protocol Layer Register Port 1 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080000 │ 0x342e │ 0000:00:14.0 │ 21 │ 0x8086 │ 7500/5520/5500/X58 I/O Hub System Management Registers │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080000 │ 0x3422 │ 0000:00:14.1 │ 22 │ 0x8086 │ 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080000 │ 0x3423 │ 0000:00:14.2 │ 23 │ 0x8086 │ 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080000 │ 0x3438 │ 0000:00:14.3 │ 24 │ 0x8086 │ 7500/5520/5500/X58 I/O Hub Throttle Registers │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼──────────────────┼
│ 0x080020 │ 0x342d │ 0000:00:13.0 │ 20 │ 0x8086 │ 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────────────┼──────┼───────
root@pvemacpro:~#Okay in the new PCI slot the ExpressSAS shares the IOMMU group only with one other device called
5520/5500/X58 I/O Hub PCI Express Root Port 7. I can't enable ACS in my BIOS because it's an Apple Mac Pro 5,1. Or maybe there is a way but I don't know how.
So do I have to apply the
pcie_acs_override=downstream,multifunction again before I start a VM?Thank you.
That's probably fine. There are often PCI(e) Bridges and Root Ports in IOMMU groups and they can be ignored. You also don't need to bind them to vfio-pci because they are a normal part of the PCIe infrastructure and handled automatic.
It's probably enabled by default, otherwise you would not have groups.
I would try without it as it is a last resort and introduced a huge security/isolation issue. And I don't think you need it as explained above.
Hmm, I could boot the VM once. But the ExpressSAS didn't show up in the hardware list of the VM. I installed the drivers and rebooted the VM and Proxmox crashed. After I started Poxmox again, I tried to start the VM and Proxmox crashed again.
What else can I try? Thank you.
That not showing up in the VM problem happened also with another PCIe card. How can that happen?
And it does not separate the devices in unique groups?
What else can I try? Thank you.
That not showing up in the VM problem happened also with another PCIe card. How can that happen?
Sounds like passthrough works in principle, but the device does not reset properly and/or does not work with passthrough at all. Because it is real hardware, it can take the whole machine down via the PCIe bus.
Sometimes devices don't reset properly and don't work with passthrough or not more than once per host reboot. You can try early binding to vfio-pci to prevent Proxmox from touching the device before the VM starts. You might also need a softdep to make sure vfio-pci is loaded (and takes the device) before the actual Linux driver loads.
PCIe passthrough is hit or miss and sometimes requires work-arounds. Manufacturers don't design and test for such usage. Best to buy hardware that is known to work well with passthrough.
Thank you. Will try this. With the early binding does it still matter in which IOMMU group the device is?
I am just starting and it's a test run. So I try it with the hardware which is available. How can I find hardware which is known to work well? Is there a list somewhere or do I have to search through forums?
Thank you.
Is it correct like this? I don't know where to find the name for the softdep. I used "atto"?
Code:
root@pvemacpro:~# lspci -nn | grep ExpressSAS
04:00.0 Serial Attached SCSI controller [0107]: ATTO Technology, Inc. ExpressSAS 6Gb/s SAS/SATA HBA [117c:0042]
root@pvemacpro:~#
Code:
echo "options vfio-pci ids=117c:0042" >> /etc/modprobe.d/vfio.conf
Code:
echo "softdep atto pre: vfio-pci" >> /etc/modprobe.d/vfio.confYes it still matters very much. Devices in the same IOMMU cannot be safely/securely shared between VMs and/or the Proxmox host. IOMMU groups are determined by the physical motherboard PCIe layout (and BIOS). Early binding to vfio-pci is about chosing a driver for the device, which is completely separate.
I wish it was easier, but search this forum and the internet in general. Level1techs also has a good forum on VFIO.
Use
lspci -ks 04:00 to see the driver in use and kernel drivers that would otherwise be used.I cannot say if "atto" is correct. You can edit the file with
nano /etc/modprobe.d/vfio.conf, or any other editor you prefer.Thank you so much.
So pm80xx should be the driver name, right? So I chanced the vfio.conf to:
updated the initramfs with
I used the
And still crashing...
So looks like I am out of luck and options, right?
That means this card is not working for passthrough or at least not in this computer?
Code:
root@pvemacpro:~# lspci -ks 04:00
04:00.0 Serial Attached SCSI controller: ATTO Technology, Inc. ExpressSAS 6Gb/s SAS/SATA HBA
Subsystem: ATTO Technology, Inc. ExpressSAS H680
Kernel driver in use: pm80xx
Kernel modules: pm80xx
root@pvemacpro:~#So pm80xx should be the driver name, right? So I chanced the vfio.conf to:
Code:
softdep pm80xx pre: vfio-pciupdated the initramfs with
update-initramfs -u -k all and rebooted.I used the
lspci -nnk to verify the configuration:
Code:
04:00.0 Serial Attached SCSI controller [0107]: ATTO Technology, Inc. ExpressSAS 6Gb/s SAS/SATA HBA [117c:0042]
Subsystem: ATTO Technology, Inc. ExpressSAS H680 [117c:0042]
Kernel driver in use: vfio-pci
Kernel modules: pm80xxKernel driver in use: vfio-pci So ready for passthrough.And still crashing...
So looks like I am out of luck and options, right?
That means this card is not working for passthrough or at least not in this computer?
Last edited:
Maybe you can do more in the motherboard BIOS to make sure not to initialize the device looking for boot drives maybe?
Maybe dump the device ROM and add it to the VM configuration?
Maybe find someone one the internet that knows a work-around?
Maybe study the open-source driver and see if you can improve it to reset properly?
The fact that it crashes (the host) does not bode well. The things above usually help when the device is passed through but does not function correctly.
What is the output of
cat '/sys/bus/pci/devices/0000:04:00.0/reset_method'? Maybe there are multiple reset methods? You can try each one by writing it to that same file, like echo bus >'/sys/bus/pci/devices/0000:04:00.0/reset_method' before starting the VM.Thank you so much for your efforts.
Only bus. So I wrote the bus reset before starting the VM:
But Proxmox crashed.
It's a Mac Pro 5,1 with the Apple EFI. Therefore changing BIOS settings is not possible. I don't know if OpenCore could influence it.
Will try it.
ATTO has a Linux driver for that ExpressSAS card. Would it make sense to install that driver?
Code:
root@pvemacpro:~# cat '/sys/bus/pci/devices/0000:04:00.0/reset_method'
bus
root@pvemacpro:~#Only bus. So I wrote the bus reset before starting the VM:
echo bus >'/sys/bus/pci/devices/0000:04:00.0/reset_method'But Proxmox crashed.
So next the device ROM dump:
Do you know why that
Code:
root@pvemacpro:~# cd /sys/bus/pci/devices/0000:04:00.0/
root@pvemacpro:/sys/bus/pci/devices/0000:04:00.0# echo 1 > rom
root@pvemacpro:/sys/bus/pci/devices/0000:04:00.0# cat rom > /usr/share/kvm/vbios.bin
cat: rom: Input/output error
root@pvemacpro:/sys/bus/pci/devices/0000:04:00.0#Do you know why that
Input/output error shows up?
Uncommon hardware might make it harder to apply any work-around you might find from others.
Proprietary drivers are often not (quickly or regularly) updated for new kernel versions and generally you don't want a driver touching the device. So I would guess not. It could make some kind of difference, but the fact that the whole system crashes (instead of the device not functioning inside the VM) does not give me hope.
That said, maybe ATTO support can help you get the device working with passthrough or explain why it might never work.
Then
bus is also the default and used each time before, and therefore you get the exact same result as before.Sometimes reading the device ROM does not work, possibly because the device was already initialized by the motherboard BIOS or a driver.
I don't really see an approach that might yield any positive result, sorry. Try the device on another motherboard or just use it on the Proxmox host and not in a VM.
