[SOLVED] PCIe Passthrough

[cpd1900]

I've been trying to get my new LSI9211-8i IT mode sas controller passed through to a VM and I've run into a ton of problems but I've got to a point where I've gone through PCI(e) passthrough documentation https://pve.proxmox.com/wiki/Pci_passthrough#Verify_Operation and this documentation that tells me how to blacklist the device https://pve.proxmox.com/wiki/PCI(e)_Passthrough.

I am able to have the VM boot when I have not added the card in the hardware tab of the VM but as soon as I add the card I get
kvm: -device vfio-pci,host=0000:04:00.0,id=hostpci0,bus=pci.0,addr=0x10,rombar=0: vfio 0000:04:00.0: failed to setup container for group 17: Failed to set iommu for container: Operation not permitted TASK ERROR: start failed: QEMU exited with code 1
as well as I lose access to all the drives in the disk manager of proxmox web GUI until I reboot the whole server. When I run lspci I get the device as
04:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03).

My configs for /etc/modprobe.d/pve-blacklist.conf are
# This file contains a list of modules which are not supported by Proxmox VE # nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701 blacklist nvidiafb options vfio-pci ids=1000,0072

I would love any help anyone can provide
Thanks,
Chris

 

[leesteken]

Can you please check journalctl -b 0 (or the Syslog or dmesg) for the message DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement?
Your error message does come up on this forum for servers which BIOS prevents passthrough. Can you tell us the make and model of your motherboard?

 

[cpd1900]

Sure! Thanks for the help and sorry about the late reply.

Its in a gen II dell r710.

I ran the VM without the card passed through then card passed through then I ran journalctl -b 0 and this is the output
Jul 01 16:36:52 Test pvedaemon[6697]: <root@pam> successful auth for user 'root@pam' Jul 01 16:39:11 Test pvedaemon[6696]: <root@pam> starting task UPID:Test:0000AB9B:001AEE04:62BF5B6F:vncshell::root@pam: Jul 01 16:39:11 Test pvedaemon[43931]: starting termproxy UPID:Test:0000AB9B:001AEE04:62BF5B6F:vncshell::root@pam: Jul 01 16:39:11 Test pvedaemon[6698]: <root@pam> successful auth for user 'root@pam' Jul 01 16:39:11 Test login[43936]: pam_unix(login:session): session opened for user root(uid=0) by (uid=0) Jul 01 16:39:11 Test systemd-logind[885]: New session 8 of user root. Jul 01 16:39:11 Test systemd[1]: Started Session 8 of user root. Jul 01 16:39:11 Test login[43941]: ROOT LOGIN on '/dev/pts/1' Jul 01 16:40:39 Test systemd-logind[885]: Session 8 logged out. Waiting for processes to exit. Jul 01 16:40:39 Test systemd[1]: session-8.scope: Succeeded. Jul 01 16:40:39 Test systemd-logind[885]: Removed session 8. Jul 01 16:40:39 Test pvedaemon[6696]: <root@pam> end task UPID:Test:0000AB9B:001AEE04:62BF5B6F:vncshell::root@pam: OK Jul 01 16:40:44 Test pvedaemon[44150]: starting termproxy UPID:Test:0000AC76:001B1233:62BF5BCC:vncshell::root@pam: Jul 01 16:40:44 Test pvedaemon[6698]: <root@pam> starting task UPID:Test:0000AC76:001B1233:62BF5BCC:vncshell::root@pam: Jul 01 16:40:44 Test pvedaemon[6698]: <root@pam> successful auth for user 'root@pam' Jul 01 16:40:44 Test login[44170]: pam_unix(login:session): session opened for user root(uid=0) by root(uid=0) Jul 01 16:40:44 Test systemd-logind[885]: New session 9 of user root. Jul 01 16:40:44 Test systemd[1]: Started Session 9 of user root. Jul 01 16:40:44 Test login[44177]: ROOT LOGIN on '/dev/pts/1' Jul 01 16:40:56 Test pvedaemon[44207]: start VM 100: UPID:Test:0000ACAF:001B16F1:62BF5BD8:qmstart:100:root@pam: Jul 01 16:40:56 Test pvedaemon[6698]: <root@pam> starting task UPID:Test:0000ACAF:001B16F1:62BF5BD8:qmstart:100:root@pam: Jul 01 16:40:56 Test systemd[1]: Created slice qemu.slice. Jul 01 16:40:56 Test systemd[1]: Started 100.scope. Jul 01 16:40:56 Test systemd-udevd[44224]: Using default interface naming scheme 'v247'. Jul 01 16:40:56 Test systemd-udevd[44224]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jul 01 16:40:57 Test kernel: device tap100i0 entered promiscuous mode Jul 01 16:40:57 Test systemd-udevd[44223]: Using default interface naming scheme 'v247'. Jul 01 16:40:57 Test systemd-udevd[44223]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jul 01 16:40:57 Test systemd-udevd[44224]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jul 01 16:40:57 Test systemd-udevd[44223]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jul 01 16:40:57 Test kernel: vmbr0: port 2(fwpr100p0) entered blocking state Jul 01 16:40:57 Test kernel: vmbr0: port 2(fwpr100p0) entered disabled state Jul 01 16:40:57 Test kernel: device fwpr100p0 entered promiscuous mode Jul 01 16:40:57 Test kernel: vmbr0: port 2(fwpr100p0) entered blocking state Jul 01 16:40:57 Test kernel: vmbr0: port 2(fwpr100p0) entered forwarding state Jul 01 16:40:57 Test kernel: fwbr100i0: port 1(fwln100i0) entered blocking state Jul 01 16:40:57 Test kernel: fwbr100i0: port 1(fwln100i0) entered disabled state Jul 01 16:40:57 Test kernel: device fwln100i0 entered promiscuous mode Jul 01 16:40:57 Test kernel: fwbr100i0: port 1(fwln100i0) entered blocking state Jul 01 16:40:57 Test kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state Jul 01 16:40:57 Test kernel: fwbr100i0: port 2(tap100i0) entered blocking state Jul 01 16:40:57 Test kernel: fwbr100i0: port 2(tap100i0) entered disabled state Jul 01 16:40:57 Test kernel: fwbr100i0: port 2(tap100i0) entered blocking state Jul 01 16:40:57 Test kernel: fwbr100i0: port 2(tap100i0) entered forwarding state Jul 01 16:40:57 Test pvedaemon[6698]: <root@pam> end task UPID:Test:0000ACAF:001B16F1:62BF5BD8:qmstart:100:root@pam: OK Jul 01 16:41:26 Test pvedaemon[6696]: <root@pam> starting task UPID:Test:0000AD79:001B22C7:62BF5BF6:vncproxy:100:root@pam: Jul 01 16:41:26 Test pvedaemon[44409]: starting vnc proxy UPID:Test:0000AD79:001B22C7:62BF5BF6:vncproxy:100:root@pam: Jul 01 16:41:53 Test kernel: perf: interrupt took too long (2507 > 2500), lowering kernel.perf_event_max_sample_rate to 79750 Jul 01 16:42:14 Test pvedaemon[6696]: <root@pam> end task UPID:Test:0000AD79:001B22C7:62BF5BF6:vncproxy:100:root@pam: OK Jul 01 16:42:18 Test pvedaemon[44546]: shutdown VM 100: UPID:Test:0000AE02:001B3711:62BF5C2A:qmshutdown:100:root@pam: Jul 01 16:42:18 Test pvedaemon[6697]: <root@pam> starting task UPID:Test:0000AE02:001B3711:62BF5C2A:qmshutdown:100:root@pam: Jul 01 16:42:20 Test pvedaemon[6696]: <root@pam> starting task UPID:Test:0000AE03:001B37AD:62BF5C2C:vncproxy:100:root@pam: Jul 01 16:42:20 Test pvedaemon[44547]: starting vnc proxy UPID:Test:0000AE03:001B37AD:62BF5C2C:vncproxy:100:root@pam: Jul 01 16:42:29 Test pvedaemon[6696]: <root@pam> end task UPID:Test:0000AE03:001B37AD:62BF5C2C:vncproxy:100:root@pam: OK Jul 01 16:42:30 Test QEMU[44218]: kvm: terminating on signal 15 from pid 886 (/usr/sbin/qmeventd) Jul 01 16:42:30 Test kernel: fwbr100i0: port 2(tap100i0) entered disabled state Jul 01 16:42:30 Test kernel: fwbr100i0: port 1(fwln100i0) entered disabled state Jul 01 16:42:30 Test kernel: vmbr0: port 2(fwpr100p0) entered disabled state Jul 01 16:42:30 Test kernel: device fwln100i0 left promiscuous mode Jul 01 16:42:30 Test kernel: fwbr100i0: port 1(fwln100i0) entered disabled state Jul 01 16:42:30 Test kernel: device fwpr100p0 left promiscuous mode Jul 01 16:42:30 Test kernel: vmbr0: port 2(fwpr100p0) entered disabled state Jul 01 16:42:30 Test pvedaemon[6697]: VM 100 qmp command failed - VM 100 not running Jul 01 16:42:31 Test systemd[1]: 100.scope: Succeeded. Jul 01 16:42:31 Test systemd[1]: 100.scope: Consumed 1min 42.607s CPU time. Jul 01 16:42:31 Test pvedaemon[6697]: <root@pam> end task UPID:Test:0000AE02:001B3711:62BF5C2A:qmshutdown:100:root@pam: OK Jul 01 16:42:32 Test qmeventd[44592]: Starting cleanup for 100 Jul 01 16:42:32 Test qmeventd[44592]: Finished cleanup for 100 Jul 01 16:42:46 Test pvedaemon[6697]: <root@pam> update VM 100: -hostpci0 0000:04:00.0 Jul 01 16:42:49 Test pvedaemon[44636]: start VM 100: UPID:Test:0000AE5C:001B430F:62BF5C49:qmstart:100:root@pam: Jul 01 16:42:49 Test pvedaemon[6697]: <root@pam> starting task UPID:Test:0000AE5C:001B430F:62BF5C49:qmstart:100:root@pam: Jul 01 16:42:49 Test kernel: sd 2:0:0:0: [sdb] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: sd 2:0:1:0: [sdc] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:1:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: sd 2:0:2:0: [sdd] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:2:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: sd 2:0:3:0: [sde] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:3:0: [sde] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: sd 2:0:4:0: [sdf] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:4:0: [sdf] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: sd 2:0:5:0: [sdg] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:5:0: [sdg] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221103000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x000c), sas_addr(0x4433221103000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(0) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221100000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x0009), sas_addr(0x4433221100000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(3) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221101000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x000a), sas_addr(0x4433221101000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(2) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221102000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x000b), sas_addr(0x4433221102000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(1) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221104000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x000d), sas_addr(0x4433221104000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(7) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221105000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x000e), sas_addr(0x4433221105000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(6) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: sending message unit reset !! Jul 01 16:42:49 Test kernel: mpt2sas_cm0: message unit reset: SUCCESS Jul 01 16:42:50 Test systemd[1]: Started 100.scope. Jul 01 16:42:50 Test systemd-udevd[44640]: Using default interface naming scheme 'v247'. Jul 01 16:42:50 Test systemd-udevd[44640]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jul 01 16:42:51 Test kernel: device tap100i0 entered promiscuous mode Jul 01 16:42:51 Test systemd-udevd[44640]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jul 01 16:42:51 Test systemd-udevd[44640]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jul 01 16:42:51 Test systemd-udevd[44646]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jul 01 16:42:51 Test systemd-udevd[44646]: Using default interface naming scheme 'v247'. Jul 01 16:42:51 Test kernel: vmbr0: port 2(fwpr100p0) entered blocking state Jul 01 16:42:51 Test kernel: vmbr0: port 2(fwpr100p0) entered disabled state Jul 01 16:42:51 Test kernel: device fwpr100p0 entered promiscuous mode Jul 01 16:42:51 Test kernel: vmbr0: port 2(fwpr100p0) entered blocking state Jul 01 16:42:51 Test kernel: vmbr0: port 2(fwpr100p0) entered forwarding state Jul 01 16:42:51 Test kernel: fwbr100i0: port 1(fwln100i0) entered blocking state Jul 01 16:42:51 Test kernel: fwbr100i0: port 1(fwln100i0) entered disabled state Jul 01 16:42:51 Test kernel: device fwln100i0 entered promiscuous mode Jul 01 16:42:51 Test kernel: fwbr100i0: port 1(fwln100i0) entered blocking state Jul 01 16:42:51 Test kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state Jul 01 16:42:51 Test kernel: fwbr100i0: port 2(tap100i0) entered blocking state Jul 01 16:42:51 Test kernel: fwbr100i0: port 2(tap100i0) entered disabled state Jul 01 16:42:51 Test kernel: fwbr100i0: port 2(tap100i0) entered blocking state Jul 01 16:42:51 Test kernel: fwbr100i0: port 2(tap100i0) entered forwarding state Jul 01 16:42:51 Test kernel: vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support o> Jul 01 16:42:52 Test kernel: fwbr100i0: port 2(tap100i0) entered disabled state Jul 01 16:42:52 Test kernel: fwbr100i0: port 1(fwln100i0) entered disabled state Jul 01 16:42:52 Test kernel: vmbr0: port 2(fwpr100p0) entered disabled state Jul 01 16:42:52 Test kernel: device fwln100i0 left promiscuous mode Jul 01 16:42:52 Test kernel: fwbr100i0: port 1(fwln100i0) entered disabled state Jul 01 16:42:52 Test kernel: device fwpr100p0 left promiscuous mode Jul 01 16:42:52 Test kernel: vmbr0: port 2(fwpr100p0) entered disabled state Jul 01 16:42:52 Test pvedaemon[6696]: VM 100 qmp command failed - VM 100 not running Jul 01 16:42:52 Test pvedaemon[6698]: VM 100 qmp command failed - VM 100 not running Jul 01 16:42:52 Test pvedaemon[44636]: start failed: QEMU exited with code 1 Jul 01 16:42:52 Test pvedaemon[6697]: <root@pam> end task UPID:Test:0000AE5C:001B430F:62BF5C49:qmstart:100:root@pam: start failed: QEMU exited with code 1 Jul 01 16:42:52 Test systemd[1]: 100.scope: Succeeded. Jul 01 16:42:52 Test systemd[1]: 100.scope: Consumed 1.720s CPU time. Jul 01 16:42:52 Test pvedaemon[6697]: <root@pam> starting task UPID:Test:0000AEC3:001B4465:62BF5C4C:vncproxy:100:root@pam: Jul 01 16:42:52 Test pvedaemon[44739]: starting vnc proxy UPID:Test:0000AEC3:001B4465:62BF5C4C:vncproxy:100:root@pam: Jul 01 16:42:53 Test qm[44741]: VM 100 qmp command failed - VM 100 not running Jul 01 16:42:53 Test pvedaemon[44739]: Failed to run vncproxy. Jul 01 16:42:53 Test pvedaemon[6697]: <root@pam> end task UPID:Test:0000AEC3:001B4465:62BF5C4C:vncproxy:100:root@pam: Failed to run vncproxy. Jul 01 16:45:12 Test smartd[882]: Device: /dev/sdb [SAT], removed ATA device: No such device Jul 01 16:45:12 Test smartd[882]: Device: /dev/sdc [SAT], removed ATA device: No such device Jul 01 16:45:12 Test smartd[882]: Device: /dev/sdd [SAT], removed ATA device: No such device Jul 01 16:45:12 Test smartd[882]: Device: /dev/sde [SAT], removed ATA device: No such device Jul 01 16:45:12 Test smartd[882]: Device: /dev/sdf [SAT], removed ATA device: No such device Jul 01 16:45:12 Test smartd[882]: Device: /dev/sdg [SAT], removed ATA device: No such device

Thanks for the help!

 

[leesteken]

Its in a gen II dell r710.

Sorry, but I know nothing about those nor their specific motherboards.

Jul 01 16:42:49 Test kernel: sd 2:0:0:0: [sdb] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: sd 2:0:1:0: [sdc] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:1:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: sd 2:0:2:0: [sdd] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:2:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: sd 2:0:3:0: [sde] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:3:0: [sde] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: sd 2:0:4:0: [sdf] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:4:0: [sdf] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: sd 2:0:5:0: [sdg] Synchronizing SCSI cache Jul 01 16:42:49 Test kernel: sd 2:0:5:0: [sdg] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221103000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x000c), sas_addr(0x4433221103000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(0) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221100000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x0009), sas_addr(0x4433221100000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(3) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221101000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x000a), sas_addr(0x4433221101000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(2) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221102000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x000b), sas_addr(0x4433221102000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(1) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221104000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x000d), sas_addr(0x4433221104000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(7) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221105000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: removing handle(0x000e), sas_addr(0x4433221105000000) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: enclosure logical id(0x500605b00657c6b0), slot(6) Jul 01 16:42:49 Test kernel: mpt2sas_cm0: sending message unit reset !! Jul 01 16:42:49 Test kernel: mpt2sas_cm0: message unit reset: SUCCESS

The Proxmox host appears to lose the drives connected to the controller, which is expected. But you might want to early bind the controller to vfio-pci to prevent the host from seeing the drives before starting the VM. There is an example (for GPUs) here but you need to lookup your ids with lspci -ns 04:00.0.

Jul 01 16:42:51 Test kernel: vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support o>

That message might be a clue. Have you tried allowing unsafe interrupts?

Thanks for the help and sorry about the late reply.

No worries, I'm not in a hurry. Have you taken a look at the other forum topics that had the same error message for any possible work-arounds? Have you checked for a newer system BIOS version?

 

[cpd1900]

That message might be a clue. Have you tried allowing unsafe interrupts?

Ahhh jeez I thought I allowed unsafe interrupts.... Once I turned this I can get the VM to start booting but gets hung on "Booting from hard drive" after going through multiple booting process. since I've run into the next piece and seam to have got past the original problem do you suggest early binding the device? I hadn't seen any posts that were fixing this problem but I wouldn't be surprised if I just didn't search the right stuff and or just overlooked it. I will see if i can get it to fully boot and will reply hear when I do!

Thanks so much for the help!

 

[cpd1900]

Thank you so much for the help! I was able to find this post on the Proxmox forum talking about this problem and linking to the Truenas forum https://forum.proxmox.com/threads/vm-refuses-to-boot-from-virtual-disk-when-pci-passthrough-is-enabled.38146/ and it explained my new problem was just a setting I didn't know about on the HBA.

Again Thanks so much for all the help it would have taken me all weekend to get a quarter of they way through this,
~ Chris