How to Correctly Passthrough Hard Drives to Either Unraid or TrueNAS VMs?

[nemo868]

To begin, I confirmed that my processor supports VT-x, VT-d is enabled in the BIOS and I turned IOMMU on in Proxmox

I installed Proxmox this week for the first time and created 5 VMs that all run relatively well. I am attempting to create either an Unraid or TrueNAS VM that requires hardware passthrough of a USB boot drive (for Unraid only) and a hard drive controller (or individual drives) from what I understand.

When I passthrough the USB boot drive and start the Unraid VM, the machine boots as expected.

My motherboard has an LSI 2308 onboard controller that I see listed among the PCI devices but when selected, the Unraid VM no longer sees the USB boot device but instead attempts to boot from a hard disk and loads an Avago Technologies MPT SAS2 BIOS, begins initializing and searches for devices at HBA 0 and of course loops continuously.

After spending an entire day trying to figure that out I decided to try a TrueNAS Scale VM instead. The OS installs, boots and I can access the GUI and log in remotely just fine but when I add the same controller, the VM doesn't even start and I see the following error in the Proxmox cluster log:

kvm: -drive file=/dev/disk/by-id/ata-ST2000DM001-1ER164_Z4Z3NPQP,if=none,id=drive-sata1,format=raw,cache=none,aio=io_uring,detect-zeroes=on: Could not open '/dev/disk/by-id/ata-ST2000DM001-1ER164_Z4Z3NPQP': No such file or directory
TASK ERROR: start failed: QEMU exited with code 1

Using the qm set command on the Proxmox shell I was able to individually passthrough only 5 of the 8 drives to the TrueNAS VM. When I attempt to add the 6th drive I receive the following error:

unknown option: sata6 400 unable to parse option

The 5 drives are visible in the TrueNAS GUI and are available for use in creating a ZFS pool

What am I doing wrong? What is the correct way to get all 8 drives added to either my Unraid or TrueNAS VM? I would prefer to use Unraid but at this point, I'll go with whichever one I can get working.

To be clear, none of this is mission-critical. I am new to Linux and just experimenting with the Proxmox software and the idea of virtualizing a NAS instead of running on bare metal.

Supermicro X10SL7-F motherboard
Intel Xeon E3-1245 v3 CPU
32 GB Crucial ECC RAM
8x2TB HDDs

 

[LnxBil]

To be clear, none of this is mission-critical. I am new to Linux and just experimenting with the Proxmox software and the idea of virtualizing a NAS instead of running on bare metal.

Sorry, but virtualizing and passthrough is the exact opposite. Don't do such crasy setups. Your passthrough can break on every PVE update and it eventually will, so spare yourself the frustration and the invested time and just virtualize everything or nothing. I would be glad if someone would have told me that many years ago. I went into that rabbit hole and I'm just virtualizing everything without any hardware dependend stuff like passthrough (except in LXC, but that is not the same level of passthrough).

My motherboard has an LSI 2308 onboard controller that I see listed among the PCI devices but when selected, the Unraid VM no longer sees the USB boot device but instead attempts to boot from a hard disk and loads an Avago Technologies MPT SAS2 BIOS, begins initializing and searches for devices at HBA 0 and of course loops continuously.

Yes, makes sense. Try to get into the VM BIOS and try to change the boot order there. It cannot be configured because it is a mixed setup. Such setups are always a PITA and quality therefore as crasy. You can try to passthrough your USB controller with the USB disk, thay may help, but I'm not sure.

 

[nemo868]

Sorry, but virtualizing and passthrough is the exact opposite. Don't do such crasy setups. Your passthrough can break on every PVE update and it eventually will, so spare yourself the frustration and the invested time and just virtualize everything or nothing. I would be glad if someone would have told me that many years ago. I went into that rabbit hole and I'm just virtualizing everything without any hardware dependend stuff like passthrough (except in LXC, but that is not the same level of passthrough).

Thank you for your speedy response. I reread it a few times but I'm a bit confused. You suggested that I virtualize everything or nothing. Isn't that what I am attempting to do? I currently have 5 machines including my firewall virtualized and now I want to also virtualize my NAS. How are you virtualizing without doing passthrough?

Yes, makes sense. Try to get into the VM BIOS and try to change the boot order there. It cannot be configured because it is a mixed setup. Such setups are always a PITA and quality therefore as crasy. You can try to passthrough your USB controller with the USB disk, thay may help, but I'm not sure.

Would I need to switch from the default SeaBIOS to OVMF(UEFI) in order to change the boot order? Are you suggesting that I attempt to passthrough the USB controller instead of the USB drive itself? Hmm....I'm not sure how to identify the USB controller in the list of PCI devices but I will see what I can do.

Thanks again for the suggestions.

 

[LnxBil]

Thank you for your speedy response. I reread it a few times but I'm a bit confused. You suggested that I virtualize everything or nothing. Isn't that what I am attempting to do?

Passthrough is the opposite of virtualizing. The idea behind virtualizing is abstraction and hiding everthing in their own layer, so that there are no direct requirements between the layers. You can move around the VM in a cluster and even change the underlying virtualization architecture without noticing it (besides some disk strings and virtual devices that are hypervisor-specific). Passthrough is breaking those layers ob abstraction.

How are you virtualizing without doing passthrough?

What would I need to passthrough? I'm running hundreds of VMs without any.