[SOLVED] VM refuses to boot from virtual disk when PCI passthrough is enabled

[Gerald Tan]

I'm installing FreeNAS on a VM with a 128GB boot disk (Virtio / ZFS zvol) with PCI passthrough to a LSI SAS 9207-8i with 4 disks attached. The FreeNAS installation detects all 5 disks, so I install FreeNAS onto the 128GB /dev/vtbd0, however it refuses to boot from the virtio disk after the installation.

I destroyed the VM, recreated it without the PCI passthrough, and repeated the installation onto the lone virtio drive. This time FreeNAS boots up fine after installation. I shutdown the VM to enable PCI passthrough, and restarted the VM and the VM refuses to boot from the virtio drive again.

Any ideas what I may have done wrong?

vm configuration:

bootdisk: virtio0
cores: 16
cpu: host
ide2: nfserver9-pve:iso/FreeNAS-11.0-U4.iso,media=cdrom
memory: 12288
name: storage1
net0: virtio=CA:4E:3A:CD:7D:78,bridge=vmbr1
numa: 0
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=bc758b91-d2b5-46de-86e8-1fb11192350b
sockets: 1
virtio0: local-zfs:vm-116-disk-1,size=128G
hostpci0: 04:00.0,pcie=1
machine: q35

Thanks

 

[dcsapak]

have you tried virtio-scsi instead of virtio-blk ?
i have no idea how exactly the freenas boot process works, but maybe it has to do with how it enumerates pci/pcie devices (virtio-blk uses one controller per disk)

 

[Gerald Tan]

I was able to solve the problem after finding this reddit thread titled "kvm_hba_passthrough_will_not_boot_form_virtual" (sorry, can't post link as new user) and the linked post on freenas forums.

TLDR, disable BIOS boot option in the HBA configuration