PCI passthrough causing crash of server

[cheezburger]

So I am trying to get FreeNAS working and pass through my 4x internal HDD's to the VM so that I can run a pool within the VM.

At first, I was getting the following error:

() Add: PCI Device No IOMMU detected, please activate it.See Documentation for further information.

After following the Wiki, I ensured that the BIOS had Virtualization enabled and also made sure that remapping was enabled.

I no longer get the error when adding "Sunrise Point-H SATA controller [AHCI mode]" as a PCI Device to the FreeNAS VM [101].

However, when I try and boot the VM [101] it crashes my ProxMox server with the following:

Error start failed QEMU exited with code 1

And the following in the logs:

Buffer I/O error on device dm-1, logical block
Aborting journal on device dm-1-8
Buffer I/O error on dev dm-1, logical block lost sync
EXT4-FS (dm-1): I/P error while writing superblock
EXT4-FS error (device dm-1) ext4_journal_check_start:61: Detected aborted
EXT4-FS (dm-1): remounting file system read only
Buffer I/O error on device dm-1, logical block

and then it basically loops this through and crashes my Server until I restart it.

I checked all the SMART values for drives and they seem to have PASSED?

Here is the lvs output also:

lvs
  LV            VG  Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve twi-aotz-- <151.63g             7.07   1.38                           
  root          pve -wi-ao----   59.25g                                                   
  swap          pve -wi-ao----    8.00g                                                   
  vm-100-disk-0 pve Vwi-aotz--    4.00m data        0.00                                   
  vm-100-disk-1 pve Vwi-aotz--   20.00g data        46.06                                 
  vm-101-disk-0 pve Vwi-a-tz--   50.00g data        3.01

This server crashing only happens when I add the PCI card listed above to the VM.

 

[leesteken]

It sounds like the host loses its connection to your boot/root drive when you passthrough a device to your VM. All devices in a group are lost to the host when you pass one of them to a VM.
Please run this command (as root on your host): for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done; It will show us the IOMMU groups and the PCI-ids of all your devices.
Please tell us the PCI-id of the device you are trying to pass through and the PCI-id of the controller that has you Proxmox boot/root drive.

 

[cheezburger]

Thanks for the help! Here is the output:

IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation Skylake Host Bridge/DRAM Registers [8086:1918] (rev 07)
IOMMU group 10 04:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU group 1 00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)
IOMMU group 1 00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Thermal subsystem [8086:a131] (rev 31)
IOMMU group 2 00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)
IOMMU group 3 00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)
IOMMU group 4 00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #5 [8086:a114] (rev f1)
IOMMU group 5 00:1c.5 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #6 [8086:a115] (rev f1)
IOMMU group 6 00:1c.6 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #7 [8086:a116] (rev f1)
IOMMU group 7 00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a14a] (rev 31)
IOMMU group 7 00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
IOMMU group 7 00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)
IOMMU group 8 01:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)
IOMMU group 8 02:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)
IOMMU group 9 03:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)

What I tried to passthrough is 00:17.0 SATA controller, which might be incorrect.

What I am actually trying to passthrough is the following (disk a-d):

/dev/sda 3.64TiB
/dev/sdb 3.64TiB
/dev/sdc 3.64TiB
/dev/sdd 3.64TiB

/dev/sde 238.47GiB* This is where Proxmox and VM are hosted.
/dev/sdf 2.73TiB

As you state, it must be because drive sde with Proxmox hosted is included that it causes the failure. I wasn't aware this would cause an issue. So how do I pass through drives a-d only?

 

[leesteken]

The IOMMU groups show that you have 1 SATA controller, which you want to pass through. I do not see another disk controller.
You cannot pass through the SATA controller when your Proxmox host and VM virtual disk are also connected to that same SATA controller.
The Proxmox host loses connection to the SATA controller when you pass it through to the VM, and therefore loses connection to the Proxmox disk and VM virtual disk.
You need (to buy) another SATA controller to separate the disks you want to pass through and the Proxmox & VM disks.

PS: or you need to pass through the disks, not the SATA controller. See the wiki for and example.

 

[cheezburger]

The IOMMU groups show that you have 1 SATA controller, which you want to pass through. I do not see another disk controller.
You cannot pass through the SATA controller when your Proxmox host and VM virtual disk are also connected to that same SATA controller.
The Proxmox host loses connection to the SATA controller when you pass it through to the VM, and therefore loses connection to the Proxmox disk and VM virtual disk.
You need (to buy) another SATA controller to separate the disks you want to pass through and the Proxmox & VM disks.

PS: or you need to pass through the disks, not the SATA controller. See the wiki for and example.

Thanks for the prompt response! Trying disk passthrough at the moment, thanks for linking the Wiki!

Stuck on the following line:
update VM 101: -scsi2 /dev/disk/by-id/ata-WDC_WDXXXYYYYY

-bash: update: command not found

So then I try apt:
apt update VM 101: -scsi2 /dev/disk/by-id/ata-WDC_WDXXXYYYYY

E: Command line option 's' [from -scsi2] is not understood in combination with the other options.

Any tips?

 

[leesteken]

update VM 101: -scsi2 /dev/disk/by-id/ata-WDC_WDXXXYYYYY

The command is qm set 101 -scsi2 /dev/disk/by-id/ata-WDC_WDXXXYYYYY. But only if you want to connect the drive as a SCSI drive. Make sure you are using different numbers for each disk. The example uses /dev/disk/by-id/... because it is more stable than /dev/sda etc. but I don't know your drives, so using /dev/sda will work for you.
Please search the forum, use your favorite seach engine, and read the manual for more information and examples.

 

[cheezburger]

The command is qm set 101 -scsi2 /dev/disk/by-id/ata-WDC_WDXXXYYYYY. But only if you want to connect the drive as a SCSI drive. Make sure you are using different numbers for each disk. The example uses /dev/disk/by-id/... because it is more stable than /dev/sda etc. but I don't know your drives, so using /dev/sda will work for you.
Please search the forum, use your favorite seach engine, and read the manual for more information and examples.

Apologies, I thought I had to run the command lines one after the other (qm set followed by update).

Again, thank you for the super prompt response and all your assistance!