[SOLVED] Move pass through disks to pass through sata controller

[keithstric]

Hello,
I've got a Proxmox 8.0 setup with 4 disks passed through to a TrueNas Scale VM. This works perfectly, no issues. I have purchased more disks and a SATA controller. I moved all the physical disk connections (including the new disks) to this new controller and the pass throughs are still working perfectly without having to do anything. I have not passed through the new disks. Since I have this new controller I would prefer to pass through the controller instead of the individual disks, I'm just not sure how to accomplish that.

1. Do I need to undo the pass through for those disks?
   1. If i need to undo the pass through, is there any documentation describing how to do that?
2. Are there any other considerations I should take into account when I pass through the sata controller?
   1. IOMMU is enabled in bios and Proxmox

What I've tried so far is just removing the pass through disks from the VM configuration file, then passed through the controller. This didn't work at all and actually crashed Proxmox when I started the VM. I restored the working VM configuration file which got everything working again (thankfully). Any guidance in this matter would be great.

Thanks in advance for your help

 

[keithstric]

OK, reached out to folks on Discord who gave me ideas on how to proceed. Here's what I ended up doing which hopefully may help someone in the future.

• In the VM -> Hardware screen: "Detach" all passed through disks (NOT the vm boot disk)
• Pass through the new SATA Controller
• Start VM

This resulted in Proxmox crashing. After some research figured out that the pass through PCIe device needed to be in it's own IOMMU group. By passing through that card before addressing this, basically everything in that group was being passed through to the VM and thus unavailable to Proxmox. That particular IOMMU group included the new SATA controller, network card, the built-in SATA controller and some USB Ports. This is the remaining part of the process I followed to address the IOMMU group issue.

• Ran this script to see the IOMMU groups. Noticed a lot of devices in the same group
  

  for d in $(find /sys/kernel/iommu_groups/ -type l | sort -n -k5 -t/); do    n=${d#*/iommu_groups/*}; n=${n%%/*}
      printf 'IOMMU Group %s ' "$n"
      lspci -nns "${d##*/}"
  done;

• Edit the /etc/default/grub file
• Append pcie_acs_override=downstream,multifunction to the GRUB_CMDLINE_LINUX_DEFAULT line
• run update-grub
• Reboot Proxmox
• Pretty much every device got put into it's own IOMMU group
• Since the ID of the SATA card changed, need to re-pass through that card to the VM
• Start the VM

The result of that was a successful boot of TrueNas with all my existing and new disks and pools available and functioning. Looked at the Proxmox node's disks and none of the drives connected to the new SATA controller are visible which is expected.

What helped me figure this out is this post. It explained the IOMMU issue I was experiencing perfectly.

 

[powerjc]

Created an account just to say thank you for coming back and closing up on what you ended up doing. Saved me a bunch of messing around. Awesome interneting @keithstric

Not today:
https://xkcd.com/979/