RAID controller PCI-E passthrough only works on first VM boot

MetricVoid

Active Member
Nov 11, 2018
8
0
41
24
I'm running FreeNAS/TrueNAS in a virtual machine with RAID controller Adaptec 71605. The adapter is passed through to the VM.

Here are the guides I followed when setting up the passthrough:
- https://pve.proxmox.com/wiki/Pci_passthrough
- https://www.thomas-krenn.com/en/wiki/Enable_Proxmox_PCIe_Passthrough
- Additionally, for TrueNAS to work with this controller, I enabled the "aac" module in loader.conf: https://www.freebsd.org/cgi/man.cgi?query=aac

The problem is as follows:
- When the VM is started for the first time, everything looks normal.
- However, the VM will fail to boot on subsequent starts and reboots. The only way to resolve this is to reboot the whole Proxmox.

Here is the settings of the VM. No sensitive information because the whole system is experimental.
"All Functions" was enabled in the PCI settings.
1620193478434.png

1620193527054.png


When the machine boots normally, there will be a RAID card self-diagnosis page, followed by
Code:
SeaBIOS (version ....)
Machine UUID ...
Booting from hard disk...
then followed by the operating system.

When the machine fails to boot, it will stuck on a screen like this:
1620193599349.png

And, in other cases, (this is more common)
1620194686941.png

After which the whole machine will be freeze and stop responding to key inputs. Still no operating system.

My guess was that some additional steps would be required to de-initialize the controller, which was not performed correctly.

=========================================

System information: Ryzen 7 5800X with Asrock Rack X570D4U-2L2T (server board with onboard graphics).

I also went through this post: https://forums.unraid.net/topic/93787-adaptec-asr-71605-drives-not-showing-up-after-a-reboot/
Swapping PCI-E slots around had no avail.
 

Attachments

  • 1620192867916.png
    1620192867916.png
    63 KB · Views: 1
Just found a post saying turning rombar off. That helped a little bit, but now the system still won't boot. The OS stucks here.

1620197937054.png
 
OK, so my new assumption is - the system BIOS starts the controller, hands it over to Proxmox then to the VM. When the VM is shut down, the controller also shuts down, but neither SeaBIOS nor OVMF can start the controller back up.

Waiting for more advice...
 
Some controllers just don't reset properly or have no FLR functionality. Maybe dumping the ROM of the controller to a file and adding romfile= to hostpci can help. I assume it would be similar to GPU passthrough and you need to make sure the controller is not initialized by the BIOS or used by the Linux driver when you read the ROM, but I'm not sure how to do that. Since it works when starting the VM once, maybe that is not necessary and you can simply read the ROM before starting the VM.
 
Some controllers just don't reset properly or have no FLR functionality. Maybe dumping the ROM of the controller to a file and adding romfile= to hostpci can help. I assume it would be similar to GPU passthrough and you need to make sure the controller is not initialized by the BIOS or used by the Linux driver when you read the ROM, but I'm not sure how to do that. Since it works when starting the VM once, maybe that is not necessary and you can simply read the ROM before starting the VM.
Thanks for your advice!

Dumping the ROM gives Input/Output errors when the VM is not started, and (I don't know why but it's counter-intuitive) it will only success when the VM is started. Adding romfile to VM options did not help. This now looks more like a FreeBSD problem than a Proxmox problem.
 
Any further movement on this? I'm currently having the same issue and would love to find a solution to the problem.
 
I cannot help you in particular, but can say, that I use TrueNAS with a Fujitsu controller, followed the Proxmox PCI passthrough documentation and everything works as a charm.
 
I am not having this exact issue. Did anyone ever find a solution?
I solved it by changing the card. Older ones worked less consistent than newer ones. I also omit passthrouhing stuff on machines that are critical. Passthrough is unfortunately not always production-ready, too much hardware dependencies (what mobo, bios and pcie cards).
 
  • Like
Reactions: leesteken
Ok ... well, grr. If I were going to replace the card which card is the one of choice? I have seen the LSI card mentioned but would need to go dig to find the model people have mentioned. What have you seen or what did you use? I also have a cabling issue ... maybe. These cards have SFF-8644 connectors, which is the cable that is currently in use to go to the backplane for the hard drives.
 
Update: I ordered the 9300-8i LSI card. Waiting for it to arrive. Any tips once I get it?
 
Necrobumping: The problem went away beautifully as I migrated from TrueNAS CORE to TrueNAS SCALE.
It might be a FreeBSD problem after all.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!