Drives Not Spun Down Gracefully on Proxmox Shutdown (HBA Passthrough to TrueNAS)

[Gutenberg]

Hello everyone,

I'm using Proxmox VE 8.4.5 to host a TrueNAS SCALE VM. I have four hard drives connected to an IBM M1015 HBA (flashed to IT mode), which is passed through to the TrueNAS VM via PCI passthrough.
The S.M.A.R.T. data for one of the drives shows a high number of ungraceful power-offs, which might be caused by a misconfiguration on my part:

192 Power-Off_Retract_Count -O--CK   100   100   000    -    326

When I shut down the Proxmox host, I can hear that the drives are not spinning down gracefully. I suspect this happens because the HBA is passed through to the TrueNAS VM, and the host no longer has access to it. Interestingly, when I shut down the TrueNAS VM, the drives remain spinning, suggesting that Proxmox may be cutting power to the HBA abruptly during shutdown. This would also explain the mechanical noise I’m hearing.
I'm not sure how to properly resolve this. Since the host can't access the HBA once it's passed through, spinning down the drives before shutdown doesn’t seem possible from the Proxmox side.

While the server now runs 24/7 and isn't powered down often, I still feel this is something that should be addressed properly.

Any ideas or suggestions would be greatly appreciated!

 

[leesteken]

I'm not sure how to properly resolve this. Since the host can't access the HBA once it's passed through, spinning down the drives before shutdown doesn’t seem possible from the Proxmox side.

Indeed this is not possible without a driver for it loaded. The vfio-pci driver is still connected (due to and for the passthrough) but it does not know anything about your specific device. Unbind vfio-pci and rebind/load the actual Linux driver for the device. There are several threads on the forum about doing this for a GPU but the principle is the same.

 

[Gutenberg]

Thanks! I added a hook script to load the driver once the VM is shut down. The drives are now visible and the data is accessible.
However, they still make a mechanical "clunk" sound when Proxmox shuts down — as if they're being powered off abruptly.

Is that normal behavior?
Or do I need to manually spin them down before shutdown to avoid this?

 

[leesteken]

Thanks! I added a hook script to load the driver once the VM is shut down. The drives are now visible and the data is accessible.
However, they still make a mechanical "clunk" sound when Proxmox shuts down — as if they're being powered off abruptly.

Happy to hear that you figured it out.

Is that normal behavior?
Or do I need to manually spin them down before shutdown to avoid this?

I don't know and have no experience with your hardware or its driver. Maybe also send some kind of spin-down command in the hookscript? This is most certainly not specific to Proxmox and internet guides or other Linux user experiences with this controller/driver and drives will probably apply.

 

[cucumberjuice66]

Hey there @Gutenberg would you mind sharing your script since I ran into the same issue with the ungraceful shutdowns. Or maybe someone else can help? Thanks a lot

 

[uzumo]

With the pass-through virtual machine stopped (and auto-boot also stopped), I think you can use lspci -nnk to check the driver and ID and do the following.

If you start the virtual machine without unbinding it at boot, I think it will be vfio-pci.

Test each command and edit the script.

mkdir -p /var/lib/vz/snippets
nano /var/lib/vz/snippets/hba.sh
----
#!/bin/bash

phase="$2"
echo "Phase is $phase"
if [ "$phase" == "pre-start" ]; then
    echo "0000:xx:00.0" > /sys/bus/pci/drivers/<drivername>/unbind 2>/dev/null
    sleep 2
elif [ "$phase" == "post-stop" ]; then
    sleep 5
    echo "0000:xx:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null
    sleep 2
    echo "0000:xx:00.0" > /sys/bus/pci/drivers/<drivername>/bind 2>/dev/null
    sleep 2
fi
----
qm set vmid -hookscript local:snippets/hba.sh

test

echo "0000:xx:00.0" > /sys/bus/pci/drivers/<drivername>/unbind 2>/dev/null
lspci -nnk
echo "0000:xx:00.0" > /sys/bus/pci/drivers/vfio-pci/bind 2>/dev/null
lspci -nnk
echo "0000:xx:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null
lspci -nnk
echo "0000:xx:00.0" > /sys/bus/pci/drivers/<drivername>/bind 2>/dev/null
lspci -nnk

 

[Gutenberg]

Hey there @Gutenberg would you mind sharing your script since I ran into the same issue with the ungraceful shutdowns. Or maybe someone else can help? Thanks a lot

Hi,

I can share my script, but unfortunately it doesn’t solve the issue with the ungraceful shutdowns

I am able to access the drives after the VM shuts down, but I can’t get them to spin down properly. I’ve tried different commands like

hdparm -Y ...

or

sg_start --stop ...

– also in combination with

echo 1 | tee /sys/block/.../device/delete

and at different points, e.g. inside the hook script or in a service with

Before=final.target

However, just before the Proxmox server shuts down, the disks spin up again only to be cut off from power right afterwards, which causes that unpleasant mechanical noise.

In some forums I’ve read that people patched the HBA driver to allow for a graceful spin-down, but I don’t really know how to do that, and I’m a bit hesitant to experiment with my otherwise perfectly working system.

 

[cucumberjuice66]

Have you tried not passing through the drives to see if a graceful shutdown works then? I don't use an HBA card just mainboard SATA ports. I wonder if there is a difference.