What happens to a HBA when not PCI passthroughed?

[Dunuin]

Hi,

My homelab is running out of ressources and instead of buying a forth server I was thinking about merging stuff down to 2 servers. Electricity in Germany is way too expensive

Right now I got 1x bare metal PVE + 2x bare metal TrueNAS servers. And I would need another PVE server to be able to run more VMs and so that I'm not totally screwed up if my only PVE server would fail so I atleast could backups of the most important VMs to another node.

Virtualizing my primary TrueNAS Server wouldn't be a big problem. I already got two Dell PERC H310 in IT mode (so basically a LSI 9211-8i HBA) so I could passthrough my 8 HDDs and 8 SSDs to a TrueNAS VM. That way I could get rid of one of my TrueNAS servers.

But the second TrueNAS server I only use for backups and it is shutdown most of the time and is only booted (over IPMI using a script) once per week to receive ZFS replications so the HDDs are totally powered down most of the time so they should live long. Now I was thining if it would be possible to virtualize that backup TrueNAS server too...

What will happen if I block the LSI drivers in PVE and PCI passthrough such a LSI 9211-8i HBA to a TrueNAS VM? If I get it right the HBAs firmware will spinup the disks at boot so they are ready to be booted from. But what happens if the drivers are then blocked once PVE has started and the autostart of the TrueNAS VM is disabled? Will the disks spindown and the heads go into parking position? Such a setup would only make sense to me if the drives are powered down as long as the TrueNAS VMs isn't started so they don't wear if the TrueNAS VM isn't started. Is such a thing possible? So what actually happens to a PCI card if it is neighter used by the host nor a guest?

Another problem I would think of is the ATX power supply. If I merge my 8 HDDs + 8 SSDs of my main TrueNAS server with the 8 SSD + 2 HDDs of my PVE server I would got 10 HDDs and 16 SSDs in one server powered by a single Consumer ATX power supply. HDDs and SSDs use 12V + 5V and I checked the datasheet of my SSDs and they use up to 4.6W on the 5V rail + up to 5.5W on the 12V rail per drive. So these 16 SSDs alone would add up to 74W on the 5V rail. IF I look at the ATC power supply rating it doesn't matter if I buy a 300W or a 1600W power supply. All ATX power supplys nowadadys will only offer 20-30A or 100-150W on the 5V rails. Only difference between a 300W and 1600W PSU is the amount of power it can handle on the 12V rail. So I'm a bit worried to not find a suitable PSU to power all those drives.

Does someone got any experiences with those two problems?

 

[janssensm]

I checked the datasheet of my SSDs and they use up to 4.6W on the 5V rail + up to 5.5W on the 12V rail per drive

Interesting case, I don't have that much drives. But you got me checking the specs for my ssd's as I use similar drives as you do.
There is an interesting detail in the specs for Intel S3700 and S3710, assuming you have the S3710.

S3700:

2.5-inch: 5V or 12V SATA Supply Rail (note10: Defaults to 12V, if both 12V and 5V are present)

S3710:

5V or 5V+12V SATA Supply Rail (note11: If both 12V and 5V power supplies are present, defaults to 5V+ 12V power supplies. Does not support 12 volt only.)

So when stuffing your system with a lot of these drives, the S3700 is favoured as it seems to use 12V only when possible and leaving the 5V rail alone. I understand this info will not help you a lot, searching the web I only found some people modding their psu to convert a 12V output to 5V with a regulator/converter.
You could also configure a dual psu setup

 

[Dunuin]

Interesting case, I don't have that much drives. But you got me checking the specs for my ssd's as I use similar drives as you do.
There is an interesting detail in the specs for Intel S3700 and S3710, assuming you have the S3710.

S3700:


S3710:


So when stuffing your system with a lot of these drives, the S3700 is favoured as it seems to use 12V only when possible and leaving the 5V rail alone. I understand this info will not help you a lot, searching the web I only found some people modding their psu to convert a 12V output to 5V with a regulator/converter.
You could also configure a dual psu setup

14 of the SSDs are S3710 and two are S3700.

I also thought about using an 12V to 5V converter but could be hard to fit that inside the case (glue it somewhere) and I'm not sure if I want to trust such converters. Would be really bad if it fries all that expensive hardware.

And redundant PSUs could indeed double the power on the 5V rail (founs some with power distribution board that can handle 50A instead of 20-30A but they cost alot more and should be quite noisy with that 40mm fans and if I understand it right they will be less efficient and I already have problems to pay the electricity bills.

The 5V rails are really a problem nowadays. I also got some retro PCs that need alot of 5V power. Prior to 2002 all the ATX PSUs got like 50A on the 5V rail because the CPUs were working with 5V and not 12V. With the 12V CPUs after 2002 the computers didn't need that much 5V power anymore so all the PSU manufacturers lowered the 5V performance. Because ATX is backwards compatible it would have been totally fine to use a modern PSU in such a 20 years old retro PC but I needed to buy a 20 years old used PSU because the new PSUs just cant handle all the 5V load.
So buying a 20+ years old PSU would work too but thats also not really an option because of the bad power efficiency and reliability.

 

[janssensm]

I agree with all your thoughts above, no easy and reliable solution.
Just another thought, but a rather bold one, sell all s3710 and buy 3700, assuming all info in the spec sheet from Intel is accurate.
Specs are comparable for these drives, S3710 is just a tad better.