HDD Raid config with PVE and VMs on SSD

[robin91]

Hello,

I have a server running Proxmox with two VMs (Ubuntu& Homeassistant) for home use. Currently all VMs and PVE are running on a single SSD.
I am considering to install a RAID array of HDDs on the server to have the server also function as a NAS - though still have a question about this and would appreciate any feedback/tips.

The setup I'm now considering is as follows:

• Proxmox and VMs stay on the single SSD to have a quick boot speed/ good OS responsiveness
• I install 3 HDDs and then create a RAID-Z1 array in PVE (ZFS)
• The full storage of that HDD raid array I make visible to Ubuntu as virtual disk.

My questions:

1. In above setup, how do I recover the RAID array if my SSD fails (which has Proxmox/VMs on it)? After I reinstall a fresh version of Proxmox and Ubuntu VM on a new SSD, can I then restore the RAID array? (and how?)
2. Any other suggestions to improve this/other suggested RAID configs? I would like to keep the setup relatively power efficient and cheap - my current motherboard, power supply, case can only handle 4x SATA - at this point I do not intend to upgrade this to allow more drives (hence no extra SSD to allow mirroring for now).

Appreciate any feedback/answers!

Robin

 

[Dunuin]

The full storage of that HDD raid array I make visible to Ubuntu as virtual disk.

Than don't forget to increase the blocksize of that ZFS storage to 16K before creatingthat virtuak disk or you will only be able to use 50% of the raw capacity. And keep in mind when dimensioning the disks, that 20% of the pool should be kept free for best performance and to minimize fragmentation (as you can't defrag a HDD that is part of a ZFS pool). So with 3 disks in a raidz1 and a volblocksize of 16K you get around 53% of the raw capacity as usable storage for your data.
And make sure not to buy SMR HDDs.

In above setup, how do I recover the RAID array if my SSD fails (which has Proxmox/VMs on it)? After I reinstall a fresh version of Proxmox and Ubuntu VM on a new SSD, can I then restore the RAID array? (and how?)

Thats not that easy. When backing up your NAS VM it would also backup that big virtual disk on the HDDs so your backup would be huge. You could tell PVE to not backup that big virtual disk, but then doing a restore of that backup would wipe your excluded big virtual disks and you lose all your data. See here: https://forum.proxmox.com/threads/feature-request-advanced-restore-options-in-gui.109707/
So you might need to restore that VM using another VMID, so the existing disks won't get wiped, then move that NAS disks between the VMs.
In case of a new PVE install you would need to restore that NAS VM with excluded disks, then do a "zpool import" and add a new storage of type "ZFSpool" pointing to your imported pool. You then would need to edit your VMs config file to add the virtual NAS disks.

 

[robin91]

Thats not that easy. When backing up your NAS VM it would also backup that big virtual disk on the HDDs so your backup would be huge. You could tell PVE to not backup that big virtual disk, but then doing a restore of that backup would wipe your excluded big virtual disks and you lose all your data. See here: https://forum.proxmox.com/threads/feature-request-advanced-restore-options-in-gui.109707/
So you might need to restore that VM using another VMID, so the existing disks won't get wiped, then move that NAS disks between the VMs.
In case of a new PVE install you would need to restore that NAS VM with excluded disks, then do a "zpool import" and add a new storage of type "ZFSpool" pointing to your imported pool. You then would need to edit your VMs config file to add the virtual NAS disks.

Thanks a lot for your feedback! This indeed sounds complicated to restore, I was already a bit afraid of this. Is there any way to make this easier? Would it for instance be better to passthrough the disks instead to the VM directly instead of making it virtual? Any other suggestions?

 

[Dunuin]

Thanks a lot for your feedback! This indeed sounds complicated to restore, I was already a bit afraid of this. Is there any way to make this easier? Would it for instance be better to passthrough the disks instead to the VM directly instead of making it virtual? Any other suggestions?

There aren't many options. Passthrough of the real disks is only possible by passing through the whole disk controller with all of its ports, so all 4 disks. Best would be to buy a PCIe HBA card and then use PCI passthrough to passthrough that dedicated disk controller with your 3 disks attached.

To passthough single disks you would need to use disk passthrough: https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)
But with this your VM will still work with virtual disks mapped to the physical disks sectors, so stuff like SMART won't work inside the VM and you get bit of additional overhead.