Storage (how would you?)

[Soogs]

Hello, I am in the process of refreshing my lab/pve cluster.

Will not be clustering this time and just running solo nodes.

My main questions is which is a better solution from the following:

Boot and machines: 2TB NVMe + Additional VM storage 2TB SATA SSD

or

Boot: 256GB SATA SSD + 2TB NVMe for machines + USB3 SATA SSD for additional storage for VMs

Also should I be using ZFS for all two/three drives?

All PCs are micros so only have one NVMe and one SATA connection so USB or NFS/SMB are the only other options.

All network attached drives are USB3 so was figuring having 2TB attached directly by USB should be better than USB over NFS/SMB

How would you do this?

Thanks.

 

[Dunuin]

Boot and machines: 2TB NVMe + Additional VM storage 2TB SATA SSD

or

Boot: 256GB SATA SSD + 2TB NVMe for machines + USB3 SATA SSD for additional storage for VMs

I personally would mirror those two 2TB SSDs for reliability and use that mirror to store everything. Even if that means losing half of the raw capacity and that the NVMe SSD will be slowed down to SATA performance.
If you don't want to mirror I would use the first option but then make sure to have a properly tested disaster recovery plan as there is no official way to backup the PVE nodes yet and people often use problematic stuff like PBS virtualized on the same PVE node.

Also should I be using ZFS for all two/three drives?

Depends on how much you care about data integrity and those ZFS features. Keep the ZFS recommendations in mind like lots of ECC RAM (usually not an option when using MiniPCs) and Enterprise/Datacenter grade SSDs with integrated power-loss protection (possible with MiniPCs but don't expect them to be cheap).

All PCs are micros so only have one NVMe and one SATA connection so USB or NFS/SMB are the only other options.

USB-disks usually aren't great and tend more to cause problems than internal disks.

All network attached drives are USB3 so was figuring having 2TB attached directly by USB should be better than USB over NFS/SMB

How would you do this?

Getting a platform that got enough drive bays and connectors to fit all your drives internally and a good onboard NIC (or even some PCIe slots for future expansion of HBAs or NICs). Then run some NAS OS and share the disksas cold storage via SMB/NFS/iSCSI to your small PVE nodes.

 

[Soogs]

I personally would mirror those two 2TB SSDs for reliability and use that mirror to store everything. Even if that means losing half of the raw capacity and that the NVMe SSD will be slowed down to SATA performance.
If you don't want to mirror I would use the first option but then make sure to have a properly tested disaster recovery plan as there is no official way to backup the PVE nodes yet and people often use problematic stuff like PBS virtualized on the same PVE node.

Depends on how much you care about data integrity and those ZFS features. Keep the ZFS recommendations in mind like lots of ECC RAM (usually not an option when using MiniPCs) and Enterprise/Datacenter grade SSDs with integrated power-loss protection (possible with MiniPCs but don't expect them to be cheap).

USB-disks usually aren't great and tend more to cause problems than internal disks.

Getting a platform that got enough drive bays and connectors to fit all your drives internally and a good onboard NIC (or even some PCIe slots for future expansion of HBAs or NICs). Then run some NAS OS and share the disksas cold storage via SMB/NFS/iSCSI to your small PVE nodes.

The only machine mirroring would be great for (in my case) is the only machine that can't do it (machine hosting virtual pfSense, pci card blocking sata connection and space).

I do generally run PBS virtualised - but backups are on a NAS, a clone of PBS exists on all nodes and I also have a baremetal PBS machine which I also run weekly backups to.

I do intend to build an ITX NAS at some point but cash is a little tight at the moment (just purchased our first house).

I will ditch the USB storage idea for now if it is at all likely to be problematic.

I am still unsure whether to keep or ditch ZFS - as a whole I do have enough RAM across lab ( 16GB, 32GB, 32GB & 64GB )
Things have felt a bit smoother since moving to ZFS but not sure if that has been a placebo.
Going back to ext4 would mean I could old off on upping the RAM as I add more systems to the project

two of the nodes currently have 2TB nvme and 2TB HDD's (instead of SSDs)
Using HDD's as the data held in them are generally temporary so thought it might be better for wear but has come at the cost of performance which is one of the main reasons for starting over... but now thinking it might not be the worst idea to stick with this and then add the SSD's to a NAS for the long term benefit that all nodes can use them

Thanks for you advice

 

[justinclift]

Using ZFS gives you extra capabilities, like easy snapshots and bitrot protection. Having the Proxmox install be done to both of the 2TB SSDs (as a ZFS1 mirror) means that if one dies it shouldn't take much effort to get things running again.

Also, when you install using a ZFS1 mirror Proxmox automatically sets things up so the left over space on those drives can be used for ISO images, templates, containers, VMs, etc.

Not sure about the ram aspect that you're talking about. While ZFS can indeed use a bunch of ram when configured to do so, it's very much optional and configurable.

 

[louie1961]

I personally would mirror those two 2TB SSDs for reliability

+1 This is what I do in my two mini PC nodes. I have two 2TB NVMe drives in a zfs mirror. This mirror holds the boot drive and the running VM/CT storage. All data is sent by NFS to my Synology. I rely on the Synology for RAID, snapshots and backups of data to the cloud, etc. I also back up my VMs and CTs to a separate NAS via a different NFS share. If I lose a Proxmox node, it is a pretty trivial matter to re-install Proxmox on a new machine, then backup all the VMs/CTs, then re-attach the Synology share, and be be back up and running in no time.

 

[Dunuin]

Not sure about the ram aspect that you're talking about. While ZFS can indeed use a bunch of ram when configured to do so, it's very much optional and configurable.

You can lower it to some amount but I wouldn't go below 2GB ARC for small systems and then its still 2GB more than ext4. And especially when HDDs are involved more RAM is great for performance. And without ECC-RAM, you can never fully trust the bit rot protection, as data corrupted in RAM and then be written to the disks can't be detected by ZFS and the scrub will tell you that everything is fine while it isn`t. But even without ECC better than running ext4 without any checksumming.

Also, when you install using a ZFS1 mirror Proxmox automatically sets things up so the left over space on those drives can be used for ISO images, templates, containers, VMs, etc.

That would work with all ZFS layouts. Even a single disk.

Having the Proxmox install be done to both of the 2TB SSDs (as a ZFS1 mirror) means that if one dies it shouldn't take much effort to get things running again.

If I lose a Proxmox node, it is a pretty trivial matter to re-install Proxmox on a new machine, then backup all the VMs/CTs, then re-attach the Synology share, and be be back up and running in no time.

Yes. Saves hours of work if you don`t have to start from scratch each time a disk dies. Especially once you customize PVE a lot because you want stuff like UPS client, monitoring agent, encryption, SMTP relay server, log collector, security hardening, ...
If it is easy to reinstall PVE, you either spend a lot of time initially creating scripts or ansible playbooks to automate all set up. Or you didn`t and left everything stock but then its not well secured nor optimized.