Storage Upgrade Recommendations

[meichthys]

We have recently realized (as many others on this forum eventually realize) that our consumer grade SSDs just aren't cutting it when it comes to everyday IOPS performance:

• Terrible random write IOPS below 20 using the proxmox zfs benchmark tests
• System load spikes when running all our VMs simultaneiously.

We are looking for a recommended path forward based on our existing setup.

## Current Setup:

• Use Case: Approximately 10 general purpose VMs like (Nextcloud, Docker, UCS domain controller, Windows 10, etc)
• Cluster: We have two other nodes in the cluster that are older machines that we've been using to run some of the VMs to limit load on this newer system
• Chassis: Dell Precision 3650 Tower
• PSU: 300W
• CPU: Intel i9 - 11900
• RAM: 4x 32GB (128GB Total) Crucial 3200MHZ UDIMM CL22 DDR4
• STORAGE: 4x 4TB Samsung QVO 870 SATA SSD <-- We are looking to replace these
  • Storage Configuration: ZFS Striped Mirror

## Additional Notes

• Settings zfs `sync=disabled` does drasitcally improve the write performance of the system, and we do have battery backups, but it's something we would like to avoid doing for the long-term.
• Currently using all 4 SATA ports (which seem to all be on the same pcie lane although the documentaiton isn't clear) on the machine which makes me wonder if there is a bottleneck there.
• The are 3 empty M.2 (2280) ports which I'm thinking we should use with better data-center grade SSD storage devices (with power-loss-protection). After reading many of @Dunuin 's great storage-related posts, it looks like there are few if any m.2 ssds (especially in the 2280 form factor that our chasiss supports).

## Next Steps
We are thinking of purchasing the following to improve the situation. Could you please provide some critiques or recommendations other than the following?

• Purchase x2 4TB U.2 Samsung PM9A3 TLC V-NAND SSDs
• Purchase x2 M.2 to U.2 adapters
• Repurpose old consumer grade Samsung SSDs as a Proxmox Backup Server storage pool
  • Is this a reasonable use case for these drives, or will the poor IOPS be an issue here too?

 

[Dunuin]

Repurpose old consumer grade Samsung SSDs as a Proxmox Backup Server storage pool

• Is this a reasonable use case for these drives, or will the poor IOPS be an issue here too?

PBS needs IOPS performance as everything will be stored as millions of small (<4MB, usually more like 1-2MB) files. But PBS allows for incremental backups so bad write IOPS are maybe not such a big problem in case your data isn't changing that much. And PBS is doing stuff like compression, bit rot detection, deduplication, syncing, encryption and so on the software level. So not that bad when not using ZFS there. But if you want to use ZFS performance will be as bad as now with PVE.

 

[showiproute]

@meichthys
I had the same issues with consumer grade SSDs.

If your mainboard has a PCIe x8 slot and you are not using a GPU AND if your mainboard additionally supports bifurcation you can also use a PCIe to U.2 card like this

I am using the PCIe card to run 2x Kingston DC1500M U.2 SSDs to replace some SATA SSDs.

 

[meichthys]

@showiproute Thanks for this We were wondering if such a configuration was possible but did not investigate it further when we found the m.2 to the u.2 adapter.

 

[showiproute]

@showiproute Thanks for this We were wondering if such a configuration was possible but did not investigate it further when we found the m.2 to the u.2 adapter.

Initially I used on PCIe to M.2/U.2 (I use both formfactors) per PCIe slot but found out that both of my server (Supermicro) supports PCIe bifurcation.
So I bought a PCIe x16 card which takes up to four M.2 SSDs.
Proxmox sees each as a separate device and can be used.