Suggestion for new proxmox server setup

[Lousyclient]

First time post so bear with me if I get anything wrong.

Im looking for suggestions on what to do with a few SSDs that I have for my system, should I just create a separate storage pool for just the SSDs

I’m currently in the process of building a NAS/server that will be running proxmox on it. My hardware specs are as listed
CPU: AMD 9600x
Ram ddr5 32 gb 6000 speed
Motherboard: asrock a620ai with WiFi
Storage connection hardware: LSI login 179356 with sas to sata cables.
Storage:
2 m2 ssd 512 gb <- will be raided in raid 1 for proxmox OS
3 250 gb sata ssd <- still planning on what to do with this
1 500gb sata ssd <- still planning on what to do with this
6 2 tb sata hdd <- will be raided in zfs raidz-1 this will be my main storage for the vms/containers

 

[leesteken]

6 2 tb sata hdd <- will be raided in zfs raidz-1 this will be my main storage for the vms/containers

Using the slowest drives (with also the lowest IOPS) in a slow ZFS configuration that has the lowest IOPS (especially writes) is a poor choice. See also https://forum.proxmox.com/threads/fabu-can-i-use-zfs-raidz-for-my-vms.159923/ . ZFS raidz1 (on consumer SSD or any HDD) is not at all the same as hardware RAID with BBU.
Proxmox itself runs fine on slow old HDDs (and only requires 8GB and the rest of the space can be used for cold storage). It's the (many) VMs that need fast drives with lots of IOPS, like enterprise SSDs with PLP.

Ram ddr5 32 gb 6000 speed

RAM is (almost) always the limiting factor, as Proxmox does NOT work well with memory overcommit, and 32GiB is not much nowadays. (EDIT: NOT was missing).

 

[Johannes S]

RAM is (almost) always the limiting factor, as Proxmox does work well with memory overcommit, and 32GiB is not much nowadays.

This should have been "doesn't work well" shouldn't it?

OP: Depending on the capacity and redundant you want to achieve I would go with these setup:

- Setup the HDDs as striped mirror ( RAID10) or RAIDZ1 for OS install and raw data store ( vm application data ( NOT VM OS), iso images, templates etc). Striped mirror will have better performance, RAIDZ1 give more capacity and failure-safety, calculate with https://www.raidz-calculator.com/

- add the three 250GB SSDs as special device to your hdd pool after installing Proxmox VE to speedup medatadata access ( useful for applications with a lot of small files ): https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_special_device

Please note that after adding the special device you can't remove or loose it without breaking the pool.

- Create one mirror out of your three 500GB SSDs for vm storage.

Of course this can be varied, maybe you don't want to sacrifice the smaller ssds as special device but want to have another mirror for VM storage.

You might also want to split the hdds in one mirror for the OS and another mirror, striped mirror ( RAID10) or RAIDZ1 as cold data storage ( so you can reinstall the OS without erasing the data). This might be the best option, but then you can only add the special device to one of them.

You could also setup the ssds not in mirrors to get more capacity at the cost of relieability. I wouldn't do it though, three-way-mirrors will give you a bigger relieability and better performance