New Proxmox build

[karlm]

Hello,

I have a new setup arriving soon,

I have a new case which holds 10 3.5 inch disks and 3 2.5.
I have (10) 3.5" 7200RPM 2TB datacenter SATA drives. (Well, I have 14, so 4 for spares)
I have (3) 2.5" 400GB SAS SSDs for the Proxmox install (Well, I have 4, so one for a spare)

I want the 10 3.5's set as a RAID 10 configuration as they are not new drives.

I have an AMD Ryzen 7 5700 CPU, an 850W PSU, an ASRock motherboard, 64GB of DDR5 5200MHz ram, a 16 port SATA expansion card, and a 360mm liquid cooler.

Should I install Proxmox on the SSD's before I install the SATA card and drives, or install all the hardware first?

And, how do I install or configure Proxmox so that it uses the three SSD's for failover and the 10 spinners in a RAID 10 configuration?

Right now I have Promox on an old Dell rack mount with 12 of the 3.5" SSD's and did not use a seperate disk for the proxmox install so I am a little lost

I want at least 4TB of the 3.5's to be setup as a NAS but the rest can be used for LXCs and VM's

Thanks

Karl

 

[SteveITS]

You can choose on what disk(s) to install Proxmox during the install.

You can use ZFS mirroring for boot and /or data drives if that’s what you mean. As long as you’re not using a RAID controller.

 

[louie1961]

I don't think you really get any benefit from having three boot drives. In my opinion a pair in a mirror is best. A full Proxmox install. My boot drive on my main node has only has 73GB on it currently, including 6 VMs and 24 containers. Mirrored VDEVs are going to be faster than a radiZ1 or raidz2 set up. Its probably best if you set up all four of your SSDs in 2 mirrored VDEVs, and then stripe them. so you'll have 800GB of storage to use for VMs. As far as the spinners, ZFS doesn't really have a raid 10 mode, per se. You can approximate it by striping mirrored VDEVS (like I suggest for your SSDs), but I am not sure that would be the best use of those. Spinners really don't give great performance on VMs, so I would probably use them for data. if you take that approach, a raidz1 or raidz2 might be a better choice for the spinners.

 

[UdoB]

And another completely a little bit different approach:

Install PVE only on all spinners during setup. In pairs of mirrors of course, similar Raid10. Each and every one will be bootable later on, maintained by the PVE tooling.

After initial setup add two fast SSDs as a Special Device. (Also mirrors; if your paranoia level is high opt for a triple mirror.) I am not sure, but this has probably to be done via CLI. Do this before filling the pool with much data.

That Special Device will store all metadata only, by default. This reduces the need of physical head movements on the data drives drastically. Because there are multiple of these per write it speeds up the whole pool by a factor of three to ten. For most workloads, not necessarily for all.

The goal is to have some speed of the SSD and the pure size of the rotating rust combined in one single pool. For me this works really great.

That SD can be relatively small, usually below one percent of the data part. 10 TB data --> 100 GB SD. If those SSDs are much larger it should be configure to hold "Small Blocks". The parameters are difficult to choose and as the SD CAN NOT GET REMOVED they need to be chosen carefully. This has to be tested extensively before going productive.

Do NOT prepare an SLOG or a Cache using "not used space" on these SSDs, except you have very specific use cases where these are proven to be useful. Usually SLOG/Cache does NOT work the way most users expect.

If you want to separate Data from Operating System (which is still recommended) use two disks in a mirror for this. While HDD is okay for the OS alone, you need to use SSDs if the VMs shall live here. Then add the large data pool implementing the described scheme.

Have fun