Question about disk config

[S0mE1]

Hello everyone,

I'm new here and if my post isn't in the right place or I don't respect certain rules, please excuse me.

I would like to install Proxmox VE on a machine with 3 disks and I don't know what is the best choice for partitioning.

To be more precise, I'm in a homelab and not in production, the machine is a Lenovo m80q Gen3 (i5-12500t) and the 3 disks are made up of 2 nvme (pcie 4.0) and a Sata SSD.

According to my research, it's better to install the OS on the SATA and keep the NVMEs for the virtual machines, but should I use ZFS? Would you advise me to use raid 1 with the NVMEs? Or 1 NVME disk for the VMs and the second which I use as a cache?

What would you do in my place?

In view of the information you have, would you recommend an NVME drive in particular? I was hesitating between the crucial P3 Plus and the Samsung 980 Pro.

I'd also like to manage the backup of my VMs on another device, so I don't need to include any VMs backup in the 3 disks, and all my VMs will largely fit on 1 NVME disk only (like 1tb disk).

If you need any further information, I'll be happy to answer you.

Thanks in advance for your answers, any advice is good to take.

Matt.

 

[leesteken]

What would you do in my place?

I would (and do) use ZFS with enterprise NVMe (with PLP), mirrored (so a single failure or corruption can be fixed automatically). Use the SATA for cold storage. Proxmox will probably be fine on it but Proxmox writes a lot and might wear is out quickly.

I'd also like to manage the backup of my VMs on another device

Smart! I prefer backups on ZFS (to detect bitrot) on multiple types of storage and also a copy off-site.

 

[Dunuin]

I was hesitating between the crucial P3 Plus and the Samsung 980 Pro.

Especially in case of ZFS an enterprise SSD would be highly recommended when hitting SSDs with server workloads. So something like a Samsung PM9A3 in case M.2 22110 formfactor will fit. Those consumer SSDs will wear very fast and perform nearly as slow as a HDD when hitting it with sync writes because of the missing power-loss protection, so those can't make use of the DRAM cache, of consumer SSDs.
I would mirror those NVMe SSDs with ZFS and use them for system + VM/LXC storage. I wouldn't use the SATA SSD for anything important as you can't mirror it without another disk, so I would probably use it to store backups.

 

[S0mE1]

I would (and do) use ZFS with enterprise NVMe (with PLP), mirrored (so a single failure or corruption can be fixed automatically). Use the SATA for cold storage. Proxmox will probably be fine on it but Proxmox writes a lot and might wear is out quickly.

Smart! I prefer backups on ZFS (to detect bitrot) on multiple types of storage and also a copy off-site.

Thank you for your answers. Based on your experience, do you have any other type of NVME to recommend other than the Samsung?

Given that I am not in "Prod", I can take a SSD SATA enterprise to install the OS (which will be more resistant than public disks)?

As far as backup is concerned, I was initially thinking of an external hard drive that I'd connect via USB in the meantime, and then investing in a homemade NAS or Synology, what do you think?

I want to get used to good practice, but on the other hand, I'm not even sure that my machine will be on all the time, apart from a few Linux and Windows vms for testing, I won't be running any games or video services.

 

[Dunuin]

Given that I am not in "Prod", I can take a SSD SATA enterprise to install the OS (which will be more resistant than public disks)?

Sure you can. But keep in mind that SSDs are consumables that will fail sooner or later. I had multiple consumer SSDs here failing within a year. Others failed only after a decade. And that there is no built-in way to backup the system disk or export/import the PVE config files. When your system disk fails and it is not mirrored you are screwed and have to wait for a replacement SSD, install PVE again and start from scratch. Having a copy of the old config files might help a bit to speed things up. But it is still annoying that you maybe can't use your server for a couple of days.

Thank you for your answers. Based on your experience, do you have any other type of NVME to recommend other than the Samsung?

There aren't many options if you want M.2 NVMe as M.2 is a terrible formfactor for any serious stuff. Good SSDs use U.2 instead. The only options for M.2 are: Samsung PM9A3/983, Micron 7300/7400/7450 Pro/MAX, Seagate Nytro 5000, Intel P4801X/P4511, Kingston DC1000B
For anything else you need free PCIe Slots or M.2 to U.2 adapters.

As far as backup is concerned, I was initially thinking of an external hard drive that I'd connect via USB in the meantime, and then investing in a homemade NAS or Synology, what do you think?

For backups I would build a PBS server.