Proxomox Setup: System/VMs/ZFS Cache on which SSD/HDD?

[achimcc]

Hello,

I plan to build my own server with Proxmox. I want to use 3HDDs as RaidZ1 Volume with ZFS. And I want to install some fast NVMe SSDs. But I'm still confused about an optimal Setup for Proxmox/VMs/Caching:

-I could use 2 NVMe SSDs in Raid1/RaidZ1 config with either Hardware or ZFS Raid for Proxmox System?
-Can I use the same SSD Array for VMs/Caching? Or is it advisable to use separate SSDs here?

In case that I use 2 SSD Raid(z)s: What should I put where? Like Proxmox+VMs on one and Cache/Log on the other? Or Proxmox on one distinct and VM/Cache together?

Many thanks in advance and best regards,
Achim

 

[Dunuin]

Hello,

I plan to build my own server with Proxmox. I want to use 3HDDs as RaidZ1 Volume with ZFS. And I want to install some fast NVMe SSDs. But I'm still confused about an optimal Setup for Proxmox/VMs/Caching:

Keep in mind that enterprise grade SSD are recommended for server workloads like ZFS or virtualization and normal consumer SSDs may die within months depending on your workload. Its not unusual that you will see a write amplification of factor 20 or more (if you write 1GB inside a VM 20GB or more will be written to the SSDs to store that and SSDs will be damaged with each write).

-I could use 2 NVMe SSDs in Raid1/RaidZ1 config with either Hardware or ZFS Raid for Proxmox System?

ZFS mirror = like raid1
raidz1 = like raid5
Its a good idea to use atleast a ZFS mirror or hardware/software raid1 so you don't loose any data and dont have to setup everything again if a SSD fails. If you use some kind of mirroring or parity like raidz ZFS is also able to detect and fix corrupted data.

-Can I use the same SSD Array for VMs/Caching? Or is it advisable to use separate SSDs here?

Basically yes. Running Proxmox and VMs from the same SSD array/pool isn't a problem. You could use them for caching too, but I wouldn't recommend that. Most of the time SSD caching with ZFS isn't useful and may even slow down your system or destroy everything on your HDDs if the SSD fails. If you really want to use SSDs for that I would recommend buying two additional good high endurance SSDs for that to use them mirrored as SLOG and/or "special device". If you want read caching just buy more RAM.

 

[achimcc]

Basically yes. Running Proxmox and VMs from the same SSD array/pool isn't a problem. You could use them for caching too, but I wouldn't recommend that. Most of the time SSD caching with ZFS isn't useful and may even slow down your system or destroy everything on your HDDs if the SSD fails. If you really want to use SSDs for that I would recommend buying two additional good high endurance SSDs for that to use them mirrored as SLOG and/or "special device".

Oh, great, good to know! Then I'll get to SSDs first for Proxmox/VMs and might add another Array later on! Thanks!

Keep in mind that enterprise grade SSD are recommended for server workloads like ZFS or virtualization and normal consumer SSDs may die within months depending on your workload.

Can you recommend an Enterprise grade NVMe SSD which is not to expensive? I think I read somewhere, that WD Black SSDs are already quite good and reliable, despite being consumer SSDs?

 

[Dunuin]

Oh, great, good to know! Then I'll get to SSDs first for Proxmox/VMs and might add another Array later on! Thanks!

Its no problem to add/remove them later.

Can you recommend an Enterprise grade NVMe SSD which is not to expensive? I think I read somewhere, that WD Black SSDs are already quite good and reliable, despite being consumer SSDs?

There is a reason why they are so expensive. If you buy a 1TB SSD that isn't just 1TB of flash storage. SSDs always got more storage that you can't use. This additional storage is used to replace failing flash cells so the drive lives longer. A 1TB consumer grade SSD might use 1,2TB and a good enterprise SSD 1,7TB. Often they also use more expensive but faster and more durable flash cells like SLC/MLC and not cheap TLC/QLC like consumer grade SSDs. And enterprise grade SSDs got additional backup power (powerloss protection) so they can use the RAM cache even for sync writes what extremly reduces the write amplification if you got workloads like DBs with a lot of small sync writes.
I bought my enterprise grade SSDs second hand so they weren't more expensive than new consumer SSDs. I first got some new consumer Samsung 970 Evos but needed to remove them because they wouldn't surive a year.

WD black SN750 1TB - 130€ = 600 TBW
Intel SSD DC S3710 800GB - 360€ new / 110€ used = 16900 TBW
You pay tripple the price but they SSD will live 35x longer. So in the long run the enterprise SSDs are way cheaper because you need to replace them less often.

 

[achimcc]

Its no problem to add/remove them later.

There is a reason why they are so expensive. If you buy a 1TB SSD that isn't just 1TB of flash storage. SSDs always got more storage that you can't use. This additional storage is used to replace failing flash cells so the drive lives longer. A 1TB consumer grade SSD might use 1,2TB and a good enterprise SSD 1,7TB. Often they also use more expensive but faster and more durable flash cells like SLC/MLC and not cheap TLC/QLC like consumer grade SSDs. And enterprise grade SSDs got additional backup power (powerloss protection) so they can use the RAM cache even for sync writes what extremly reduces the write amplification if you got workloads like DBs with a lot of small sync writes.
I bought my enterprise grade SSDs second hand so they weren't more expensive than new consumer SSDs. I first got some new consumer Samsung 970 Evos but needed to remove them because they wouldn't surive a year.

WD black SN750 1TB - 130€ = 600 TBW
Intel SSD DC S3710 800GB - 360€ new / 110€ used = 16900 TBW
You pay tripple the price but they SSD will live 35x longer. So in the long run the enterprise SSDs are way cheaper because you need to replace them less often.

But the Intel SSD DC S3710 800GB is a 2.5" SATA SSD? Is there any recommendation for a comparable NVMe SSD?

 

[Dunuin]

Intel DC P3700 for example is NVMe and many other U.2 drives (you need an M.2 to U.2 cable to use them). But if you want NVMe thats really expensive. The "Intel DC P3700 800GB for example is 1340€.

The Proxmox Team published a ZFS Benshmark some weeks ago: https://www.proxmox.com/en/downloads/item/proxmox-ve-zfs-benchmark-2020

 

[achimcc]

Intel DC P3700 for example is NVMe and many other U.2 drives (you need an M.2 to U.2 cable to use them). But if you want NVMe thats really expensive. The "Intel DC P3700 800GB for example is 1340€.

The Proxmox Team published a ZFS Benshmark some weeks ago: https://www.proxmox.com/en/downloads/item/proxmox-ve-zfs-benchmark-2020

Okay, thanks, good to know! A NVMe ssd (or two) are not affordable for me I guess I'll go with something like the WD Black. I mean, I can buy two in a Raid configuration and replace one IF it dies? And would still save a lot, even iff I've to do this a few times? Or would you recommend 2.5 SSD instead? I guess that the performance would be much worse?

 

[Dunuin]

You can try the cheap M.2 SSDs. As long as you are using a zfs mirror you won't loose any data if a SSD fails. But check the smart attributes regularily and be prepared to replace them sooner or later. They might work several years or just some months. Really depends on your workload. My homeserver is writing about 1TB per day to the flash so I would need to replace both every 2 years.
And just because they are faster on paper it doesn't mean that they will perform better than a SATA enterprise SSD on continous utilization and random reads/writes or sync writes. Consumer SSDs are fast for short bursts of sequential writes. As soon as as RAM cache gets full or you do alot of random writes or sync writes they get really slow. My Evos got speeds up to 3.400 MB/s on paper but for random writes they got down to just 30MB/s.

 

[mmenaz]

But the Intel SSD DC S3710 800GB is a 2.5" SATA SSD? Is there any recommendation for a comparable NVMe SSD?

Try Kingston DC1000M ('M' and NOT 'R'!!!) that is U.2 format (you need to buy a M.2 adapter or a pcie adapter), 960GB is around 300$, Power Loss protection and 3500 TBW (1DWPD / 5 years), I've the 1.92TB model, there is also a 3.84 one.

 

[achimcc]

Thanks for the hint! Actually, since I'm building a home server and I'm on a low budget, I decided to buy two Seagate Ironwolf 510 SSDs with 480gb each! They seemed like a good compromise of TBW and price. And they have power loss protection. Plus, I can use them on my mainboard (Gigabyte C246 - WU4 ) without extra adapter.

 

[Dunuin]

Thanks for the hint! Actually, since I'm building a home server and I'm on a low budget, I decided to buy two Seagate Ironwolf 510 SSDs with 480gb each! They seemed like a good compromise of TBW and price. And they have power loss protection. Plus, I can use them on my mainboard (Gigabyte C246 - WU4 ) without extra adapter.

Don't look like they got powerloss protection. Only M2 SSD I can find where it is mentioned in the datasheets is the Intel DC P4801X and that one is problematic, because there are chips on both sides so it won't fit on every mainboard and you can't cool the chips on the lower side. M.2 is just to small to place a big capacitator for powerloss protection. Thats why all enterprise SSDs only use U.2 and not M.2.

So keep the write amplification in mind if you got sync writes. Without the powerloss protection it can be extremely high.

 

[achimcc]

Well, actually the Asus C246M WU4 Mainboard is not starting at all with any of the two Ironwolf 510 being plugged in I own another NVMe SSD, a Sandisk A400 256gb and with this it works flawlessly. I tried to update the Bios and I also tried a few different BIOS Settings. And I can confirm that both Ironwolf SSDs are working fine without problems when I put them into an external M.2 SSD case.

It seems to me like a incompatibility issue, but just in case that someone here has any idea what I could try before buying other SSDs, I would be highly thankful

 

[mscott89]

Intel DC P3700 for example is NVMe and many other U.2 drives (you need an M.2 to U.2 cable to use them). But if you want NVMe thats really expensive. The "Intel DC P3700 800GB for example is 1340€.

The Proxmox Team published a ZFS Benshmark some weeks ago: https://www.proxmox.com/en/downloads/item/proxmox-ve-zfs-benchmark-2020

The ZFS looks really promising, will probably be worth picking up tbh.

 

[mmenaz]

Well, actually the Asus C246M WU4 Mainboard is not starting at all with any of the two Ironwolf 510 being plugged in I own another NVMe SSD, a Sandisk A400 256gb and with this it works flawlessly. I tried to update the Bios and I also tried a few different BIOS Settings. And I can confirm that both Ironwolf SSDs are working fine without problems when I put them into an external M.2 SSD case.

It seems to me like a incompatibility issue, but just in case that someone here has any idea what I could try before buying other SSDs, I would be highly thankful

I have a Asus MB too (ASUS Prime X570-Pro) and a U.2 Kingston DC1000M NVMe ssd. To be more "compatible" I bought the "Asus Hyper Kit" (M.2 to SFF-8643 converter for U.2 cable). It worked fine until I updated the DC1000M firmware, after that the HyperKit did not see the SSD anymore.
I then bought a cheap, not branded, M.2 to SFF-8643 adapter and everything works flawlessy!
Just to say that compatibility is not as easy as it should be.
Btw, with NVMe that are on the PCIe bus, plugging and unplugging makes my ethernet ports change name. I had to set systemd to use the "mac address" naming convention to have a predictable (and stable regarding names) setup.