Server Design for Proxmox w/ ZFS

[Steve88]

Hi,

I am currently configuring a server with the goal of running Proxmox for several Windows 10 VMs and a Linux Server. The total amount of disk space should be approx. 4 TB.

My current favorite design would be Raid-Z2 with 6 x 1TB SAS HDD for the VMs. Now I am struggeling a bit with ZFS caching and Proxmox installation. My initial idea was to install proxmox on a DOM and use 1 SSD for L2ARC and 2 SSDs in Raid1 for ZIL. Then I heard that it is probably a bad idea to install Proxmox on a DOM for endurance reasons. Would it be possible to install it on the mirrored SSDs in a separate partition (on top of Debian in this case) and would there be performance implications? Would a separate SSD be the better solution? Or is there any other good option I am not considering right now?

Many thanks in advance!
Cheers

 

[Romsch]

Hi. you really mean 4 TB RAM?
What do you thinking about CEPH in a three node cluster? Thats perfect and it is a fast high availibility solution for all VMs in the cluster.
Best regards,
roman

 

[Steve88]

Hi Roman, no, sorry I meant disk space (just corrected my post). The amount of available RAM should be 128 GB in total.

Actually I have little clue about CEPH. But as you suggested it, I will take a look into it.

Anyway, are there also any suggestion about a good design for ZFS usage?

 

[Romsch]

Hi Steve,
it depends on the requirements to be given.
Should ceph come into question as storage solution, there must be at least three nodes (servers).
Right now in the fresh version 5.4 it is possible to completely configure ceph via the interface. ceph has almost only advantages, here should also no RAID for the ceph hard drives disks - except with the operating system Proxmox.
If you "only" have one server for proxmox, ZFS with SSD-Disk works very fast on a local node.

Best regards, roman

 

[Steve88]

Many thanks for the information! Then I guess I will stay with ZFS as I will only have this one server.

So for ZFS, what do you think about reserving 32 GB ARC and a 128 GB mirrored ZIL as many sync writes are expected. Proxmox on a separate SSD. Would an L2ARC make sense in this config?

 

[alexskysilk]

you dont need that much ZIL; consider underprovisioning your SSDs so at least they live longer. Have a look here: https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/

With current versions of zfs, slog failure is not fatal; you may consider not bothering with the mirror depending on your tolerances.

Would an L2ARC make sense in this config?

Not usually, although some use cases can see benefit depending on amount of RAM in the system. In a nutshell, when using l2arc A large fraction of the l2arc size is required in RAM for metadata so the end result could be slower (or not meaningfully faster) if you just left ARC in RAM. Search the interwebs for benchmarks most resembling your load type, but generically I'd say dont bother.

 

[Steve88]

Many thanks for the link and the information! I guess I'll go for a little SSD for ZIL and some additional RAM for ARC instead of L2ARC.

Thank you so much for your help!

 

[LnxBil]

Proxmox on a separate SSD.

Why don't you just install it on your main ZFS pool? Those few GB don't need extra hardware.