We are planning our Proxmox VE 4 cluster, and decided on ZFS (provided that snapshot backups will work for both KVM and LXC guests). We plan to use small nodes (4C/8T CPU, 32 GB RAM, 5-8 disk RAIDZ2 in the 3-6 TB usable range). We will employ one SSD drive per node as ZIL and L2ARC (if using 2, ZIL will be mirrored, L2ARC striped), and need to decide on how big SSDs to buy. We were planning to buy 512GB SSDs (using 16GB for ZIL, rest for L2ARC), but it looks like from people on the web that with only 32GB memory, we might be better off with much smaller L2ARC partitions.
According to this ZFS on Linux issue, one guy with 4GB of ARC space has problems managing a 125GB L2ARC.
https://github.com/zfsonlinux/zfs/issues/1420
There is of course many info on the web, advice for ARC/L2ARC ratio ranges from 1:5 to 1:40, and I know there is no hard rule, it depends on your block size and use case.
So I am looking for some practical advice from people running ZFS with L2ARC on SSD:
- are you running KVM only or containers as well
- are you using zvols or zpools, if zvol what filesystem
- how big are your arrays
- what is your block size
- how big is your ARC (is it fixed size or dynamic)
- how big is your L2ARC SSD cache
- what are your arcstats and l2 header stats
According to this ZFS on Linux issue, one guy with 4GB of ARC space has problems managing a 125GB L2ARC.
https://github.com/zfsonlinux/zfs/issues/1420
There is of course many info on the web, advice for ARC/L2ARC ratio ranges from 1:5 to 1:40, and I know there is no hard rule, it depends on your block size and use case.
So I am looking for some practical advice from people running ZFS with L2ARC on SSD:
- are you running KVM only or containers as well
- are you using zvols or zpools, if zvol what filesystem
- how big are your arrays
- what is your block size
- how big is your ARC (is it fixed size or dynamic)
- how big is your L2ARC SSD cache
- what are your arcstats and l2 header stats
Last edited: