Are the 32gb of memory enough to do zfs - and do i need to make any speciel settings based on my amount of memory ?
Yes, 32 GB in total is fine for that workload and disk size. Per default, ZFS will use at most 16 GB of RAM. I'd change only if needed. Run your server and if everything is running smoothly, don't change anything.
You will in fact need to change a few things, because the default Proxmox ZFS setup is far from being optimized, and can lead to unstable servers (during high memory pressure and/or when vzdump backups are running).
1. Limit the ZFS ARC size
We aggressively limit the ZFS ARC size, as it has led to several spontaneous reboots in the past when left unlimited. Basically, we add up all the memory the system uses without caches and buffers (like all the KVM guest's maximum RAM combined), subtract that from total host RAM, and set the ARC to something a bit less than that, so it has to compete with system cache only. For example: if on a 32GB server the maximum RAM allocation of KVM guests is 25 GB, we set the ARC upper limit to 5GB (leaving 2GB for anything else). We also set a lower limit of 1GB to the ARC, as it has been reported that it helps performance for some reason. Note that the values are in bytes, so 5GB = 5x1024x1024x1024
To do that, you have add the following lines to
/etc/modprobe.d/zfs.conf
Code:
options zfs zfs_arc_max=5368709120
options zfs zfs_arc_min=1073741824
and after that run the following command and reboot:
Looking at the ARC of this very server with
arc_summary.py you can see it stays between the limits:
Code:
ARC Size: 30.72% 1.54 GiB
Target Size: (Adaptive) 30.72% 1.54 GiB
Min Size (Hard Limit): 20.00% 1.00 GiB
Max Size (High Water): 5:1 5.00 GiB
ARC Size Breakdown:
Recently Used Cache Size: 35.27% 554.85 MiB
Frequently Used Cache Size: 64.73% 1018.10 MiB
2. SWAP on ZFS zvol
You also have to make sure that swap behaves well if it resides on a ZFS zvol (default installation places it there). Most important is disabling ARC caching back the swap volume, but the other tweaks are important as well (and endorsed by the ZFS on Linux community):
https://github.com/zfsonlinux/zfs/wiki/FAQ
Execute these commands in your shell:
Code:
zfs set primarycache=metadata rpool/swap
zfs set secondarycache=metadata rpool/swap
zfs set compression=zle rpool/swap
zfs set checksum=off rpool/swap
zfs set sync=always rpool/swap
zfs set logbias=throughput rpool/swap
You can verify these settings by running:
3. Virtual memory settings
Set these on the Proxmox host to increase stability and performance of your VMs during vzdump backups, by adding these lines to the end of
/etc/sysctl.conf and reboot:
Code:
vm.min_free_kbytes = 262144
vm.dirty_ratio = 5
vm.dirty_background_ratio = 1
vm.swappiness = 1
You should use these values in your KVM Linux guests as well, although for min_free_kbytes the approximate rule is to use 65536 kbytes for every 4GB of system memory. (So for a 1-4GB RAM VM use 65536, 4-8 GB use 131072, more than that 262144. You don't need to use higher values.)
