- I limited my ARC to 8GB a week ago and my server has 10GB of memory free. I have 2 ssds and 4 magnetic drives using ZFS (total of 30TB), should I lower this number further? - 3. Limit your ARC usage ( you can found on pve db ) ( for 16GB ram 2 GB enough )
The lower the ARC, the less performant is your system. According to general rule that can be read all over the internet for ZFS, you should have 1-2 GB of ARC per 1 TB of data stored in your pool - if you want performance. You can get away with any lower number, but you will have a lot of cache misses. To optimize this, the general approach would be to try out different ARC settings while running standardized tests of your usual workload. Anything else is just asking the oracle.
- This server is for home use and only a few users use it sparingly for different services. Are you saying do not use a separate LOG or CACHE disk for my spinning drives? 6. Do not use LOG or CACHE disk if many many people can not access to your system at same time.
SLOG will help if you have SYNC writes, if you do not have them, you will not gain any performance improvement.
As stated previously, L2ARC can improve your performance, but most of the time it does not, because you have a very, very limited ARC with only 8 GB and L2ARC will eat away from this.
Best is to try out for yourself. If your SSD is fast and at an enterprise level (I've never heard of Seagate SSD before), this could work. If the SSD is not good, you will not see an improvement at all - you can even worse your results. Yet, try for yourself.
- Ok so before I had Discard enabled and then disabled it as suggested in this thread. How do I configure it to only run daily instead of after every delete? - only applicable for KVM VM and it'll waste storage space, so I would and also suggest to use it e.g. daily and not on every delete.
If you disabled it at the VM level, you cannot use it inside of the VM. The idea to disable it at a VM level - as you already experienced yourself - does not yield any performance gain and is for the use of ZFS counter productive, because you waste space. ZFS is faster if you have stored less in your pool. Discard takes care of releasing unused space inside of your VM (e.g. deleted data) and freeing it inside of your ZFS dataset (only applies to VMs not containers).
Please refer to the guest OS manual to setup discard as a scheduled maintenance task.
- Ok so it sounds like the KSM suggestion will not help my Windows VM - also only applicable to KVM VM due to a special syscall that only KVM/QEMU uses.
KSM can only help if you have multiple VMs with the same OS and only if you use KVM VM. Windows can only be virtualized by a KVM VM, so you already have this use case.
- My system has enough ram and I do not think I have ever needed SWAP to be used yet. Do you still suggest I do this? - Always a good suggestion, zram is extremely useful. I replaced all physical swap disks/partitions by zram in my systems.
Check swap usage with free and zram is optional. It is however better for overall performance than a storage backed swap drive. If you don't swap and you do not intend to do so, just keep your current setting. Without any swap, your system will kill processes if you run out of memory, just be aware and monitor dmesg.
Windows or Linux or ESXi guest...