From my perspective virtio scsci driver is the key here. Still waiting for any progress from virtio devs. Be free to ping them at the corresponding topic at the github (check the link in messages above)
Some images
First server (active) - noticeable memory leak (I had to drop ARC cache at the end)
Second server (exactly the same but without NFS activity - redundant storage with ZFS replicas)
There is possible memory leak with kernel 6.8.3 and nfs-kernel-server
(Check this thread: https://forum.proxmox.com/threads/memory-leak-on-6-8-4-2-3-pve-8-2.146649/ )
Same story. After upgrade to pve 8.2 (kernel 6.8.4-3) I’m facing the same memory leak.
Zfs pool with NFS shares and high IO load
(ZFS arc size limited and does not grow up)
Can confirm: kenrel 6.8.4, ksm enabled, numa balancing=1, all kernel mitigations on
Works as expected. No ICMP echo reply time increase, no any other freezes.
Finnaly!
@fweber
I've manage to dedicate 1 node with mitigations=on and single client RDS server (who is working free of charge and should not complaint to much)
So, I'm ready to test new kernel with patch if you provide such
Right now numa balancing has been switched off and RDS server works smoothly...
Thanks a lot!
Just some highlights that drove me crazy
if disabling numa balancing is completely legitimate solution shouldn't it be disabled by default?
Then check this thread and links to virtio driver git (there are some advices from devs)
If you are able to reproduce this issue easily it would be very helpful to find a solution or workaround
I performed some tests on my cluster and can confirm that tuning vzdump.conf could be used as workaround
max average throughput of my 5 nodes cluster (with CEPH) = ~800MiB/s and PBS storage = ~300MiB/s
1) I set vzdump.conf as following:
bwlimit: 150000
ionice: 8
2) On PBS I limited input...
However with ionice=8 set in vzdump.conf I can see it in backup log
INFO: starting new backup job: vzdump --exclude 101,100,103 --notes-template '{{guestname}}' --storage PBS --mode snapshot --mailto ... --all 1 --mailnotification failure --node 063-pve-04446
INFO: Starting Backup of VM 6302...
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.