@fweber
I've manage to dedicate 1 node with mitigations=on and single client RDS server (who is working free of charge and should not complaint to much)
So, I'm ready to test new kernel with patch if you provide such
Right now numa balancing has been switched off and RDS server works smoothly...
Thanks a lot!
Just some highlights that drove me crazy
if disabling numa balancing is completely legitimate solution shouldn't it be disabled by default?
Then check this thread and links to virtio driver git (there are some advices from devs)
If you are able to reproduce this issue easily it would be very helpful to find a solution or workaround
I performed some tests on my cluster and can confirm that tuning vzdump.conf could be used as workaround
max average throughput of my 5 nodes cluster (with CEPH) = ~800MiB/s and PBS storage = ~300MiB/s
1) I set vzdump.conf as following:
bwlimit: 150000
ionice: 8
2) On PBS I limited input...
However with ionice=8 set in vzdump.conf I can see it in backup log
INFO: starting new backup job: vzdump --exclude 101,100,103 --notes-template '{{guestname}}' --storage PBS --mode snapshot --mailto ... --all 1 --mailnotification failure --node 063-pve-04446
INFO: Starting Backup of VM 6302...
1) It died silently. It was not even possible to log in via console (it was no freezed but there were no services avalible)
2) Let me make it clear. PBS respects bwlimit parameter in vdump.conf? In other words: if I set bwlimit in vdump.conf PBS client will limit backup speed? Am I correct...
@fiona
Is there any way to limit read/write bandwidth of PBS client (something like bwlimit in vdump.conf) ?
there is a suggestion from virtio devs:
How should I change my Vm config to incorporate this tune?
P.S. there is another thread...
How to do that? (reduce number of concurrent backups) ? In my cluster there is only 1 backup job and if I'm not mistaken PVE sterilize backups from all nodes within cluster)
P.S. right, Vm hangs after backup (when another backup in progress). Win and Linux vm affected
Not in my case. 40Gbe link. There is powerful CPU and HW RAID with SSD disks on PBS server
CEPH datastore performance is a key limited factor (of backup speed)
And one more thing:
VM "hangs" (with different event to windows syslog + ID 129 as well) when another VM is backing up (that VM is...
Thanks for the link!
I found this event ( ID 129 ) on one of my Windows VM that randomly hangs right after (during?) backup to PBS
Hope that PVE devs will take a look at this issue
Well, I’m pretty confident that this issue is not directly related to guest OS.
In my case I have not seen such issue on PVE 7.x with kernels < 6.x
However disabling numa balancing do a trick (I’m not sure with what impact on performance)
If you do have an option to try the latest advice from...
Thank for sharing. I will give it a try.
However I’m afraid this isn’t a key to the topic problem - I’m facing this issue even on hosts with only single VM (1 vm per host) and KSM disabled (it useless on such setup)
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.