Search results

@fiona Is there any way to limit read/write bandwidth of PBS client (something like bwlimit in vdump.conf) ? there is a suggestion from virtio devs: How should I change my Vm config to incorporate this tune? P.S. there is another thread...

In my case switching to threads does not help(

How to do that? (reduce number of concurrent backups) ? In my cluster there is only 1 backup job and if I'm not mistaken PVE sterilize backups from all nodes within cluster) P.S. right, Vm hangs after backup (when another backup in progress). Win and Linux vm affected

Not in my case. 40Gbe link. There is powerful CPU and HW RAID with SSD disks on PBS server CEPH datastore performance is a key limited factor (of backup speed) And one more thing: VM "hangs" (with different event to windows syslog + ID 129 as well) when another VM is backing up (that VM is...

Thanks for the link! I found this event ( ID 129 ) on one of my Windows VM that randomly hangs right after (during?) backup to PBS Hope that PVE devs will take a look at this issue

Well, I’m pretty confident that this issue is not directly related to guest OS. In my case I have not seen such issue on PVE 7.x with kernels < 6.x However disabling numa balancing do a trick (I’m not sure with what impact on performance) If you do have an option to try the latest advice from...

Thank for sharing. I will give it a try. However I’m afraid this isn’t a key to the topic problem - I’m facing this issue even on hosts with only single VM (1 vm per host) and KSM disabled (it useless on such setup)

@spirit But how could frequency increase if we disable p-states and pin frequency of all the cores to maximum within given TDP?

I would say that KSM is out of scope. I'm facing this issue even with 1 VM per node (2 sockets board) with almost all cores assign to VM. On such nodes I disable KSM as service (it's useless on such setup)

The difference is that ICMP echo reply spikes become packet drops and micro freezes become RDS session disconnects/reconnects In other words with mitigations=on the problem is more representative

Many thanks for clarification. Is there anything alike for Intel?)

@spirit Thanks for sharing! Have you performed some tests and why did you decide to disable pstate and fix frequency at max? (comparing to rely on power management and frequency boost)

Correct. Some data from that node below mitigations=on, numa_balancing=1, KSM active, RDP disconnects / ICMP packets drop root@pve-node-04348:~# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53...

I was able to rerun test with mitigations=on and can confirm it has huge impact. With mitigations=on i see icmp packet loss and rdp disconnects rather than icmp reply time increase and micro freezes However disabling numa balancing helps even with mitigations=on

Unfortunately, there is no difference with 6.5 kernel. Just tested Still stay with mitigations=off and numa_balancing disabled P.S. will try to rerun the latest tests with mitigations enabled (but it's gonna be very painful if it gets worse)

Same node, same load but numa_balancing disabled mitigations=off, numa_balancing=0, KSM active, no RDP freezes root@pve-node-03486:~# echo 0 > /proc/sys/kernel/numa_balancing root@pve-node-03486:~# ./strace.sh 901 strace: Process 2896 attached strace: Process 2896 detached % time seconds...

mitigations=off, numa_balancing=1, KSM active, RDP freezes root@pve-node-03486:~# cat /proc/sys/kernel/numa_balancing 1 root@pve-node-03486:~# ./strace.sh 901 strace: Process 2896 attached strace: Process 2896 detached % time seconds usecs/call calls errors syscall ------...

Here we go: Node: CPU(s) 48 x Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2 Sockets) Kernel Version Linux 6.5.11-6-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-6 (2023-11-29T08:32Z) RAM: 8*32=256 DDR3 (distributed equally - 4*32 per socket) PVE Manager Version pve-manager/8.1.3/b46aac3b42da5d15 10...

@fweber any news?

Totally agreed. Satadoms on supermicro are nightmare. Try to avoid it