Super slow, timeout, and VM stuck while backing up, after updated to PVE 9.1.1 and PBS 4.0.20
- Thread starter Heracleos
- Start date
Me too. All my PVE nodes run with a bond lacp 803.2ad layer 3+4 with MTU 1500, on two 10g cards.
The PBS is a VM that runs on one of the nodes, even though the virtual disks are qcow2 files on a NAS via NFS.
So, with a 9.1.1 node and 6.14.11-4-pve kernel, I can restore without any problem.
PS: i had not problems with mass live migration, and I use always the same bond of 4x10Gbit with vlans and mtu 9000
I've seen several mention the Intel 82599ES NIC which is also what I'm running in my dev PBS Server (and PVE Hosts). 20GB LAGG's as well.
Given the drop in Disk I/O it is certainly pointing towards a networking issue on the surface.
Given the drop in Disk I/O it is certainly pointing towards a networking issue on the surface.
I have downgraded kernel on two clusters and PBS, so they are to the last software available (no subscription) but kernel 6.14.11-4-pve.
One backup was normal, another one is still running, very slow; linux vm with guest agent:
INFO: 32% (19.2 GiB of 60.0 GiB) in 23m 32s, read: 16.0 MiB/s, write: 15.8 MiB/s
Incremental, new dirty-map, Really too slow....
so downgrading kernel doesn't resolve the issue for me,.
Which PVE version are you using ? I have the last no-subscription with downgraded kernel too, but some vm is still very very very very slow... (i am testing only a few and less important vm).
INFO: 67% (40.2 GiB of 60.0 GiB) in 49m 34s, read: 9.2 MiB/s, write: 0 B/s
Last edited:
it is in my earlier message, but our main production clusters we have both 9 and 8 versions, both fully updated as of last weekend. All PBS is 4 though (4.0 with the 6.14 kernel)
Hope a fix comes out soon.
Since kernel releases in enterprise repositories should be tested and stable.
... any update from maintainers or dev team ?
Since kernel releases in enterprise repositories should be tested and stable.
... any update from maintainers or dev team ?
I started to have problems upgrading PBS from 4.0 to 4.1; no any problem found with PBS 4.0
We were not able to reproduce the issue yet unfortunately. Again, if you are not using ZFS, you can help narrow it down by testing mainline kernels:
Did you disabled on PBS host or also on PVE hosts ? Have you noticed performance issues with 10Gbit nics like decreasing transfer speeds ?
Mmm... Heracleos said to have the problem with LAGG and MTU 1500...
My configuration has 2 x 10G LAGG with MTU 9000 with MLAG switches...
I'll give it a try. Just installed latest 6.17.2 kernel and set MTU to 1500 only on pbs. Next backup on 12:00 local time. I'll report.
Best
Knuut
No, we've had no issue but hard to compare since we have a bandwith limit on our Backup jobs due to with 802.3ad we hit the limits on our core switch. We have roughly 22 backup servers running at the same time, all of them with 10Gbit nics and the backbone is MLAG with 2x100Gbit.
Is the issue reproducible if stressing the network, e.g. by iperf?
Edit: Also, do you see high memory pressure on the PBS while this issue appears?
iperf -s on the PBS host and iperf -c <PBS-host-IP> -t 600 -i 10 on the backup source.Edit: Also, do you see high memory pressure on the PBS while this issue appears?
Last edited: