I might have gotten something related to this as well but on smalls disks. CPU stalls, qemu-agent involved.
See https://forum.proxmox.com/threads/rcu-info-rcu_sched-self-detected-stall-on-cpu.109112/post-469315
See https://forum.proxmox.com/threads/rcu-info-rcu_sched-self-detected-stall-on-cpu.109112/post-469315
I have switched to VirtIO SCSI single and turned iothread on on each disk. No change.
Log excerpt:
Log excerpt:
Node and VM were restarted before the backup attempt. Backup on an old node with virtio scsi single and iothread worked like a charm.
Yes, but I don't quite understand what to do now. Do we need to downgrade pve-qemu-kvm on all our nodes? If yes, will it affect stability of the cluster?
@Fabian_E would be great to hear some recommendations.
Since the workaround with iothread didn't work for both of you, the issue I managed to run into is obviously different. Unfortunately, I haven't been able to reproduce the issue with the timed out QMP commands yet. There were a lot of upstream changes between QEMU 6.1.1-2 and 6.2.0-1 so it's not obvious what started triggering this. The backup code from our side mostly stayed the same between these.
If no other workaround helps, I'm afraid keeping
pve-qemu-kvm downgraded might be the only way for now. It should not affect the stability, but you won't get any fixes/updates for QEMU of course...As for finding other workarounds/trying to track this down more:
Does spacing the backups out more help? Can you trigger the issue reliably or does it depend on system/PBS load? Could you share your storage configuration and the configuration of some affected VMs? The one for the minimal VM with empty disks would be interesting too.
I have stopped upgrading my cluster after running into this bug, so i can provide you with both a working and a not working config or logs. I have the affected VM running alone on an upgraded host and it is absolutely reliable some time into the backup starting to produce the errors, while running fine on an not upgraded node or with the version of pve-qemu-kvm downgraded.
@Fabian_E Please let me know if you need the output of specific commands. In general, the VM resides on a ceph cluster and the backup goes to a Proxmox Backup Server running locally on dedicated hardware, connected with a 10GBit/s link.
VM config:
Last edited:
Yes, it happens reliably 100% of the time. The PBS load is ~0% (nothing happening), same with the node. The issue happens regardless of whether the VM is on or off.
Code:
agent: 1
balloon: 131072
bios: ovmf
boot: order=scsi0;net0
cores: 35
efidisk0: cpool:vm-110-disk-2,efitype=4m,size=528K
memory: 196608
meta: creation-qemu=6.1.0,ctime=1651062805
name: ****
net0: virtio=56:57:17:5E:60:F7,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: l26
scsi0: cpool:vm-110-disk-0,discard=on,iothread=1,size=300G,ssd=1
scsi1: cpool:vm-110-disk-1,discard=on,iothread=1,size=6300G,ssd=1
scsi2: cpool:vm-110-disk-3,discard=on,iothread=1,size=10G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=ad9152a8-5683-4601-b725-12d134f00e65
sockets: 2
vmgenid: 74ed1dbe-56ca-4cdb-95c2-3188dda5be0athe "creation-qemu" was apparently different because I had to downgrade everything.
I think I was finally able to reproduce the problem, using Ceph without krbd as the underlying storage for the large disks. When krbd is enabled, it doesn't trigger for me, so that should be another workaround. I'll try to find the root cause now with the reproducer.
Upstream bug report with description of the root cause: https://gitlab.com/qemu-project/qemu/-/issues/1026
Patch to revert the offending change for now: https://lists.proxmox.com/pipermail/pve-devel/2022-May/053021.html
Real fix would be to improve the performance of the problematic function, but that needs more time and hopefully we get some feedback from upstream QEMU developers as well.
Patch to revert the offending change for now: https://lists.proxmox.com/pipermail/pve-devel/2022-May/053021.html
Real fix would be to improve the performance of the problematic function, but that needs more time and hopefully we get some feedback from upstream QEMU developers as well.
Hello, KRBD doesn't help here, it was already enabled.
pve+ceph 13 nodes cluster alls ssd. pbs all ssd. separate 10gbit networks for ceph and public+backup.
just now updated and rebooted whole cluster.
backups always maxed out the 10gbit network (900MBs for all nodes together), since update last saturday backups are slow as 70-110MBs (all nodes together). only few smaller vms are backed up successfully. everything above 128GB disk size is timing out (different qmp timeouts).
pve+ceph 13 nodes cluster alls ssd. pbs all ssd. separate 10gbit networks for ceph and public+backup.
just now updated and rebooted whole cluster.
backups always maxed out the 10gbit network (900MBs for all nodes together), since update last saturday backups are slow as 70-110MBs (all nodes together). only few smaller vms are backed up successfully. everything above 128GB disk size is timing out (different qmp timeouts).
Hi,
this might be a different issue then. Please open a new thread and link to it here or ping me with @Fabian_E with the following information:
pveversion -v- config of an affected VM
- full backup log and
/var/log/syslogduring the backup - If the issue is also present when backing up to a non-PBS storage
- Whether downgrading to
pve-qemu-kvm=6.2.0-1helps - Whether downgrading to
pve-qemu-kvm=6.1.1-2helps - Whether switching the ISCSI controller type to
Virtio SCSI singleand enablingiothreadon the disks helps
Morning from PST all,
Just a note to perhaps help someone else experiencing this frustrating issue.
We experienced out 1.5 hour multi-VM backup (Ceph and Proxmox's built in backup, not PBS) suddenly changing to 12+ hours. On top of that the VMs with the largest disks (750GB) would drop in performance by 95% causing our help desk to blow up with complaints. I've checked KRBD and tested. This didn't help. The VMs needed to be shut down and restarted. This solved the issue. I'm guessing that this is obvious to some but not others ... not when you're under pressure to fix an issue.
That said, I see new patches for backup related components this morning awaiting install ... if I recall (I see a lot of posts from Bugzilla in my email) the appropriate components have been rolled back a version to fix this issue.
best,
James
Just a note to perhaps help someone else experiencing this frustrating issue.
We experienced out 1.5 hour multi-VM backup (Ceph and Proxmox's built in backup, not PBS) suddenly changing to 12+ hours. On top of that the VMs with the largest disks (750GB) would drop in performance by 95% causing our help desk to blow up with complaints. I've checked KRBD and tested. This didn't help. The VMs needed to be shut down and restarted. This solved the issue. I'm guessing that this is obvious to some but not others ... not when you're under pressure to fix an issue.
That said, I see new patches for backup related components this morning awaiting install ... if I recall (I see a lot of posts from Bugzilla in my email) the appropriate components have been rolled back a version to fix this issue.
best,
James
yes, this need a vm stop/start (or a live migration) to use the krbd path.
Same when qemu will be updated with the fix for non-krbd access.(as it's a regression in qemu).
https://lists.proxmox.com/pipermail/pve-devel/2022-May/053021.html
Thank You Fabian,
sadly i can test the older pve-qemu-kvm coming weekend at earliest (service time window), since all vms need to be rebooted and i lose dirty map again.
If i understand it correctly, todays updates contain a regression patch for the "bdrv_co_block_status" issue, i will try this one first, if it doesn't help, test the older pve-qemu-kvm versions as you suggested. i will provide all the requested logs and infos in a new thread, if issue persists.
Again, thanks!
sadly i can test the older pve-qemu-kvm coming weekend at earliest (service time window), since all vms need to be rebooted and i lose dirty map again.
If i understand it correctly, todays updates contain a regression patch for the "bdrv_co_block_status" issue, i will try this one first, if it doesn't help, test the older pve-qemu-kvm versions as you suggested. i will provide all the requested logs and infos in a new thread, if issue persists.
Again, thanks!