Guest VM soft lockups / RCU stalls on all vCPUs during VBR 13 backup of Proxmox VE 9 VM (worker-based)

[10u10]

Hi,
Product: Veeam Backup & Replication 13 (build: 13.1.1.18 ), Proxmox VE plug-in/worker version.
Hypervisor: Proxmox VE 9.2.20 1 node of cluster, storage: Ceph.

PROBLEM
During the backup job, the guest VM stops responding for 20-50 s and the guest kernel reports soft lockups on almost all vCPUs at once, followed by RCU stalls. This happens repeatedly on every job run. The same VM does not show this outside the backup window.

GUEST LOG (anonymized, uptime timestamps)
[ 628.38] watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [snapd:<PID>]
[ 633.26] watchdog: BUG: soft lockup - CPU#7 stuck for 28s! [kworker/7:2]
[ 712.38] watchdog: BUG: soft lockup - CPU#12 stuck for 22s!
[ 724.38] soft lockup CPU#8, CPU#7, CPU#13 stuck for 22s
[ 726.70] soft lockup CPU#3 stuck for 35s, CPU#9 for 32s
[ 776.38] soft lockup CPU#4 stuck for 49s, CPU#14 for 49s
[ 786.70] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks (t=60002 jiffies)
[ 796.37] rcu: rcu_preempt kthread starved for 9662 jiffies, state=0x0, cpu=6
[ 796.37] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[ 865.92] watchdog: BUG: soft lockup - CPU#4 stuck for 27s! [kworker/4:1]
Affected CPUs: 3,4,7,8,9,12,13,14,15 (VM has 16 vCPU). Pattern indicates the whole VM was stalled, not a single guest process.

VM CONFIGURATION (anonymized)
- Guest OS: Ubuntu 26.04.1 LTS, 16 vCPU (cpu=host), 64 GiB RAM, no ballooning, machine q35, SeaBIOS
- SCSI controller: VirtIO SCSI (not 'single'), IO thread: disabled, aio=io_uring, cache=none, discard=on
- scsi0: 100 GB on Ceph RBD (included in backup)
- scsi1: 40 TB on Ceph RBD HDD pool, backup=0 (excluded from backup)
- QEMU Guest Agent: enabled, freeze-fs-on-backup: disabled (Proxmox-side option)

I experience the same behavior on a Windows VM as well.

Veeam support investigated the issue and referred me to PVE support. Do you have any ideas on how to resolve the problem?

 

[kawano-y]

To help narrow down the cause, could you please check the following?

1. Do you observe any significant increase in CPU utilization or system load on the Proxmox VE host during the backup?
2. Do you notice any increase in Ceph latency or changes in cluster health around the time of the issue?
3. Does the issue still occur if you reduce the number of concurrent backup jobs?
4. Could you share the Proxmox VE host system logs and Veeam job logs from the time the issue occurred?

 

[10u10]

1. No, I didn't notice anything
2. No, everything looks normal
3. Yes
4. Yes, the PVE logs are attached

 

[kcastner_de]

Hi, @10u10;

as you experience the issue on multiple vms, i suggest to check your ram.
past 1 year already i got a similiar issue. in my case i got faulty ram at my pve host. it only occurs at my side during heavy ram usage. day to day use there was no problem.

can you check your ram via extended memtest? could take some hours maybe days....

 

[10u10]

Hi @kcastner_de thank you but that’s not issue in my case. I’m encountering this problem on every node within the cluster; all the nodes are a few weeks old.

 

[kcastner_de]

Hi @10u10,

every node within the cluster; all the nodes are a few weeks old.

Can you share your node(s) info regarding to your hardware specs, please?

As @kawano-y already said, can you share these infos aswell, please?

To help narrow down the cause, could you please check the following?

1. Do you observe any significant increase in CPU utilization or system load on the Proxmox VE host during the backup?
2. Do you notice any increase in Ceph latency or changes in cluster health around the time of the issue?
3. Does the issue still occur if you reduce the number of concurrent backup jobs?
4. Could you share the Proxmox VE host system logs and Veeam job logs from the time the issue occurred?

What about you backbone? What uplinks do you have to your CEPH? Are they directly connected? Any VLAN/Networking/Switch involved? What are these specs?

Any bandwith limits set to your pve cluster regarding backups?