Hi,
Product: Veeam Backup & Replication 13 (build: 13.1.1.18 ), Proxmox VE plug-in/worker version.
Hypervisor: Proxmox VE 9.2.20 1 node of cluster, storage: Ceph.
PROBLEM
During the backup job, the guest VM stops responding for 20-50 s and the guest kernel reports soft lockups on almost all vCPUs at once, followed by RCU stalls. This happens repeatedly on every job run. The same VM does not show this outside the backup window.
GUEST LOG (anonymized, uptime timestamps)
[ 628.38] watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [snapd:<PID>]
[ 633.26] watchdog: BUG: soft lockup - CPU#7 stuck for 28s! [kworker/7:2]
[ 712.38] watchdog: BUG: soft lockup - CPU#12 stuck for 22s!
[ 724.38] soft lockup CPU#8, CPU#7, CPU#13 stuck for 22s
[ 726.70] soft lockup CPU#3 stuck for 35s, CPU#9 for 32s
[ 776.38] soft lockup CPU#4 stuck for 49s, CPU#14 for 49s
[ 786.70] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks (t=60002 jiffies)
[ 796.37] rcu: rcu_preempt kthread starved for 9662 jiffies, state=0x0, cpu=6
[ 796.37] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[ 865.92] watchdog: BUG: soft lockup - CPU#4 stuck for 27s! [kworker/4:1]
Affected CPUs: 3,4,7,8,9,12,13,14,15 (VM has 16 vCPU). Pattern indicates the whole VM was stalled, not a single guest process.
VM CONFIGURATION (anonymized)
- Guest OS: Ubuntu 26.04.1 LTS, 16 vCPU (cpu=host), 64 GiB RAM, no ballooning, machine q35, SeaBIOS
- SCSI controller: VirtIO SCSI (not 'single'), IO thread: disabled, aio=io_uring, cache=none, discard=on
- scsi0: 100 GB on Ceph RBD (included in backup)
- scsi1: 40 TB on Ceph RBD HDD pool, backup=0 (excluded from backup)
- QEMU Guest Agent: enabled, freeze-fs-on-backup: disabled (Proxmox-side option)
I experience the same behavior on a Windows VM as well.
Veeam support investigated the issue and referred me to PVE support. Do you have any ideas on how to resolve the problem?
Product: Veeam Backup & Replication 13 (build: 13.1.1.18 ), Proxmox VE plug-in/worker version.
Hypervisor: Proxmox VE 9.2.20 1 node of cluster, storage: Ceph.
PROBLEM
During the backup job, the guest VM stops responding for 20-50 s and the guest kernel reports soft lockups on almost all vCPUs at once, followed by RCU stalls. This happens repeatedly on every job run. The same VM does not show this outside the backup window.
GUEST LOG (anonymized, uptime timestamps)
[ 628.38] watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [snapd:<PID>]
[ 633.26] watchdog: BUG: soft lockup - CPU#7 stuck for 28s! [kworker/7:2]
[ 712.38] watchdog: BUG: soft lockup - CPU#12 stuck for 22s!
[ 724.38] soft lockup CPU#8, CPU#7, CPU#13 stuck for 22s
[ 726.70] soft lockup CPU#3 stuck for 35s, CPU#9 for 32s
[ 776.38] soft lockup CPU#4 stuck for 49s, CPU#14 for 49s
[ 786.70] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks (t=60002 jiffies)
[ 796.37] rcu: rcu_preempt kthread starved for 9662 jiffies, state=0x0, cpu=6
[ 796.37] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[ 865.92] watchdog: BUG: soft lockup - CPU#4 stuck for 27s! [kworker/4:1]
Affected CPUs: 3,4,7,8,9,12,13,14,15 (VM has 16 vCPU). Pattern indicates the whole VM was stalled, not a single guest process.
VM CONFIGURATION (anonymized)
- Guest OS: Ubuntu 26.04.1 LTS, 16 vCPU (cpu=host), 64 GiB RAM, no ballooning, machine q35, SeaBIOS
- SCSI controller: VirtIO SCSI (not 'single'), IO thread: disabled, aio=io_uring, cache=none, discard=on
- scsi0: 100 GB on Ceph RBD (included in backup)
- scsi1: 40 TB on Ceph RBD HDD pool, backup=0 (excluded from backup)
- QEMU Guest Agent: enabled, freeze-fs-on-backup: disabled (Proxmox-side option)
I experience the same behavior on a Windows VM as well.
Veeam support investigated the issue and referred me to PVE support. Do you have any ideas on how to resolve the problem?