VM becomes unresponsive during Veeam backup

S

sudomaster.ljs

Member
Oct 29, 2022
17
0
6
When a Veeam backup job runs against a VM on our HDD-backed Ceph cluster, the VM becomes unresponsive partway through the backup. In some cases, it recovers automatically once the job completes. Sometimes, it requires a hard reset to come back online.

We have reproduced this across multiple VMs, and it appears to affect only our HDD-based clusters. Backups using Proxmox Backup Server from the same cluster complete without issue. In both cases, the backup source and target storage are the same, and the backups use the same network path.

Has anyone experienced a similar issue or any insight into its possible cause?
 
Hi @sudomaster.ljs ,
There have been multiple reports in the last few years of users experiencing similar issues with the combination of Veeam and PVE. Unfortunately I do not recall any of the reporters coming back with a concrete resolution.
Note that Veeam has a unique way of integrating with PVE that may be playing a role here. Have you reached out to them to start the investigation?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
We are experiencing exactly the same issue with our production environment when running Veeam against CEPH RBD that is backed up by SSD disks.

Environment:

Veeam Backup & Replication: 13.0.2.29
Proxmox VE: 9.2.5
Kernel: Linux 7.0.14-6-pve (2026-07-20T14:45Z)
3-node Proxmox cluster
Ceph cluster - version: 20.2.2
24 SSD OSDs (8 OSDs per node)
Ceph health is OK during the issue
Additional storage:
Dell PowerStore presented via iSCSI Thick LVM

Inside the guest OS we see repeated messages when Veeam backup is running (with Hot-Add):
watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [ksoftirqd/4:32]
watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [ksoftirqd/4:32]
watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [ksoftirqd/4:32]

rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
rcu: 4-...: (75 ticks this GP) idle=8b2/1/0x4000000000000000 softirq=29246/29246 fqs=673

Observations:
Only the VM currently being backed up is affected.
Other VMs continue running normally on CEPH nodes. If running DD and testing performance, they give roughly 400-700MB/s throughput (command: dd if=/dev/zero of=/root/dd-test.img bs=1M count=16384 oflag=direct status=progress)
Proxmox host remains healthy:
CPU usage approximately 10%
IO delay below 1%
No memory pressure on hosts.
Ceph health remains OK.
Backup completes successfully.
After the backup finishes, the VM becomes responsive again without requiring host intervention.
If the same VM disk is migrated from Ceph RBD to Dell PowerStore iSCSI Thick LVM, Veeam backup completes normally and the issue does not occur. VM remains accessible and there are no problems with performance.

Reproduction:
VM located on Ceph RBD.
Start Veeam backup.
VM becomes nearly unresponsive.
Guest logs fill with soft lockup / RCU stall messages.
Backup completes.
VM becomes responsive again.

Veeam Job Statistics
We compared the same VM while it was located on different storage backends.
When the VM resides on Ceph RBD:
Veeam reports the bottleneck as Source (~99%)
The VM becomes almost completely unresponsive during backup.
Guest kernel reports repeated CPU soft lockups and RCU stalls.
Applications running inside the VM (such as MySQL) become unresponsive.

After migrating the same VM to Dell PowerStore (iSCSI Thick LVM):
The issue no longer occurs.
The VM remains fully responsive during backup.
Veeam reports the bottleneck as Target (~99%), which is the expected behavior.
We also made a test where we tested just using NBD (disabled proxies and forced Veeam Proxy on Proxmox VE to use NBD).

Ceph RBD using Hot-Add
Veeam reports Source 99% as the bottleneck.
The protected VM becomes almost completely unresponsive.
Guest kernel reports repeated CPU soft lockups and RCU stalls.
MySQL becomes unresponsive or crashes.
The VM recovers after the backup activity finishes.

Ceph RBD using NBD
Veeam still reports Source 99% as the bottleneck.
The VM remains responsive throughout the backup.
No CPU soft lockups or RCU stalls are observed.
Applications inside the VM continue operating normally.

My guess here would be that if the transfer is a bit slower, it does not "choke" the entire VM. However according to DD tests when doing this, write speed to CEPH dropped from 700MB/s to 80MB/s when running Veeam Backup on NBD mode.

I ran some DD commands through SSH when Veeam backup was running on Hot-Add mode (it was really slow and painful, but managed to do this):
Tested before starting Veeam backup:
~# dd if=/dev/zero of=/root/dd-test.img bs=1M count=16384 oflag=direct status=progress
16816013312 bytes (17 GB, 16 GiB) copied, 41 s, 410 MB/s

When Veeam backup is running:
~# dd if=/dev/zero of=/root/dd-test.img bs=1M count=16384 oflag=direct status=progress
7340032 bytes (7.3 MB, 7.0 MiB) copied, 70 s, 105 kB/s

Entire VM choked completely.

For ending words:
We also have lab environment with 4 hosts having exactly same hardware, but they are running Proxmox PBS as backup solution and do not experience this problem when backups are running.

I did create a Veeam ticket regarding to this same issue, we are also seeing this problem with every VM. Hopefully they do respond and have some ideas.
 
Veeam support asked me to do additional testing and according to the results, problem seems to be in Hot-Add with Veeam Proxy. This still has to be verified by Veeam, but below are my findings.

When Veeam PVE proxy Hot-Adds the disk => Almost all I/O activity within the VM stops and VM becomes sluggish / slow and all kind of kernel errors will start popping up "if you can see the console". Otherwise they just can be checked from /var/log/messages later. Sometimes the vncproxy simply just refuses to connect as the VM is not responding in timely manner.

Workaround (at least on our case) to avoid VM stuns / crashes at the moment is to run Veeam backups just using NBD. This requires you to have the Veeam PVE Proxy on separate host than the VM that is getting backed up - unfortunately there is no setting in Veeam from where you can choose the backup mode. Another problem with this is that NBD backups are also slower ~ roughly by 1/3 compared to Hot-Add.

I did some additional testing when backup was running on with NBD mode running dd command on backed up VM. Performance on writes dipped on SSD Backed Ceph to half when running backups but it is still very good performance. Results are below:

#When Veeam backup was running using NBD Mode:
~# dd if=/dev/zero of=/root/dd-test.img bs=1M count=16384 oflag=direct status=progress
16283336704 bytes (16 GB, 15 GiB) copied, 33 s, 493 MB/s
16384+0 records in
16384+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 33.7971 s, 508 MB/s

#After Veeam backup was finished:
:~# dd if=/dev/zero of=/root/dd-test.img bs=1M count=16384 oflag=direct status=progress
16984834048 bytes (17 GB, 16 GiB) copied, 17 s, 999 MB/s
16384+0 records in
16384+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 17.1844 s, 1000 MB/s

Similar test above when using Hot-Add - speed was limited to mere Kilobytes and VM was really hard to operate at all:
#When Veeam backup is running (Hot-Add):
~# dd if=/dev/zero of=/root/dd-test.img bs=1M count=16384 oflag=direct status=progress
7340032 bytes (7.3 MB, 7.0 MiB) copied, 70 s, 105 kB/s
 
just out of curiosity, did it only happen when updating to 9.2.5? i am using cephrbd with veeam + pve 9.2.3 and do not seem to have this issue
 
gasherbrum said:
just out of curiosity, did it only happen when updating to 9.2.5? i am using cephrbd with veeam + pve 9.2.3 and do not seem to have this issue
Click to expand...
I started to see this issue on 9.2.4 this Monday, just updated yesterday to 9.2.5 on hopes that it would solve the problem. I did not see this on 9.2.3 or atleast our monitoring did not trigger any alerts on that version, but after this Monday I started receiving process + ping alerts on the affected VMs. We have had Veeam backups running on this cluster from 9.1.x version and we did not have any issues when we started running backups with Veeam (Hot-add).

As for Veeam itself, we are running Veeam Linux Appliance instead of having it installed on Windows. It is possible that appliance patched itself + proxies and issues triggered after that.

Veeam support is investigating this, hopefully they come up with some ideas.
 
sakrizz said:
I started to see this issue on 9.2.4 this Monday, just updated yesterday to 9.2.5 on hopes that it would solve the problem. I did not see this on 9.2.3 or atleast our monitoring did not trigger any alerts on that version, but after this Monday I started receiving process + ping alerts on the affected VMs. We have had Veeam backups running on this cluster from 9.1.x version and we did not have any issues when we started running backups with Veeam (Hot-add).

As for Veeam itself, we are running Veeam Linux Appliance instead of having it installed on Windows. It is possible that appliance patched itself + proxies and issues triggered after that.

Veeam support is investigating this, hopefully they come up with some ideas.
Click to expand...
thanks, i previously had an issue with veeam restores + a regression in a qemu package, it looks like there was some changes that might be related to this issue in a recent update so it may be worth asking proxmox support to take a look, for now ill remain on 9.2.3 :)
 
You must log in or register to reply here.
Top Bottom
Back