Proxmox 9 CPU/IO starved when doing a restore from PBS

[koiji]

Hi,

We've been doing some research on "Proxmox / storage load" and it seems that the information on the topic is somewhat scattered.

The setup

• We have a new, rather powerful application server (48 core EPYC Zen 4, 768GB RAM, 2x 8TB NVMe PCI-e 4.0 in ZFS RAID 1 (no HW RAID)) connected to a decent backup server (16 core EPYC Zen 3, 128 GB RAM, 32 TB NVMe PCI-e 3.0 storage)
• The application server load is about 2% and IO Pressure Stall nears 0% without any spikes (not used for production yet)
• Everything is snappy as expected
• The servers are connected via 2x 10 Gbit/s local network.
• Latest versions of software: Proxmox 9.2.4, PBS 4.2.1
• ZFS storage, no special configuration

The problem

• When I try to restore a bigger VM backup (say around 3-4 TB), the load on the server spikes massively ("server load" into hundreds) as well as I/O delay well over 50%. The VMs inside become unresponsive and eventually after a while (say 20-30s) we start to get virtio driver errors in Windows Event Logs suggesting that the I/O is delayed so much that Windows tries to reset the virtio controller.

Observations

• When doing a restore I ran iftop on the PBS and it seems that during a bigger disk restore the read/send speed is about 400 MB/s, so we figured out we might slow down the restore speed:
• We tried to slow it down to 200 MiB/s:
  • Datacenter > server > PBS (as a backup storage) > choose VM backup > Restore > Bandwidth limit = 200 MiB/s
  • This option seems to us to be ignored because
    1. The restore process behaved exactly the same (i.e. after a few dozens of seconds the server came to a crawl)
    2. The actual restore command looked like this, implying there's no bw limit applied:
       restore proxmox backup image: /usr/bin/pbs-restore --repository appserver@pbs@192.168.13.50:appserver vm/150/2026-07-09T21:59:27Z drive-efidisk0.img.fidx /dev/zvol/rpool/data/vm-150-disk-2 --verbose --format raw --skip-zero
• We also tried doing an "within the VM" copy to simulate a big write operation which might disturb either the server or the disks. We copied 50 GB file in Windows to the same virtual disk) and the speed was about 1.4GB/s while the CPU and IO delay on the host server barely noticed (I think the highest the IO delay spiked was about 1.6%).

---

How should we go about diagnosing this?

While I read the documentation suggesting that it might be helpful to slow down the restore speed, I couldn't find any "support" data on what is actually the expected restore speed for a (in my opinion) rather fast system.

Maybe if anyone can shed some light on why a ~400MiB/s speed would *destroy* such a setup or give us pointers on how to limit the restore speed properly?

Thanks

 

[Francois Baillargeon]

400 MiB/s is already like 5Gbit/s, how is your lag setup? I could only manage faster than line speed in L3+L4 mode in most cases.

can you run

iftop

, you would see if your interfaces are saturated

That said, do you have PLP on those disks? It seems like a cache flush problem due to consumer hardware.

 

[koiji]

400 MiB/s is already like 5Gbit/s, how is your lag setup? I could only manage faster than line speed in L3+L4 mode in most cases.

can you run

iftop

, you would see if your interfaces are saturated

That said, do you have PLP on those disks? It seems like a cache flush problem due to consumer hardware.

1) LAG is the "default", like so:

iface bond0 inet manual
        bond-slaves nic2 nic3
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        bond-miinmon 100

There's no other traffic between the app and backup server during the restore (the only time there's any traffic is during daily backups during the night).
I don't specifically need the restore to be faster (at least before I solve the issue of 400 MiB/s bringing the server down to its knees).

2) The SSDs are Samsung PM9D3a. Would you like me to check for something specifically?

 

[uzumo]

There are risks involved, but what would happen if you tried the following commands? Would the average load increase?

zfs set sync=disabled <Pool>
echo 1 > /sys/module/zfs/parameters/zfs_nocacheflush

*Execute these on the PVE side.
*Either command is fine.
*This is for testing purposes only. It is not intended for regular use.
*However, even if data is lost, I cannot offer any guarantees whatsoever.

 

[guyman]

Severe I/O stalls and VM corruption during PBS restore on Proxmox VE 8 and 9

We are running a single-node Proxmox VE host that regularly backs up its VMs to a Proxmox Backup Server. Backups themselves have been working reliably without any noticeable impact.

The host is fairly powerful:

• Dual Intel Xeon CPUs
• 1 TB RAM
• Samsung Enterprise NVMe SSDs
• ZFS mirror
• ZFS reports no errors and the pool is healthy

Today, however, we tried to restore a relatively small Linux VM from PBS and experienced what can only be described as a nightmare.

As also reported in several other forum threads, during the restore the I/O performance of the entire hypervisor deteriorates dramatically. Read/write operations appear to stall or become heavily blocked, and the other VMs on the host effectively freeze or become unusable.

Unfortunately, in our case this was not just a temporary performance issue.

During the restore:

• one Windows Domain Controller suffered filesystem corruption and had to be recovered;
• another Windows Server used as a print server also suffered filesystem corruption;
• a MySQL table on a Linux VM became corrupted;
• the Linux VM running MySQL experienced a kernel hang.

The underlying ZFS pool itself remains healthy. Scrubs do not report errors and there are no apparent NVMe or ZFS integrity issues.

The problem therefore appears to be related to the extreme I/O contention/stalling caused by the restore operation rather than an underlying storage failure.

We initially encountered this on the latest Proxmox VE 8 release. Because of the severity of the issue, we upgraded the host to Proxmox VE 9 today and repeated the test.

Unfortunately, we can reproduce the same behaviour on Proxmox VE 9.

While a PBS restore is running, normal operation of the other VMs on the host is effectively impossible. More concerningly, the I/O stalls appear severe enough to cause filesystem/database corruption inside unrelated running VMs.

For us, this makes the restore functionality extremely difficult to use safely in production. A backup solution obviously needs to provide reliable restores without putting unrelated workloads on the same host at risk.

Given the hardware involved — dual Xeon CPUs, 1 TB RAM and enterprise NVMe storage in a ZFS mirror — this does not appear to be a simple case of insufficient host resources.

Has anyone else been able to reproduce this behaviour on recent PVE 8.x or PVE 9.x releases?

In particular, we would be interested in:

• whether this is a known ZFS/PBS restore issue;
• whether there is a known regression in the I/O scheduling or restore path;
• recommended ways to throttle PBS restore I/O;
• whether specific ZFS, QEMU or Linux I/O scheduler settings mitigate the issue;
• and whether there is an existing Proxmox bug report tracking this behaviour.

At this point we are reluctant to perform another restore on a production host, as the last test resulted in corruption of multiple completely unrelated VMs.

 

[longer]

...

As also reported in several other forum threads, during the restore the I/O performance of the entire hypervisor deteriorates dramatically. Read/write operations appear to stall or become heavily blocked, and the other VMs on the host effectively freeze or become unusable.

Unfortunately, in our case this was not just a temporary performance issue.

During the restore:

• one Windows Domain Controller suffered filesystem corruption and had to be recovered;
• another Windows Server used as a print server also suffered filesystem corruption;
• a MySQL table on a Linux VM became corrupted;
• the Linux VM running MySQL experienced a kernel hang.

The underlying ZFS pool itself remains healthy. Scrubs do not report errors and there are no apparent NVMe or ZFS integrity issues.

The problem therefore appears to be related to the extreme I/O contention/stalling caused by the restore operation rather than an underlying storage failure.

We initially encountered this on the latest Proxmox VE 8 release. Because of the severity of the issue, we upgraded the host to Proxmox VE 9 today and repeated the test.

Unfortunately, we can reproduce the same behaviour on Proxmox VE 9.
...
At this point we are reluctant to perform another restore on a production host, as the last test resulted in corruption of multiple completely unrelated VMs.

Does it makes a difference if you set Bandwidth Limits? Datacenter -> Options -> Bandwidth Limits - Backup Restore.

 

[fiona]

Hi,
a similar issue was also reported in our enterprise support and investigated by our support engineers. We suspect that the issue is related to the pbs-restore process using the host page cache and filling it up for ZFS and are currently evaluating a fix.