Proxmox 9 CPU/IO starved when doing a restore from PBS

[koiji]

Hi,

We've been doing some research on "Proxmox / storage load" and it seems that the information on the topic is somewhat scattered.

The setup

• We have a new, rather powerful application server (48 core EPYC Zen 4, 768GB RAM, 2x 8TB NVMe PCI-e 4.0 in ZFS RAID 1 (no HW RAID)) connected to a decent backup server (16 core EPYC Zen 3, 128 GB RAM, 32 TB NVMe PCI-e 3.0 storage)
• The application server load is about 2% and IO Pressure Stall nears 0% without any spikes (not used for production yet)
• Everything is snappy as expected
• The servers are connected via 2x 10 Gbit/s local network.
• Latest versions of software: Proxmox 9.2.4, PBS 4.2.1
• ZFS storage, no special configuration

The problem

• When I try to restore a bigger VM backup (say around 3-4 TB), the load on the server spikes massively ("server load" into hundreds) as well as I/O delay well over 50%. The VMs inside become unresponsive and eventually after a while (say 20-30s) we start to get virtio driver errors in Windows Event Logs suggesting that the I/O is delayed so much that Windows tries to reset the virtio controller.

Observations

• When doing a restore I ran iftop on the PBS and it seems that during a bigger disk restore the read/send speed is about 400 MB/s, so we figured out we might slow down the restore speed:
• We tried to slow it down to 200 MiB/s:
  • Datacenter > server > PBS (as a backup storage) > choose VM backup > Restore > Bandwidth limit = 200 MiB/s
  • This option seems to us to be ignored because
    1. The restore process behaved exactly the same (i.e. after a few dozens of seconds the server came to a crawl)
    2. The actual restore command looked like this, implying there's no bw limit applied:
       restore proxmox backup image: /usr/bin/pbs-restore --repository appserver@pbs@192.168.13.50:appserver vm/150/2026-07-09T21:59:27Z drive-efidisk0.img.fidx /dev/zvol/rpool/data/vm-150-disk-2 --verbose --format raw --skip-zero
• We also tried doing an "within the VM" copy to simulate a big write operation which might disturb either the server or the disks. We copied 50 GB file in Windows to the same virtual disk) and the speed was about 1.4GB/s while the CPU and IO delay on the host server barely noticed (I think the highest the IO delay spiked was about 1.6%).

---

How should we go about diagnosing this?

While I read the documentation suggesting that it might be helpful to slow down the restore speed, I couldn't find any "support" data on what is actually the expected restore speed for a (in my opinion) rather fast system.

Maybe if anyone can shed some light on why a ~400MiB/s speed would *destroy* such a setup or give us pointers on how to limit the restore speed properly?

Thanks

 

[Francois Baillargeon]

400 MiB/s is already like 5Gbit/s, how is your lag setup? I could only manage faster than line speed in L3+L4 mode in most cases.

can you run

iftop

, you would see if your interfaces are saturated

That said, do you have PLP on those disks? It seems like a cache flush problem due to consumer hardware.

 

[koiji]

400 MiB/s is already like 5Gbit/s, how is your lag setup? I could only manage faster than line speed in L3+L4 mode in most cases.

can you run

iftop

, you would see if your interfaces are saturated

That said, do you have PLP on those disks? It seems like a cache flush problem due to consumer hardware.

1) LAG is the "default", like so:

iface bond0 inet manual
        bond-slaves nic2 nic3
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        bond-miinmon 100

There's no other traffic between the app and backup server during the restore (the only time there's any traffic is during daily backups during the night).
I don't specifically need the restore to be faster (at least before I solve the issue of 400 MiB/s bringing the server down to its knees).

2) The SSDs are Samsung PM9D3a. Would you like me to check for something specifically?

 

[uzumo]

There are risks involved, but what would happen if you tried the following commands? Would the average load increase?

zfs set sync=disabled <Pool>
echo 1 > /sys/module/zfs/parameters/zfs_nocacheflush

*Execute these on the PVE side.
*Either command is fine.
*This is for testing purposes only. It is not intended for regular use.
*However, even if data is lost, I cannot offer any guarantees whatsoever.