Backup issues with Windows VMs - 100% active time

vescovialex

Active Member
Aug 22, 2020
43
6
28
37
www.auxbit.it
I'm having some trouble with backups on my Proxmox cluster and I'm hoping someone can help me troubleshoot. I've been searching for solutions online for a while now, but I haven't been able to find anything that helps.

Here's my setup:

  • 3-node Proxmox cluster
  • Backups to both a NAS and a PBS
  • 10Gbps fiber networking (except for the NAS)
  • 10Gbps fiber-connected storage
  • ZFS over iSCSI storage
  • SSD and HDD pool on storage in raidz2

The problem:

Windows VM backups are taking significantly longer than usual. Backups that used to take 15 minutes now take 2.5 hours.


What I've tried:

  • Checked disk throughput: Disk throughput is high during backups, but it's also high during normal VM operation.
  • Checked I/O delay: I/O delay is normal (<1%).
  • Run scrubs on the ZFS pool: Scrubs have completed successfully.
  • Tested the SSDs: I've pulled the SSDs out of the storage and tested them separately, and they seem to be fine.

Additional observations:
  • Some Proxmox systems in my environment do not experience slow backups.
  • The disk on the backup target is pegged at 100% during backups.
  • The problem seems to have started recently (1-2 weeks ago).

I'm at a loss for what to try next. Any suggestions would be greatly appreciated.


Here are some additional details that may be helpful:

  • Proxmox VE version: 8.0.3
  • All disks on VMs are scsi
  • SCSI Controller: VirtIO SCSI
  • Virtio Guest: Installed
I'm hoping that someone in the community can help me figure out what's going on. Thanks in advance for your help!
 
Last edited: