Very slow backups

[mn124700]

Good morning all. I just rebuilt my Proxmox server after a hardware failure (bad SSD). I'm up and running again witn Proxmox 7.3-1. I'm running into an issue with the backup function. I started a backup of 5 VMs last night. As of now, 12 hours later, it has still not finished! No errors, but very slow. Initially, it was backing up at about 5-6 MB/sec, but this is now down to about 1-2 MB/s. The "IO Delay" for the node is at about 16%.

My setup is a single node, Proxmox 7.3. The system is installed on an SSD and the VMs are on a separate SSD. The backups are being done to an internal HDD (ext4).

How to I go about diagnosing the problem?

Thanks,
Eric

 

[_gabriel]

Are you using ZFS datastore for VM ?
What ssd model ? what HDD model ?
I bet on a SMR HDD

 

[mn124700]

Thanks for the quick response.
No ZFS.
HDD: Western Digital 2TB WD Blue PC Internal Hard Drive HDD - 5400 RPM, SATA 6 Gb/s, 256 MB Cache, 3.5" - WD20EZAZ
SSD: SAMSUNG Electronics 870 EVO 2TB 2.5 Inch SATA III Internal SSD (MZ-77E2T0B/AM)

I've got the HDD configured with Type: Directory and Content: VZDump backup file

Any ideas? Thanks again
Eric

 

[_gabriel]

backup VM one by one , to check if its VM related.
I recommend first backup during stopped vm, then next hot backup only need to read , then next-next hot backup will use dirty bitmaps of qemu to backup only changed blocks.

 

[mn124700]

I don't think it's VM related. In looking at the backup log, all VM backups have been very slow. Never more than 4-5 MB/s. And they've all been in a stopped state, except for the first one.

 

[leesteken]

SMR HDD's like your cannot handle a lot of writes for a long time and will become very slow due to the way whole tracks need to be read an rewritten by SMR for each write.

 

[mn124700]

Yuck. They should put a warning label! Maybe I'll swap it for another SSD. Is there any good reason to have an HDD?

 

[Dunuin]

HDD: Western Digital 2TB WD Blue PC Internal Hard Drive HDD - 5400 RPM, SATA 6 Gb/s, 256 MB Cache, 3.5" - WD20EZAZ

That's a SMR HDD so terrible write performance, even for a HDD.

Is there any good reason to have an HDD?

More capacity for the same price. Won't heavily wear when writing a lot to it (will just slowly wear all time when powered on).

 

[leesteken]

Yuck. They should put a warning label! Maybe I'll swap it for another SSD. Is there any good reason to have an HDD?

Please don't replace it with a QLC SSD, they are even slower in the same scenario.

EDIT: Or just stop writing to the drive and let is settle for a day to reorganize itself (it does need to stay on) and try again? CMR HDDs are still larger/cheaper than SSDs.