28GB Backup, Restore has Taken 4hr 15min

[7ekhed]

Hey Proxmox Forum,
I've backed up one of my VM's on an old node, and moved the backup file to a new server
I began the restore this morning around 8:50AM, and now at 1:38, it says, "Progress 100%" ONLY for read

Things were taking aggregiously long previously, up to 5 minutes per GB


What could be causing this?

I

I am restoring this VM onto a 1TB SSD Raid 5 Array, the HDD space is just where the backup is stored for now, as to why it is full

The VM Disk Size is only set to 80, and should only be using 40-50 of that

 

[narrateourale]

How fast is the network to the PBS? Do I understand it correctly, that HDDs are used for the PBS storage? Where there other jobs going on while the restore was happening? Other backups, garbage collection or verify jobs?

Since the backups are split into many small chunks, for VMs, 1 chunk per 4 MB of disk, a restore needs to read all these files that are spread out on the PBS storage -> many IOPS. HDDs are notoriously bad at IOPS and therefore, that could be one reason why it is slow. If other jobs were accessing the PBS at the same time... even worse.

But of course, the network could also be an issue, if it is only running at 100 Mbit for example due to a fault cable.

 

[7ekhed]

How fast is the network to the PBS? Do I understand it correctly, that HDDs are used for the PBS storage? Where there other jobs going on while the restore was happening? Other backups, garbage collection or verify jobs?

Since the backups are split into many small chunks, for VMs, 1 chunk per 4 MB of disk, a restore needs to read all these files that are spread out on the PBS storage -> many IOPS. HDDs are notoriously bad at IOPS and therefore, that could be one reason why it is slow. If other jobs were accessing the PBS at the same time... even worse.

But of course, the network could also be an issue, if it is only running at 100 Mbit for example due to a fault cable.

Not a PBS server, a VM backup on a local drive

No other jobs, either

No HDDs present, all SSDs, 120gb SSD boot drive, 1TB RAID 5 SSD

10GB/s Lan, 400mbps down 30mbps up to Internet

Some parts took 5 minutes per completion percent, the first 26% took 4 seconds per percent, 26%-80 something % took 5 minutes per percentage, then the last percentages completed nearly instantly, and stayed on 100% for 5 hours until it finally completed, took 2 hours to get to 100% and 5 more hours to STATUS:OK

 

[Dunuin]

Some parts took 5 minutes per completion percent, the first 26% took 4 seconds per percent, 26%-80 something % took 5 minutes per percentage, then the last percentages completed nearly instantly, and stayed on 100% for 5 hours until it finally completed, took 2 hours to get to 100% and 5 more hours to STATUS:OK

Sounds like consumer SSDs hitting the wall after 26% was finished because DRAM-/SLC-cache got filled up?

 

[7ekhed]

Sounds like consumer SSDs hitting the wall after 26% was finished because DRAM-/SLC-cache got filled up?

That would make sense, I spose!

I've got a cheapass PNY 120gb SSD for my proxmox disk, and I wouldn't doubt that it probably doesn't have 30gb if dram cache, and probably doesn't have the best methods for dealing with clearing cache, so you may be right about that consider the backup was also stored in that disk, but then restored to my RAID array

I'm a little surprised it didn't read, push to ram, and keep reading, considering I have way more than the VM capacity

Thank you for this insight, I may need to just unify my SSDs and push my OS disk, too, to a Samsung

Is there a graphic way to display this, or should I just up and replace the drive? Considering a 500gb Samsung Sata SSD only costs ~30$ new on Amazon, I won't be sad if it's the second option

 

[Dunuin]

If you want consistent write performance (so at least a mixed-workload enterprise SSD) it is more like 120$ for a 500GB SSD.

 

[Enlightend]

Several questions come up.

Is that "1TB SSD Raid 5 Array" a zfs type "raid 5" array (aka RaidZ1)? Or a controller based Raid 5?

If the first, why put SSDs in a raid 5 config? If the second, why use a raid 5 controller with ZFS. Any form of raid higher than 0/1/10 will cause unneeded writes on the disks that will murder their cells.

On top of that Dunuin is absolutely right, many/most low tier consumer grade SSDs can have silly slow writes once their caches are full. Even the modern 3GB/s style NVMe drives can go down to a kilobyte crawl when pumped with a ton of IO once their cache is full.


That combined with RAID5 SSD arrays amplifies write slowness to even more problematic extents.


With the information already provided, you're dealing with layers upon layers of bottlenecks that shouldn't be there.


Your disk setup should be one or more a r1/r10 (aka mirror/mirror+stripe) for redundancy and nothing in between but straight HBA mode or basic SATA bus in between.

 

[Dunuin]

If the first, why put SSDs in a raid 5 config? If the second, why use a raid 5 controller with ZFS. Any form of raid higher than 0/1/10 will cause unneeded writes on the disks that will murder their cells.

Depends. I benchmarks a lot of possible pool layouts 8/6/4 disk raid10, 3/5/7 disk raidz1, 4/6/8 disk raidz2, ... for write amplification and finally have chosen a 5 disk raidz1 for important guests + single disk LVM-Thin for unimportant guests as this caused less SSD wear than a 6 disk raid10. With a 5 disk raidz1 you only got +25% parity overhead while with a 6 disk raid10 it would be +100%. With a raidz1 you will still get some additional overhead because of the bigger volblocksize that is needed to minimize padding overhead, but this overhead was still less than writing every block of data twice to the disks.

But if you care more about IOPS performance than capacity or SSD wear, a raid10 of cause would be the better choice.