Abysmally slow restore from backup

I will try this on a bigger VM, but I'm not really having any issues with PBS restoring moderately sized VMs.

Code: 
new volume ID is 'local-zfs:vm-170-disk-0'
new volume ID is 'local-zfs:vm-170-disk-1'
restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@10.60.110.155:truenas --ns DER vm/170/2025-03-03T20:21:36Z drive-sata0.img.fidx /dev/zvol/rpool/data/vm-170-disk-0 --verbose --format raw --skip-zero
connecting to repository 'root@pam@10.60.110.155:truenas'
open block backend for target '/dev/zvol/rpool/data/vm-170-disk-0'
starting to restore snapshot 'vm/170/2025-03-03T20:21:36Z'
download and verify backup index
progress 1% (read 12582912 bytes, zeroes = 0% (0 bytes), duration 0 sec)
progress 2% (read 25165824 bytes, zeroes = 0% (0 bytes), duration 0 sec)
progress 3% (read 33554432 bytes, zeroes = 0% (0 bytes), duration 0 sec)
progress 4% (read 46137344 bytes, zeroes = 18% (8388608 bytes), duration 0 sec)
progress 5% (read 54525952 bytes, zeroes = 23% (12582912 bytes), duration 0 sec)
...truncated...
progress 98% (read 1031798784 bytes, zeroes = 12% (130023424 bytes), duration 54 sec)
progress 99% (read 1044381696 bytes, zeroes = 12% (130023424 bytes), duration 54 sec)
progress 100% (read 1052770304 bytes, zeroes = 12% (134217728 bytes), duration 54 sec)
restore image complete (bytes=1052770304, duration=54.83s, speed=18.31MB/s)
restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@10.60.110.155:truenas --ns DER vm/170/2025-03-03T20:21:36Z drive-sata1.img.fidx /dev/zvol/rpool/data/vm-170-disk-1 --verbose --format raw --skip-zero
connecting to repository 'root@pam@10.60.110.155:truenas'
open block backend for target '/dev/zvol/rpool/data/vm-170-disk-1'
starting to restore snapshot 'vm/170/2025-03-03T20:21:36Z'
download and verify backup index
progress 1% (read 859832320 bytes, zeroes = 13% (117440512 bytes), duration 10 sec)
...truncated...
progress 97% (read 83328237568 bytes, zeroes = 95% (79666610176 bytes), duration 73 sec)
progress 98% (read 84188069888 bytes, zeroes = 95% (80526442496 bytes), duration 73 sec)
progress 99% (read 85047902208 bytes, zeroes = 95% (81386274816 bytes), duration 73 sec)
progress 100% (read 85902491648 bytes, zeroes = 95% (82237718528 bytes), duration 73 sec)
restore image complete (bytes=85902491648, duration=73.93s, speed=1108.06MB/s)
rescan volumes...
TASK OK

80GB over 54 seconds isn't terrible, but I could see this being quite the issue when you are pulling a couple of TBs.
 
Yeah, except that's not actually 80 GBs of data you read from backup files, as most of that is probably unused space... and if it was backed up recently it could have been read mostly from cache, as well. I see that you're using TrueNAS via iSCSI - and we know nothing about the TrueNAS setup of yours to give those numbers any meaning.
 
Check my older posts in this thread... Veeam reads about 10x faster than PBS, when doing restore. That's on same hardware, with PBS and Veeam both using a ZFS dataset as it's backing store (configured with same parameters, on the same pool, on the same server).
 
Please explain your setup too, especially what is being used to store backups (what hard drives, what FS, what RAID level, ZFS details if using that, etc).
 
It is a TrueNAS 25.04 system with PBS running as a container with all available system resources accessible.

2x Intel(R) Xeon(R) CPU E5-2670 v3
160GB of DDR4 ECC
5x 1.92TB Samsung PM893 SATA SSD in RAIDz1
A 1.6TB NVMe SLOG

ZFS details on the dataset:
Type: FILESYSTEM
Sync: STANDARD
Compression Level: Inherit (LZ4)
Enable Atime: OFF
ZFS Deduplication: OFF
Case Sensitivity: ON
Path: ssd/pbs
Record Size: 128KiB

The TrueNAS and PVE are at totally different sites, using a site-to-site connection to backup and restore from. So it will max out at 1gbps no matter what. I can try some local restores once there is some Proxmox infra on that site, but currently there isn't.


The on-prem storage is a Pure X20 array attached with 2x25GbE links with iSCSI. It can currently max out the combined 25GbE connections.

Currently running a PBS backup of a 300GB VM that is 100% full (for testing). While running the backup I ran a fio test on the Pure storage to make sure it's not bottlenecking, I'd say it isn't.
1741630366028.png

Current status of the backup
Code: 
INFO: starting new backup job: vzdump 103 --remove 0 --notes-template '{{guestname}}' --mode snapshot --node der-pve3 --notification-mode auto --storage gar-truenas
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2025-03-10 08:24:15
INFO: status = running
INFO: VM Name: ubuntu
INFO: include disk 'scsi0' 'pure:vm-103-disk-0' 300G
Info :: Waiting (0s) for map volume "vm-103-disk-0"...
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/103/2025-03-10T17:24:15Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '62f959e2-e125-44f3-a36f-58ee0e7928ff'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: created new
INFO:   0% (2.6 GiB of 300.0 GiB) in 3s, read: 873.3 MiB/s, write: 66.7 MiB/s
INFO:   2% (6.1 GiB of 300.0 GiB) in 7s, read: 910.0 MiB/s, write: 50.0 MiB/s
INFO:   3% (9.1 GiB of 300.0 GiB) in 13s, read: 504.7 MiB/s, write: 38.0 MiB/s
INFO:   4% (12.2 GiB of 300.0 GiB) in 22s, read: 353.8 MiB/s, write: 24.9 MiB/s
INFO:   5% (16.1 GiB of 300.0 GiB) in 42s, read: 202.6 MiB/s, write: 17.6 MiB/s
INFO:   6% (18.0 GiB of 300.0 GiB) in 1m 18s, read: 53.1 MiB/s, write: 52.3 MiB/s
INFO:   7% (21.0 GiB of 300.0 GiB) in 2m 42s, read: 36.7 MiB/s, write: 35.6 MiB/s
INFO:   8% (24.6 GiB of 300.0 GiB) in 4m 4s, read: 44.4 MiB/s, write: 18.5 MiB/s
...truncated...
INFO:  33% (99.0 GiB of 300.0 GiB) in 48m 17s, read: 12.5 MiB/s, write: 12.1 MiB/s
INFO:  34% (102.0 GiB of 300.0 GiB) in 50m 6s, read: 28.6 MiB/s, write: 28.3 MiB/s

Current load on the PBS and PVE
1741630584169.png

1741630622583.png

So no resource contesting on the systems as a whole, but on the PVE server you can see one thread is totally maxed out
1741630746160.png
 
Last edited:
A very fast storage subsystem, all flash... I'd be interested in final results.
You could also use recordsize=1M on the recordset being used for PBS chunk store.

PwrBank said:
The TrueNAS and PVE are at totally different sites, using a site-to-site connection to backup and restore from. So it will max out at 1gbps no matter what. I can try some local restores once there is some Proxmox infra on that site, but currently there isn't.
Click to expand...
I suppose 1 Gbps link will be limiting factor here, but if you don't get restores up to max speed than software is the limit...
Please make sure to check actual network speed, because the numbers that PBS is giving you are... ehm... calculated, to give you more impressive numbers. So check what you get on the wire. If you get >100 MB/s (120 MB/s actually) than network was the limit, not PBS. And vice versa.
 
Last edited:
Info dump time. This is an incomplete backup test, but just wanted to get it out there for now. I will do a separate test on restores.

The 300GB backup over the WAN finished with PBS
Code: 
INFO: starting new backup job: vzdump 103 --remove 0 --notes-template '{{guestname}}' --mode snapshot --node der-pve3 --notification-mode auto --storage gar-truenas
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2025-03-10 08:24:15
INFO: status = running
INFO: VM Name: ubuntu
INFO: include disk 'scsi0' 'pure:vm-103-disk-0' 300G
Info :: Waiting (0s) for map volume "vm-103-disk-0"...
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/103/2025-03-10T17:24:15Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '62f959e2-e125-44f3-a36f-58ee0e7928ff'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: created new
INFO: 0% (2.6 GiB of 300.0 GiB) in 3s, read: 873.3 MiB/s, write: 66.7 MiB/s
INFO: 2% (6.1 GiB of 300.0 GiB) in 7s, read: 910.0 MiB/s, write: 50.0 MiB/s
...truncated...
INFO: 91% (273.0 GiB of 300.0 GiB) in 3h 26m 30s, read: 40.8 MiB/s, write: 39.1 MiB/s
INFO: 92% (276.2 GiB of 300.0 GiB) in 3h 28m 27s, read: 28.0 MiB/s, write: 25.9 MiB/s
INFO: 93% (279.1 GiB of 300.0 GiB) in 3h 31m 38s, read: 15.4 MiB/s, write: 14.8 MiB/s
INFO: 94% (282.1 GiB of 300.0 GiB) in 3h 33m 11s, read: 33.0 MiB/s, write: 30.6 MiB/s
INFO: 95% (285.1 GiB of 300.0 GiB) in 3h 35m 30s, read: 21.8 MiB/s, write: 20.8 MiB/s
INFO: 96% (288.1 GiB of 300.0 GiB) in 3h 37m 10s, read: 31.2 MiB/s, write: 28.8 MiB/s
INFO: 97% (291.1 GiB of 300.0 GiB) in 3h 38m 8s, read: 51.9 MiB/s, write: 49.4 MiB/s
INFO: 98% (294.0 GiB of 300.0 GiB) in 3h 39m 20s, read: 41.8 MiB/s, write: 40.1 MiB/s
INFO: 99% (297.0 GiB of 300.0 GiB) in 3h 40m 27s, read: 46.4 MiB/s, write: 42.7 MiB/s
INFO: 100% (300.0 GiB of 300.0 GiB) in 3h 41m 52s, read: 35.7 MiB/s, write: 34.2 MiB/s
INFO: backup is sparse: 15.54 GiB (5%) total zero data
INFO: backup was done incrementally, reused 47.12 GiB (15%)
INFO: transferred 300.00 GiB in 13317 seconds (23.1 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 103 (03:41:58)
INFO: Backup finished at 2025-03-10 12:06:13
INFO: Backup job finished successfully
TASK OK

So 3.75 hours

The same VM, with the same connection over the WAN, to the same TrueNAS but using Veeam (The worker had 32 vCPU and 32GB of RAM)
1741720358664.png

It took longer for Veeam to do the backup.


Now, this is where I started a different test using an onsite TrueNAS system. This system is pretty bonkers, but we wanted an all flash backup repo. This is a PBS container running on the TrueNAS system with no limit on resources.
1741721026853.png

Specs:
2x AMD EPYC 9354
192GB of DDR5 ECC
5x 61.44TB Solidigm D5-P5536 NVMe in RAIDz1 (Tested read speeds over 30GB/s)

ZFS details on the dataset:
TrueNAS 25.04
Type: FILESYSTEM
Sync: STANDARD
Compression Level: Inherit (LZ4)
Enable Atime: OFF
ZFS Deduplication: YES
Case Sensitivity: ON
Path: ssd/pbs
Record Size: 128KiB

So from PVE to PBS over a 10GbE link
Code: 
INFO: starting new backup job: vzdump 103 --storage pbsnvme --notification-mode auto --mode snapshot --node der-pve3 --notes-template '{{guestname}}' --remove 0
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2025-03-11 09:38:50
INFO: status = running
INFO: VM Name: ubuntu
INFO: include disk 'scsi0' 'pure:vm-103-disk-0' 300G
Info :: Volume "vm-103-disk-0" is already connected to host "der-pve3".
wwid '3624a937096622a35a07e4a33000124ac' added
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/103/2025-03-11T18:38:50Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '2833c208-9002-43a2-bc07-74fb081ca1d1'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (1.3 GiB of 300.0 GiB) in 3s, read: 438.7 MiB/s, write: 429.3 MiB/s
INFO: 1% (3.1 GiB of 300.0 GiB) in 7s, read: 477.0 MiB/s, write: 470.0 MiB/s
INFO: 2% (8.3 GiB of 300.0 GiB) in 10s, read: 1.7 GiB/s, write: 369.3 MiB/s
INFO: 3% (9.6 GiB of 300.0 GiB) in 13s, read: 434.7 MiB/s, write: 432.0 MiB/s
INFO: 4% (12.1 GiB of 300.0 GiB) in 18s, read: 524.8 MiB/s, write: 512.8 MiB/s
...truncated...
INFO: 96% (288.4 GiB of 300.0 GiB) in 6m 50s, read: 791.0 MiB/s, write: 764.0 MiB/s
INFO: 97% (291.5 GiB of 300.0 GiB) in 6m 54s, read: 786.0 MiB/s, write: 778.0 MiB/s
INFO: 98% (294.6 GiB of 300.0 GiB) in 6m 58s, read: 791.0 MiB/s, write: 773.0 MiB/s
INFO: 99% (297.6 GiB of 300.0 GiB) in 7m 3s, read: 633.6 MiB/s, write: 620.8 MiB/s
INFO: 100% (300.0 GiB of 300.0 GiB) in 7m 7s, read: 602.0 MiB/s, write: 592.0 MiB/s
INFO: Waiting for server to finish backup validation...
INFO: backup is sparse: 9.11 GiB (3%) total zero data
INFO: backup was done incrementally, reused 9.64 GiB (3%)
INFO: transferred 300.00 GiB in 431 seconds (712.8 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 103 (00:07:12)
INFO: Backup finished at 2025-03-11 09:46:02
INFO: Backup job finished successfully
TASK OK

You can see a few cores on the host is getting pegged
1741720799717.png

Confirmed the speeds on the TrueNAS and host
1741720817233.png
1741720822283.png

I'm trying to figure out how to get Veeam to use the 10GbE network, the Proxmox config is a little complicated and I don't think Veeam was anticipating people tagging VLANs on the NICs of the VMs instead of the bridges. So, I'll post results of the backup test from PVE to Veeam to the stupid flash server when I can get it working.

Here is it on the 1GbE though.
1741720931881.png
 
Last edited:
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom