vm restore slower than absolute 0

J

jwsl224

Member
Apr 6, 2024
76
4
13
we have
Code: 
proxmox-backup-server 3.4.9-2 running version: 3.4.9
(yes i know; we'll be upgrading soon) and it's been absolutely stellar up till now. but today, when i went to restore a vm, i got
Code: 
restore image complete (bytes=540672, duration=0.22s, speed=2.33MB/s)
it's an r730 running all ssd's on zfs. there are no warnings in idrac, the cpu is close to 0 utilization, and io delay is not even present. as you can see in the log below, the restore process is taking hours. the destination pve host is equally bored with this transfer. they are connected to each other at 10gb/s rate, and backups to the pbs ~200 MB/s. there must be something about the read of the drives, but i'm sure not finding anything. can anybody perhaps point me in a direction? this is the weirdest thing ever..



new volume ID is 'nvme4r10:vm-209-disk-0'
new volume ID is 'nvme4r10:vm-209-disk-1'
new volume ID is 'nvme4r10:vm-209-disk-2'
restore proxmox backup image: /usr/bin/pbs-restore --repository backup@pbs@10.1.100.243:storage1 vm/208/2026-09-03T23:32:05Z drive-efidisk0.img.fidx /dev/zvol/nvme4r10/vm-209-disk-0 --verbose --format raw --keyfile /etc/pve/priv/storage/slpbs4.enc --skip-zero
connecting to repository 'backup@pbs@10.1.100.243:storage1'
using up to 4 threads
open block backend for target '/dev/zvol/nvme4r10/vm-209-disk-0'
starting to restore snapshot 'vm/208/2026-09-03T23:32:05Z'
download and verify backup index
fetching up to 16 chunks in parallel
progress 100% (read 540672 bytes, zeroes = 0% (0 bytes), duration 0 sec)
restore image complete (bytes=540672, duration=0.22s, speed=2.33MB/s)
restore proxmox backup image: /usr/bin/pbs-restore --repository backup@pbs@10.1.100.243:storage1 vm/208/2026-09-03T23:32:05Z drive-scsi0.img.fidx /dev/zvol/nvme4r10/vm-209-disk-1 --verbose --format raw --keyfile /etc/pve/priv/storage/slpbs4.enc --skip-zero
connecting to repository 'backup@pbs@10.1.100.243:storage1'
using up to 4 threads
open block backend for target '/dev/zvol/nvme4r10/vm-209-disk-1'
starting to restore snapshot 'vm/208/2026-09-03T23:32:05Z'
download and verify backup index
fetching up to 16 chunks in parallel
progress 1% (read 536870912 bytes, zeroes = 42% (230686720 bytes), duration 71 sec)
progress 2% (read 1073741824 bytes, zeroes = 21% (234881024 bytes), duration 251 sec)
progress 3% (read 1610612736 bytes, zeroes = 15% (251658240 bytes), duration 490 sec)
progress 4% (read 2147483648 bytes, zeroes = 11% (251658240 bytes), duration 760 sec)
progress 5% (read 2684354560 bytes, zeroes = 12% (343932928 bytes), duration 910 sec)
progress 6% (read 3221225472 bytes, zeroes = 10% (348127232 bytes), duration 1166 sec)
progress 7% (read 3758096384 bytes, zeroes = 10% (381681664 bytes), duration 1354 sec)
progress 8% (read 4294967296 bytes, zeroes = 9% (390070272 bytes), duration 1592 sec)
progress 9% (read 4831838208 bytes, zeroes = 8% (390070272 bytes), duration 1793 sec)
progress 10% (read 5368709120 bytes, zeroes = 7% (390070272 bytes), duration 1986 sec)
 
Last edited:
Please share some more details of the restore target, especially pveversion -v, cat /etc/pve/storage.cfg and details on the target storage. Further, how high is iodelay and the output of zpool iostat -r (run once before restore to clear pre-accumulated vaules, then again after restore).
 
Chris said:
Please share some more details of the restore target, especially pveversion -v, cat /etc/pve/storage.cfg and details on the target storage. Further, how high is iodelay and the output of zpool iostat -r (run once before restore to clear pre-accumulated vaules, then again after restore).
Click to expand...
hi chris, thanks for dropping by.
the target server is much more powerful than the pbs. the pbs is sata ssd's and the target pve nodes are either gen 4 or gen 5 nvme's. i tried two different nodes to restore to and it didn't change anything. here are the details you requested:
Code: 
root@MD72-HB2-1:~# pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.8.12-23-pve)
pve-manager: 8.4.19 (running version: 8.4.19/a68fb383814bb1e6)
proxmox-kernel-helper: 8.1.4
proxmox-kernel-6.8.12-23-pve-signed: 6.8.12-23
proxmox-kernel-6.8: 6.8.12-23
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.10-pve2~bpo12+1
criu: 3.17.1-2+deb12u2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.2
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.3
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.3
libpve-cluster-perl: 8.1.3
libpve-common-perl: 8.3.8
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.3
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.8
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-2
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2.1
proxmox-backup-client: 3.4.7-1
proxmox-backup-file-restore: 3.4.7-1
proxmox-backup-restore-image: 0.7.0
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.4
proxmox-mail-forward: 0.3.3
proxmox-mini-journalreader: 1.5
proxmox-offline-mirror-helper: 0.6.8
proxmox-widget-toolkit: 4.3.17
pve-cluster: 8.1.3
pve-container: 5.3.5
pve-docs: 8.4.2
pve-edk2-firmware: 4.2025.05-1~bpo12+1
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.2
pve-firmware: 3.16-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.5
pve-qemu-kvm: 9.2.0-7
pve-xtermjs: 5.5.0-2
qemu-server: 8.4.8
smartmontools: 7.3-pve1
spiceterm: 3.3.1
swtpm: 0.8.0+pve1
vncterm: 1.8.2
zfsutils-linux: 2.2.9-pve1

Code: 
root@MD72-HB2-1:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,backup,vztmpl

zfspool: local-zfs
        pool rpool/data
        content rootdir,images
        sparse 1

zfspool: SATA3
        pool SATA3
        content images,rootdir
        mountpoint /SATA3
        nodes MD72-HB2-1
        sparse 1

zfspool: r730xd-1-rust
        pool r730xd-1-rust
        content images,rootdir
        nodes r730xd-1
        sparse 1

zfspool: nvme5
        pool nvme5
        content images,rootdir
        mountpoint /nvme5
        nodes MW83-RP0-1
        sparse 1

pbs: slpbs4
        datastore storage1
        server
        content backup
        encryption-key
        fingerprint
        prune-backups keep-all=1
        username @pbs

zfspool: nvme4r10
        pool nvme4r10
        content rootdir,images
        mountpoint /nvme4r10
        nodes MD72-HB2-1
        sparse 1

pbs: slrempbs
        datastore storage
        server
        content backup
        encryption-key
        fingerprint
        prune-backups keep-all=1
        username @pbs

the restore i was trying to do was from slpbs4 to zfs pool nvme4r10, which is nvme gen 4 raid 10.

here is the zpool iostat -r on the pbs during the restore:
Code: 
rpool         sync_read    sync_write    async_read    async_write      scrub         trim         rebuild
req_size      ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg
----------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
512             0      0      0      0      0      0      0      0      0      0      0      0      0      0
1K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4K          2.53K      0  58.8K      0     74      0   401K      0     34      0      0      0      0      0
8K            398      0  7.75K      0    634      0  74.0K   195K      4     12      0      0      0      0
16K           437      0      0      0     37      0  11.8K   138K      4     14      0      0      0      0
32K         1.75K      0    551      0    756      0  12.3K  63.0K     12     11      0      0      0      0
64K           862      0    617     22    637     15  9.04K  23.8K      0      7      0      0      0      0
128K          140      0    260      1     26      2  1.12K  1.33K      0      0      0      0      0      0
256K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
512K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
1M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
8M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
16M             0      0      0      0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------

storage1      sync_read    sync_write    async_read    async_write      scrub         trim         rebuild
req_size      ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg
----------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
512             0      0      0      0      0      0      0      0      0      0      0      0      0      0
1K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4K          1.03M      0  47.8K      0   259K      0  1.63M      0    129      0      0      0      0      0
8K           133K      0    372      2   127K      4  77.7K   402K     27      9      0      0      0      0
16K         30.7M    130  33.6K     26  21.9M  26.6K   593K   160K      0     12      0      0      0      0
32K             0  3.04K      0    898      0   697K      0   141K      0      4      0      0      0      0
64K            24  8.59K     48  2.19K      0   403K      0   204K      0      0      0      0      0      0
128K            0     23      0      0      0    101      0  5.20K      0      0      0      0      0      0
256K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
512K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
1M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
8M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
16M             0      0      0      0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------
root@slpbs4:~# zpool iostat -r

rpool         sync_read    sync_write    async_read    async_write      scrub         trim         rebuild
req_size      ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg
----------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
512             0      0      0      0      0      0      0      0      0      0      0      0      0      0
1K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4K          2.53K      0  58.9K      0     74      0   402K      0     34      0      0      0      0      0
8K            398      0  7.76K      0    634      0  74.0K   195K      4     12      0      0      0      0
16K           437      0      0      0     37      0  11.8K   138K      4     14      0      0      0      0
32K         1.75K      0    551      0    756      0  12.3K  63.1K     12     11      0      0      0      0
64K           862      0    617     22    637     15  9.05K  23.9K      0      7      0      0      0      0
128K          140      0    260      1     26      2  1.12K  1.33K      0      0      0      0      0      0
256K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
512K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
1M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
8M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
16M             0      0      0      0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------

storage1      sync_read    sync_write    async_read    async_write      scrub         trim         rebuild
req_size      ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg
----------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
512             0      0      0      0      0      0      0      0      0      0      0      0      0      0
1K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4K          1.03M      0  48.0K      0   259K      0  1.63M      0    129      0      0      0      0      0
8K           133K      0    372      2   127K      4  77.8K   402K     27      9      0      0      0      0
16K         30.7M    130  33.6K     26  21.9M  26.6K   593K   160K      0     12      0      0      0      0
32K             0  3.04K      0    898      0   697K      0   141K      0      4      0      0      0      0
64K            24  8.59K     48  2.19K      0   403K      0   204K      0      0      0      0      0      0
128K            0     23      0      0      0    101      0  5.20K      0      0      0      0      0      0
256K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
512K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
1M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
8M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
16M             0      0      0      0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------
i also did a zpool iostat 1 during restore on both pbs and target node. attached them as txt file since it got too long
 

Attachments

Last edited:
Chris said:
Please share some more details of the restore target, especially pveversion -v, cat /etc/pve/storage.cfg and details on the target storage. Further, how high is iodelay and the output of zpool iostat -r (run once before restore to clear pre-accumulated vaules, then again after restore).
Click to expand...
hey Chris,
thankfully, the problem seems to have solved itself. restores are now going at 10gbps as they used to. i'm not sure what the deal here was, there doesn't seem to have been any kind of background scrub going on anywhere, nor any detectable system load; hopefully it won't come up again. the storage controller is operating in passthrough mode, so there wasn't any controller tasks going on either. if you have any advice of what i should keep an eye on, i'd appreciate it. otherwise, this can be considered solved for now.
 
You must log in or register to reply here.
Top Bottom
Back