Extremely Slow PBS Speeds

ehammer

New Member
Jan 12, 2022
24
3
3
32
Keep in mind that PBS really needs low latencies. Veeam will use big incremental image files. PBS chops everything in small (some KB and up to a maximum of 4 MiB) chunks. So if you backup a 4TB disk it will need to hash and write atleast 1 million small files. NFS/SMB isn't that great at handling millions of small files.

I'm just now thinking about this, but my PBS vm sits on a shared NVMe VM storage. Couldn't I write the backups locally to disk, then use cron to move the resulting local backup files to a SMB share?

I didn't actually look at the file system to see if PBS stores the VM backups in millions of tiny individual files, or if its just tiny individual files while being transfered from a proxmox host to PBS?
 

Dunuin

Famous Member
Jun 30, 2020
6,114
1,410
149
Germany
Its stored as millions of small files and the atime is important for pruning/GC, because all backups will get deleted by the GC that were not touched by the prune task. So I guess its problematic to copy files over.
 
  • Like
Reactions: ehammer

ehammer

New Member
Jan 12, 2022
24
3
3
32
After some modification, this is my benchmark (PBS still in a VM):
I'm about to run some backups to see what the overall speed is, but this is looking much better. Pretty sure it was the cpu switch from kvm to host.

Uploaded 365 chunks in 5 seconds.
Time per request: 13784 microseconds.
TLS speed: 304.27 MB/s
SHA256 speed: 218.28 MB/s
Compression speed: 476.22 MB/s
Decompress speed: 1203.41 MB/s
AES256/GCM speed: 2464.00 MB/s
Verify speed: 339.96 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 304.27 MB/s (25%) │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 218.28 MB/s (11%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed │ 476.22 MB/s (63%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed │ 1203.41 MB/s (100%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed │ 339.96 MB/s (45%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed │ 2464.00 MB/s (68%) │
└───────────────────────────────────┴─────────────────────┘
 

ehammer

New Member
Jan 12, 2022
24
3
3
32
Well I guess I won't be running any backups...

next-run: property is not defined in schema and the schema does not allow additional properties

pbsfail.png

Don't know what that even means.
 

Dunuin

Famous Member
Jun 30, 2020
6,114
1,410
149
Germany
Did you upgrade your PVE from 7.0 to 7.1 recently? I got that error after the upgrade, because I didn'T refreshed the browser cache (CTRL+F5) after the upgrade. Because of that I was using the cached PVE7.0 webui with the PVE 7.1 backend. This resulted in wrong backup configs because PVE switched the b ackup scheduler with PVE 7.1. So best you do a CTRL+F5 and then create that backup task again.
 

ehammer

New Member
Jan 12, 2022
24
3
3
32
Did you upgrade your PVE from 7.0 to 7.1 recently? I got that error after the upgrade, because I didn'T refreshed the browser cache (CTRL+F5) after the upgrade. Because of that I was using the cached PVE7.0 webui with the PVE 7.1 backend. This resulted in wrong backup configs because PVE switched the b ackup scheduler with PVE 7.1. So best you do a CTRL+F5 and then create that backup task again.
I did an upgrade, but I was already on 7.1.
 

ehammer

New Member
Jan 12, 2022
24
3
3
32
I have tried clearing my browser cache, restarting my computer, and recreating the backup tasks. Same error
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,523
908
163
I have tried clearing my browser cache, restarting my computer, and recreating the backup tasks. Same error
Please send your:

> pveversion -v
 

ehammer

New Member
Jan 12, 2022
24
3
3
32
Please send your:

> pveversion -v
The backup job still ran at its normal time, I just cant run it manually.

# pveversion -v

proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-9 (running version: 7.1-9/0740a2bc)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.13.19-3-pve: 5.13.19-6
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.3-1
proxmox-backup-file-restore: 2.1.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1
 

ehammer

New Member
Jan 12, 2022
24
3
3
32
It seems the cpu change to host has made the biggest improvement. Backup jobs now run at 500-800Mbit/s.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!