Finding bottle necks in PBS performance.

[CelticWebs]

I've got 2 PBS setups, one runs on a VM on a Synology DS1019+ NAS as a VM and another runs on a much larger, Xeon / Truenas scale based Server.

Server 1 Spec
The Synology runs PBS in VM
VM is using 4 cores, single thread (4 thread total)
4GB ram
5 disk, BTRFS pool with single disk parity.
Storage access for PBS VM is via NFS share from its host
Total PVE Backups occupied space approx 2TiB

Server 2 Spec
The Xeon based server runs Trueness scale running PBS in a VM
VM has 8 Cores, dual thread allocated (16 thread total)
16GB memory
24 disk, ZFS Draid draid3:8d:22c:1s-0
Storage access for PBS VM is via NFS from it's host
Total PVE Backups occupied space approx 1TiB


There is a bottle neck somewhere causing the Truenas VM of PBS to be extremely slow to complete garabage collection, often a week or more. While the Synology completes in several hours. This makes no sense when the Truenas Vm has more processing power and more Ram to work with, likely a faster pool capability with the larger quantity of disks in DRAID too.

While I understand it's suggested that spinners aren't the best for PBS, I had expected the Xeon / Truenas VM to perform better than the little Synology with its tiny CPU and lesser memory.

Are there some tests that are available in PBS to find what's causing the bottle neck? Any tips for tweaks that may improve performance without additional hardware? Even if the Xeon / Truenas worked as well as the sinology unit, I'd be happy. The more powerful system is so much slower it's got to be something blatantly obvious that I'm somehow overlooking.

Thanks for your suggestions!

 

[CelticWebs]

I just did something I hadn't done before and did a very simple disk speed and latency test with dd


Speed
dd if=/dev/zero of=test1.img bs=1G count=1 oflag=dsync
Synology
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.80247 s, 138 MB/s

TrueNas
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.77884 s, 386 MB/s

Latency
dd if=/dev/zero of=test2.img bs=512 count=1000 oflag=dsync

Synology
512000 bytes (512 kB, 500 KiB) copied, 0.328728 s, 1.6 MB/s

Truenas
512000 bytes (512 kB, 500 KiB) copied, 11.3417 s, 45.1 kB/s

So here's my issue, the latency is awful on the Truenas, so likely nothing whatsoever to do with PBS, must be something to do with the connect to the host for the NFS share. Off to the TrueNas forum I go!

Thanks anyway for looking, if anyone has any suggestions, please do fire them at me!

 

[dietmar]

Are there some tests that are available in PBS to find what's causing the bottle neck?

See https://forum.proxmox.com/threads/how-fast-is-your-backup-datastore-benchmark-tool.72750/

 

[CelticWebs]

See https://forum.proxmox.com/threads/how-fast-is-your-backup-datastore-benchmark-tool.72750/

Thank you

These were my results from the serve that takes days to do a Garbage collect on 1TiB.

 

[Dunuin]

This makes no sense when the Truenas Vm has more processing power and more Ram to work with, likely a faster pool capability with the larger quantity of disks in DRAID too.

GC is all about IOPS performance of your storage as there is millions over millions of small random writes/reads. Thats why its recommended to store your backups on SSDs. Adding SSDs as special devices to your ZFS pool would help too.

dd if=/dev/zero of=test1.img bs=1G count=1 oflag=dsync

dd is not a benchmark tool. Especially with the block-level compression of ZFS those zeros will be insnely compressed so that not much will be written to the pool. Use fio for that.

 

[CelticWebs]

GC is all about IOPS performance of your storage as there is millions over millions of small random writes/reads. Thats why its recommended to store your backups on SSDs. Adding SSDs as special devices to your ZFS pool would help too.

Hi Duncan,

Thanks for your responses as always. My biggest query here is that I have a small, low powered NAS with a 5 disk array containing 2TB of backups, outperforming a much higher power machine with 24 disks striped array and only 1TB of data, though strangely when I run the tests suggested via the link given by @dietmar, the better spec system does indeedperfrom better, but when it's doing a Garabage collection it takes10 times longer than the small Nas which has twic e as much data on it. It's very strange and make no sense given the benchmark figures.