Expected backup speed

[linum]

I just created a new proxmox backup server and made my first test. The backup storage consists of 4 vdevs with a raidz1 that is build with 3x 18TB Seagate EXOS (ST18000NM000J) HDDs. I also add a log mirrored vdev that uses two Intel Optane NVMe. The network is a bond of two 10GbE cards. The backup server and the originating server are using Xeon 62xx CPU, the originating server has 1TB RAM, the backup server right now only 512GB. The originating server VMs are located on an Intel nvme ZFS mirrored storage.

Currently I get a write speed with is about 240.0 MiB/s. And I can see the 12 HDD disks activity LED flashes only every 3-4 seconds. I expected that the HDD LED stays permanently "on". Since both servers seems to have plenty of ressources I wonder what I'm missing to get the hardware better utilized.

INFO: include disk 'scsi0' 'nvm01:vm-703-disk-0' 128G INFO: snapshots found (not included into backup) INFO: creating Proxmox Backup Server archive 'vm/703/2022-05-25T20:38:55Z' INFO: starting kvm to execute backup task INFO: enabling encryption INFO: started backup task 'dab4a47b-f06e-48b3-a1eb-e414c091d8de' INFO: scsi0: dirty-bitmap status: created new INFO: 0% (764.0 MiB of 128.0 GiB) in 3s, read: 254.7 MiB/s, write: 234.7 MiB/s INFO: 1% (1.4 GiB of 128.0 GiB) in 6s, read: 228.0 MiB/s, write: 228.0 MiB/s INFO: 2% (2.8 GiB of 128.0 GiB) in 12s, read: 233.3 MiB/s, write: 233.3 MiB/s INFO: 3% (4.0 GiB of 128.0 GiB) in 17s, read: 240.0 MiB/s, write: 240.0 MiB/s INFO: 4% (5.3 GiB of 128.0 GiB) in 23s, read: 234.0 MiB/s, write: 234.0 MiB/s INFO: 5% (6.5 GiB of 128.0 GiB) in 28s, read: 232.8 MiB/s, write: 232.8 MiB/s INFO: 6% (7.8 GiB of 128.0 GiB) in 34s, read: 233.3 MiB/s, write: 233.3 MiB/s INFO: 7% (9.2 GiB of 128.0 GiB) in 40s, read: 233.3 MiB/s, write: 233.3 MiB/s INFO: 8% (10.5 GiB of 128.0 GiB) in 45s, read: 264.8 MiB/s, write: 264.0 MiB/s INFO: 9% (11.7 GiB of 128.0 GiB) in 50s, read: 252.0 MiB/s, write: 252.0 MiB/s INFO: 10% (12.9 GiB of 128.0 GiB) in 55s, read: 241.6 MiB/s, write: 241.6 MiB/s INFO: 11% (14.3 GiB of 128.0 GiB) in 1m 1s, read: 242.0 MiB/s, write: 242.0 MiB/s INFO: 12% (15.4 GiB of 128.0 GiB) in 1m 6s, read: 221.6 MiB/s, write: 221.6 MiB/s INFO: 13% (16.7 GiB of 128.0 GiB) in 1m 12s, read: 224.0 MiB/s, write: 223.3 MiB/s INFO: 14% (18.0 GiB of 128.0 GiB) in 1m 18s, read: 219.3 MiB/s, write: 217.3 MiB/s INFO: 15% (19.2 GiB of 128.0 GiB) in 1m 24s, read: 208.7 MiB/s, write: 207.3 MiB/s INFO: 16% (20.6 GiB of 128.0 GiB) in 1m 30s, read: 238.7 MiB/s, write: 238.7 MiB/s INFO: 17% (21.8 GiB of 128.0 GiB) in 1m 35s, read: 245.6 MiB/s, write: 244.8 MiB/s INFO: 18% (23.2 GiB of 128.0 GiB) in 1m 41s, read: 240.0 MiB/s, write: 234.7 MiB/s INFO: 19% (24.5 GiB of 128.0 GiB) in 1m 47s, read: 224.0 MiB/s, write: 222.0 MiB/s INFO: 20% (25.8 GiB of 128.0 GiB) in 1m 52s, read: 264.0 MiB/s, write: 240.8 MiB/s INFO: 21% (27.1 GiB of 128.0 GiB) in 1m 56s, read: 325.0 MiB/s, write: 250.0 MiB/s INFO: 22% (28.4 GiB of 128.0 GiB) in 2m, read: 347.0 MiB/s, write: 259.0 MiB/s INFO: 23% (29.4 GiB of 128.0 GiB) in 2m 3s, read: 341.3 MiB/s, write: 254.7 MiB/s INFO: 24% (31.9 GiB of 128.0 GiB) in 2m 8s, read: 501.6 MiB/s, write: 213.6 MiB/s INFO: 27% (34.9 GiB of 128.0 GiB) in 2m 11s, read: 1.0 GiB/s, write: 101.3 MiB/s INFO: 28% (36.9 GiB of 128.0 GiB) in 2m 14s, read: 680.0 MiB/s, write: 234.7 MiB/s INFO: 30% (39.2 GiB of 128.0 GiB) in 2m 17s, read: 780.0 MiB/s, write: 160.0 MiB/s INFO: 37% (48.5 GiB of 128.0 GiB) in 2m 20s, read: 3.1 GiB/s, write: 0 B/s INFO: 45% (57.8 GiB of 128.0 GiB) in 2m 23s, read: 3.1 GiB/s, write: 0 B/s INFO: 52% (67.0 GiB of 128.0 GiB) in 2m 26s, read: 3.1 GiB/s, write: 0 B/s INFO: 59% (76.3 GiB of 128.0 GiB) in 2m 29s, read: 3.1 GiB/s, write: 0 B/s INFO: 66% (85.6 GiB of 128.0 GiB) in 2m 32s, read: 3.1 GiB/s, write: 0 B/s INFO: 72% (93.4 GiB of 128.0 GiB) in 2m 35s, read: 2.6 GiB/s, write: 57.3 MiB/s INFO: 78% (99.9 GiB of 128.0 GiB) in 2m 38s, read: 2.2 GiB/s, write: 106.7 MiB/s INFO: 83% (106.9 GiB of 128.0 GiB) in 2m 41s, read: 2.3 GiB/s, write: 73.3 MiB/s INFO: 87% (111.9 GiB of 128.0 GiB) in 2m 44s, read: 1.6 GiB/s, write: 173.3 MiB/s INFO: 93% (119.3 GiB of 128.0 GiB) in 2m 47s, read: 2.5 GiB/s, write: 81.3 MiB/s INFO: 99% (127.5 GiB of 128.0 GiB) in 2m 50s, read: 2.7 GiB/s, write: 36.0 MiB/s INFO: 100% (128.0 GiB of 128.0 GiB) in 2m 52s, read: 240.0 MiB/s, write: 178.0 MiB/s INFO: backup is sparse: 95.30 GiB (74%) total zero data INFO: backup was done incrementally, reused 95.30 GiB (74%) INFO: transferred 128.00 GiB in 172 seconds (762.0 MiB/s) INFO: stopping kvm after backup task INFO: adding notes to backup INFO: Finished Backup of VM 703 (00:02:58) INFO: Backup finished at 2022-05-25 22:41:53

Software versions used (on both servers)
libproxmox-backup-qemu0 1.3.1-1 proxmox-backup-client 2.2.1-1 proxmox-backup-docs 2.2.1-1 proxmox-backup-file-restore 2.2.1-1 proxmox-backup-restore-image 0.3.1 proxmox-backup-server 2.2.1-1

 

[Dunuin]

And I can see the 12 HDD disks activity LED flashes only every 3-4 seconds. I expected that the HDD LED stays permanently "on". Since both servers seems to have plenty of ressources I wonder what I'm missing to get the hardware better utilized.

By default ZFS will only flush the async writes cached in ARC every 5 seconds to the disks. PBS needs IOPS which HDDs won't really offer so your HDDs IOPS performance might be the bottleneck. If that is the case you should think about adding a pair (or triplet) of mirrored SSDs as special devices to store the metadata so the HDDs only need to store the data. Keep in mind that PBS stores everything as small chunk files with a max size of 4MB per file. So lets say your average chunk size might be 2MB then a full 72 TB datastore would consist of 36 million files that all need to be read/written (sometimes a bit randomly and not that much sequentially because of the fragmentation caused by deduplication) when doing a maintaince task like a GC which might last many hours or even days. And that pair of log SSDs will only help with sync writes, not with async writes.

 

[linum]

Are you talking about the new "special" device vdev? I already added a (s)log vdev to the pool, but just looking with atop what's going on everything seems to be more or less idle..

PS: I will try the special vdev.

 

[Dunuin]

Are you talking about the new "special" device vdev?

Yes. "special" vdev will speed up all write and read operations, even async writes. But keep in mind that this is no cache. You can't remove that vdev later without destroying the pool and if your special vdev SSD fails you loose all data on your HDDs. So make sure you get the same reliability with the special devices as with the normal data vdevs.

I already added a (s)log vdev to the pool, but just looking with atop what's going on everything seems to be more or less idle..

Yeah, its only caching sync writes and PBS is primarily doing async writes.

 

[linum]

Not being able to remove the special vdev device makes testing not easier... But since this is a new setup I can try diffent things. The (s)log device really doesn't help, I currently monitor the iostats with "zpool iostat pbs01 -vly 1" which seems pretty informative.

The "special" vdev device seems to support only mirroring according to same statement I found. But I lack a thrid suitable nvme anyway. So maybe it's a good idea to rebuild the storage with mirrored vdevs only...

But it seems that the backup only uses one CPU core and even this one is not 100% busy. So I still wonder what is the cause for the, in my humble opionion, low backup speed.

 

[Dunuin]

Not being able to remove the special vdev device makes testing not easier...

What can be removed is a cache vdev with secondarycache=metadata. This works similar to a special vdev but can be lost/remove without a problem as it is just a read cache. But you won't see any faster writes with that. It just speeds up the maintaince tasks and restores.

The "special" vdev device seems to support only mirroring according to same statement I found. But I lack a thrid suitable nvme anyway.

Jup, only mirrors or striped mirrors (raid10). But you can create bigger mirrors that for example span ober 3 disks so 2 disks of that mirror might fail without loosing data.

So maybe it's a good idea to rebuild the storage with mirrored vdevs only...

Jup, that should give 50% more IOPS performance and faster resilvering.

As it isn't recommended at all to use HDDs with PBS (PBS documentation recommends local SSD only datastores) I also would use a 12x 18 TB HDDs as striped mirror and then 2x 500GB SSDs as mirrored special vdevs. that way you get the best IOPS performance out of that HDDs.
And if you got free 3.5" slot I also would add a 13th HDDs as a hot spare. 18TB HDDs take long to resilver.

 

[ReeceHibbett]

I am agree with your answers my problem is solved. I am very happy

Have a good day to all of you

 

[fabian]

just a note: special vdevs can actually be removed, provided you don't use raidz in your pool the data stored will then be evacuated to the rest of the pool (which means that as long as it is referenced, it will be a bit slower to access because there's a layer of indirection involved).

but there still is a big caveat compared to cache/log devices - if you lose a cache device due to hardware failure, no data is lost (it was just containing another copy of some of the data after all). if you lose a log device due to hardware failure, at most the last few seconds of sync writes should be lost (with the default transaction settings). if you lose a special device due to hardware failure, your pool is gone (because all the data stored on the special vdevs is only stored there, nowhere else). so it's rather important that your special vdev has a level of redundancy that you can tolerate (which might mean triple-mirroring on medium to big pools), that you monitor its status/wearout, and most important of all, that you have a working backup in place that gets tested regularly.

 

[linum]

And just for reference, use zdb -Lbbbs POOLNAME to determinate how big your special vdev. Level 0 ZFS Plain File and Level 0 zvol objects do NOT go to the metadata SSD, but everything else would.
I will give it a try soon, just need to get more disks to make some tests.

 

[fabian]

yep, that gives you a rough estimate. you can opt-in to storing small (data) blocks up to a threshold (configurable per dataset) on the special vdev as well, so if you for some reason have lots more capacity than you need for metadata that is something that can be explored as well