[SOLVED] PVE backups to PBS running SUPER SLOW!

[chrsgrhmgrhm]

Hello all!

I have an old Dell r610 that is running PVE 8.4, and a new (to me) Dell r740 running PVE 9.
I backed everything up on the r610 to my PBS, then restored to the r740. So far, everything is going smoothly, yay!

That is, until I started to run backups. My backup emails typically indicate it takes approx 17 minutes to back up my 15 LXCs on the r610 to my PBS over a 1G connection.

With the r740, I am seeing backup times taking 10 minutes for a 3GB lxc, and 45 minutes for 50GB lxcs. It's like my read IO is just trash.

The r610 has 2.5" spinning rust (300gb x6) using the Dell PERC card for HW RAID, and the r740 has consumer grade SSDs (2TB x16) using ZRAID10 and a Dell H330 in IT mode.

My hopes were I could get better speeds with the SSDs in ZFS, but that is not proving true for my situation. When I purchased this server and saw it was coming with SSDs, I did not expect to get consumer grade drives.

I cannot tell if this is an issue with a large zpool, with ZFS, crappy drives, or something else. At this point, my thoughts are to back everything up, replace the H330 for the PERC card that came with the r740, then use HW RAID instead of ZFS. Or, destroy my zpool and do smaller pools.

If you have any questions, please let me know! Thanks!

 

[Eduardo Taboada]

First you need to check the performance of your storage, yo can test it with fio command

https://pve.proxmox.com/wiki/Benchmarking_Storage

Also you can try
pveperf

For example

pveperf /var/lib/vz

 

[chrsgrhmgrhm]

I believe I have solved this, although I cannot exactly pinpoint the issue.

I backed everything up to PBS, then changed my pool from 14x2TB in RAID10 to 4x2TB (LXCPOOL) and 10x2TB (VMPOOL) in RAID 10.
I had an an underperforming SSD, so I replaced that as well.

Now my backups are taking 15 minutes instead of almost an hour. I updated PBS 3.4 to 4, and maintained the quick backups.

My theory is the initial zpool was too large to read effectively, but there's also the chance a bad SSD was bringing the whole thing down. Either way, I have a cleaner setup and it's working smoothly!