[SOLVED] Backup suddenly 6 times slower

[gerd_f]

Hallo,
I have a problem with the Backups of my machines. It usually took around 2:20 hours to backup the machines. Now from August it suddenly takes almost 12 hours.
Does anyone has an idea what could be the root cause?

I have 2 Mirror ZFS Pools:

the first nvme0n1 and sda, this pool contains the VMs and CTs (Internal)
the second one sdb and sdc, this pool contains the backups. (External USB3 Harddisks)

Thanks in advance
Gerd

 

[gerd_f]

It's solved. I have now installed a Proxmox backup server with the same hardware and the backup took only 1.32h. It's solved now but I would anyway be interested on the root cause. Might be it's because the Harddisks became almost full but can this really has such an impact?

 

[IsThisThingOn]

Might be it's because the Harddisks became almost full but can this really has such an impact?

Yes.
Your external HDDs could be SMR.
And ZFS recommends to be at 80% so there hopefully enough of free space (without fragmentation) to work with (but even at 50% you will already start to see a negative performance impact).

BTW: you should never use ZFS over USB!
I would just use a single disk with ext4 for backups. Mirroring them is overkill in my opinion. Especially considering ZFS snapshots. If you care that much about backups, then an offsite PBS would be the better option.

 

[gerd_f]

Thank you for your response. I have now cleaned up the Harddisk, have installed a Prosmox Backup Server that creates incremental backups instead of full backups. That approach works now since the incremental backups are much smaller .

Why is it not recomended to use ZFS over USB. Is it because of perfocmance issues? I have lost all my backups becaus of a failed Harddisk, then I have created ZFS for all the Harddisks tp prevent this in future. What I could do instad of, is to creat 2 ext4 Harddisks and then sync them with rsync but ZFS more elgant.

 

[IsThisThingOn]

Great!

Why is it not recomended to use ZFS over USB

ZFS needs direct access to the disk. No RAID, no USB, no other shenanigans in between.

I have lost all my backups becaus of a failed Harddisk, then I have created ZFS for all the Harddisks tp prevent this in future. What I could do instad of, is to creat 2 ext4 Harddisks and then sync them with rsync but ZFS more elgant.

There are many options. Each has its pros and cons.
And there are different backup methods and use cases.

There is "oops I misconfigured something in the VM". For that Snapshots are great. I do a snapshot my machines even before an update, just to be safe.

Then there is "oops, I deleted a VM by accident". For that Backups are great. If that backup is on a USB HDD with ext4, NFS share or PBS does not really matter.

Then there is "oops, my house burned down". For that an offsite PBS is great. Or a NFS share on TrueNAS that has a NFS share that is synced to S3

I don't really know if I would go for ext4 or zfs for a PBS to be honest. Most of the great features that ZFS offers like checksums are integrated in PBS. So maybe ext4 has the advantage of not having a ZIL? I honestly don't know yet and have to do some performance testing.