Dirty Bitmap Backups are Wearing on Me

W

whatishappening

Well-Known Member
Jan 2, 2021
35
10
48
41
Is there an alternative way to configure backups that I'm unaware of? I have one very large data set at just over 9TB even with a 10 gig direct connection between PVE and PBS this takes AGES due to disk IO constraints then you have to wait on verification jobs, it takes days. Now throw in the regular kernel update reboots (I'm glad things are being patched) I'm constantly running this backup for multiple days each week. It's starting to make me anxious to reboot knowing it's going to chew on this backup for days each time.

I noticed there was a document talking about dirty bitmap persistance and was curious if there was a way to utilize this with proxmox?

https://qemu-project.gitlab.io/qemu/interop/bitmaps.html#bitmap-persistence

Or if I'm just doing this wrong somehow and need to make a change to my backup method.
 
Impact said:
Related issue: https://bugzilla.proxmox.com/show_bug.cgi?id=3233
Click to expand...
Dang since 2021 eh? Guess this isn't getting implemented any time soon.

What other options do I have. I've read I could move the large data set out of a VM and into an LXC container and share it via NFS share so that I can use pxar for change tracking which (if I understand correctly) would persist through reboots. Does this check?
 
That could be an option but for NFS the CT needs to be privileged and you're involving the network which makes things more complicated/slow.
 
Impact said:
That could be an option but for NFS the CT needs to be privileged and you're involving the network which makes things more complicated/slow.
Click to expand...
Ah good point, could switch to SMB to stay unprivileged. Data would stay all on host on a virtual bridge so wouldn't think the networking would be a huge deal. Hmm might have to test that.
 
The PBS live restore functionality is nice to have, so it might be worth attempting to increase your backup performance to improve the dirty bitmap situation.

But if that is not possible, or if you want multiple backups, then consider an agent-based backup solution. You could still use PBS to back up the VM's OS virtual disk, but exclude the data virtual disk. For Windows VMs, you have the built-in Windows backup, and for Linux have a look at BorgBackup.

Or find a Synology NAS and use Active Backup for Business, which has Windows and Linux agents.
 
So I think what I'd want for you is to improve your backup / restore performance. If it takes days to back up your data, the restore is going to be painful if/when that is required.

So a lot of people think "to make it faster, I need a faster NIC". And that's not entirely wrong, but it's wrong. In your case, the NIC is not what's holding things back.

The other main things that hold people back are storage performance and processor performance. You can run a test on your processor. The command is proxmox-backup-client benchmark --repository datastore. You would replace "datastore" with your actual datastore name. If you are running this test on your PBS, then it's just the name of your datastore. But if you are running this on your PVE then it is something like backupuser@pbs@pbs.local:datastore. It's going to test how efficient your processor is with TLS, compression, decompression, encryption, etc. Some processors just can't keep up.

But the big thing that is usually the culprit is storage performance. So ... SSD runs circles around HD. And NVME runs circles around SSD. But also, how you set up your storage is important. If you are using ZFS, then whether you are using RAID 10 vs. RAID-Z2 will make a huge difference. The second needs to recompute parity and write all those drives. RAID 10 just dumps to disk. The difference in performance is significant. And if we are talking about hard drives, oh, especially so. Some don't like to use twice the storage capacity. I get it. But that's going to come with a performance penalty.

Do the proxmox-backup-client benchmark. And then tell us about your storage situation.
 
For purposes of discussion, even if you increased your performance by a factor of ten, there will still be issues. 9TB of data can be smoothed over by a huge performance boost, but what if you had 100TB? 1PB? And there are going to be situations where the bitmap MUST be flagged as dirty, so then what?

100% the PVE dirty-bitmap situation is a major PVE shortcoming, but it probably hasn't reached critical mass yet for the devs to focus on it, or they are waiting for an upstream solution to come along. My noob opinion is that we already have snapshots and ZFS replication, so why can't PBS optionally leverage some of that instead of the bitmap if the user desires?

An agent-based backup inside the VM that uses a filter driver to track changed files is probably still the best option. Even with a fully functional bitmap solution that survives all kinds of scenarios, you will still have the inefficiency of having to operate at the virtual disk level, rather than file level. So you'll pick up a lot of changed blocks with very little actual changed data inside, probably even worse if the VM uses a copy-on-write file system rather than overwrite.

Rsync is also worth checking out, but it's still not as efficient as an agent-based solution.
 
You must log in or register to reply here.
Top Bottom
Back