VZDump slow on ceph images, RBD export fast

[Lephisto]

@spirit @ozdjh the guest cpu does not only spike, i get soft lockups etc, stuff you don't want to have.

Proxmox internal backup solution is currently scuffed and unusable for production, that's why i have to handcraft a rbd solution.

 

[Alwin]

Hey @spirit, my comment related to the "bringing his vm guests to their knees" which is the problem we are seeing with backups to nfs shares. CPU in the vms goes through the roof and they drop off the network when they are being backed up.

The build-in backup works inside qemu. Each new block needs to be written twice. Once to the backup storage and once to the VM storage. This means that the slowest storage will determine the storage speed of the VM. In turn, the CPU load rises because of outstanding IO.

 

[ozdjh]

@Alwin There's a problem writing backups to NFS targets that causes guests to become unresponsive. The problem does not occur writing to the same remote server using CIFS rather than NFS. The problem is being tracked in bugzilla at

Bug 2554 - Guest slow down while backup to NFS

I'm also looking at writing an rbd solution so backups can't impact on the hypervisors. Our current cloud platform uses a dedicated node within the storage cluster for backups, which offers good separation of backups and the production VMs. We'll look at implementing something similar and will naturally contribute it back if it looks good.

 

[spirit]

With all the trouble we've been seeing backing up to NFS and CIFS volumes during our proxmox evaluation I started looking at rbd exports today. Does your code only work if it's writing to another ceph volume? I'd like to export to local storage.

yes, only to remote rbd storage. (this is because we need to manage snapshot diff restore fast, and also we need to be able to restore files inside the backup rbd fast too)

to backup to a file, look at
https://github.com/Corsinvest/cv4pve-barc

 

[ozdjh]

Thanks for the pointer @spirit, I'll check it out. We'll be starting on this next week so any info we can find is useful.

 

[EPM]

Hi,
I also have problems with vzdump speed under ceph to save to nfs:
https://forum.proxmox.com/threads/vzdump-speed-improvment.63500/
Pigz parameter "-b 8192 -3".
It worked for me .
Worth a try.
L,

 

[Lephisto]

yes, only to remote rbd storage. (this is because we need to manage snapshot diff restore fast, and also we need to be able to restore files inside the backup rbd fast too)

to backup to a file, look at
https://github.com/Corsinvest/cv4pve-barc

Yeah, I extended it a bit, to make it cluster-aware and issue fsfreeze-freeze before and fsfreeze-thaw after snapshotting:

https://github.com/lephisto/cv4pve-barc/