VZDump slow on ceph images, RBD export fast

Lephisto

New Member
Jun 22, 2019
27
3
3
42
@spirit @ozdjh the guest cpu does not only spike, i get soft lockups etc, stuff you don't want to have.

Proxmox internal backup solution is currently scuffed and unusable for production, that's why i have to handcraft a rbd solution.
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,443
295
88
Hey @spirit, my comment related to the "bringing his vm guests to their knees" which is the problem we are seeing with backups to nfs shares. CPU in the vms goes through the roof and they drop off the network when they are being backed up.
The build-in backup works inside qemu. Each new block needs to be written twice. Once to the backup storage and once to the VM storage. This means that the slowest storage will determine the storage speed of the VM. In turn, the CPU load rises because of outstanding IO.
 

ozdjh

Member
Oct 8, 2019
54
13
8
@Alwin There's a problem writing backups to NFS targets that causes guests to become unresponsive. The problem does not occur writing to the same remote server using CIFS rather than NFS. The problem is being tracked in bugzilla at

Bug 2554 - Guest slow down while backup to NFS

I'm also looking at writing an rbd solution so backups can't impact on the hypervisors. Our current cloud platform uses a dedicated node within the storage cluster for backups, which offers good separation of backups and the production VMs. We'll look at implementing something similar and will naturally contribute it back if it looks good.
 

spirit

Famous Member
Apr 2, 2010
3,820
195
83
www.odiso.com
With all the trouble we've been seeing backing up to NFS and CIFS volumes during our proxmox evaluation I started looking at rbd exports today. Does your code only work if it's writing to another ceph volume? I'd like to export to local storage.
yes, only to remote rbd storage. (this is because we need to manage snapshot diff restore fast, and also we need to be able to restore files inside the backup rbd fast too)

to backup to a file, look at
https://github.com/Corsinvest/cv4pve-barc
 
  • Like
Reactions: ozdjh

ozdjh

Member
Oct 8, 2019
54
13
8
Thanks for the pointer @spirit, I'll check it out. We'll be starting on this next week so any info we can find is useful.
 

Lephisto

New Member
Jun 22, 2019
27
3
3
42

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!