VZDump slow on ceph images, RBD export fast

iva-a-an

New Member
Apr 11, 2017
11
1
3
33
Thanks for update Tom!

Any recommendations for people who's not ready for upgrade yet and still using 4.4?
Thanks
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
13,895
460
103
Thanks for update Tom!

Any recommendations for people who's not ready for upgrade yet and still using 4.4?
Thanks
The fix is also in latest 4.4.
 
  • Like
Reactions: iva-a-an

fips

Member
May 5, 2014
148
5
18
Yesterday I updated all nodes to the latest 4.4 and here is my feedback:

Code:
Kernel Version: Linux 4.4.83-1-pve #1 SMP PVE 4.4.83-96 (Tue, 19 Sep 2017 10:30:12 +0200)
PVE Manager Version: pve-manager/4.4-18/ef2610e8
Ceph Version: 10.2.10
Through the night, backup was running and the result is slightly better than before.
For example:
One node contain two VMs with together about 600GB.

Time to backup:
The previous 4.4 version managed to backup both VMs in 7 hours 31 minutes.
The latest 4.4 managed it in 5 hours 28 minutes.

Transfer speed:
Previous 4.4: 1st VM 38MB/s; 2nd VM 36MB/s
latest 4.4: 1st VM 51MB/s; 2nd VM 52MB/s

Well its going in the right direction, but its still far away from what would be possible.
I could write to the storage with about 400-500MB/s...
 
  • Like
Reactions: iva-a-an

iva-a-an

New Member
Apr 11, 2017
11
1
3
33
@fips
thanks for the feedback.

Your result looks better but still too slow what rbd export can do. :(
For example for my environment even 50MB/s is still not affordable approach, as we have several TBs and cluster is still growing.

Could you post the exact version of the latest PVE host where you got 50MB/s?

Thanks.
 

fips

Member
May 5, 2014
148
5
18
Could you post the exact version of the latest PVE host where you got 50MB/s?
Code:
root@ceph:~# pveversion -v
proxmox-ve: 4.4-96 (running kernel: 4.4.83-1-pve)
pve-manager: 4.4-18 (running version: 4.4-18/ef2610e8)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.83-1-pve: 4.4.83-96
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-53
qemu-server: 4.0-113
pve-firmware: 1.1-11
libpve-common-perl: 4.0-96
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.9.0-5~pve4
pve-container: 1.0-101
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80
ceph: 10.2.10-1~bpo80+1
 
  • Like
Reactions: iva-a-an

Dan Nicolae

Member
Apr 27, 2016
71
2
8
39
We just upgrade (fresh install) to the latest version our Proxmox VE 5.034 cluster. Everything looks great, the new Proxmox and Ceph Luminous works well except for one thing. The well known slow vzdump legacy.

We run a 8 nodes Ceph cluster as VM storage and 5 nodes Proxmox compute nodes. The backup is made on a NFS server. We also have jumbo frames and the new Google BBR enabled. No bottleneck hardware involved.

Sadly the backup speed does not jump over 35MB/s.
 
Last edited:
  • Like
Reactions: iva-a-an

fips

Member
May 5, 2014
148
5
18
Is there any new feedback from users who upgraded Proxmox and Ceph and could increase backup speed?

I can tell for myself: Last weekend I upgraded to 5.1 and Ceph 12.2.4 and after 2 rounds of backups I can say backup speed didn't change, at least for me.
 
Apr 6, 2012
81
5
8
I can say for us it's identical: As long as we've been using Proxmox with Ceph the backup speeds are between 30 - 50MB per second ... no change

We keep up with all updates

I am subscribed to the pve-devel list and haven't seen any work being done on this so it makes sense that the backup speeds haven't changed.
 
Apr 6, 2012
81
5
8
That's kind of you to be helpful and offer another solution, thank you...the problem is that backups are an integral part of Proxmox since its inception and frankly the problems with the speed need to be addressed within Proxmox itself instead of throwing the problems off to a third party script or whatever. They've made Ceph an officially supported backend for Proxmox, part of that means that backups need to be fixed to work as expected at the speeds expected with that supported backend as well.

I watch the pve-devel list. A lot of talk of a whole lot of features etc but no talk of fixing this basic functionality that is broken that I've seen. I know someone's pride is going to be hurt with this statement and they'll be in-sensed over it and chastise me for daring to say anything as if what I'm saying is offensive. Pride needs to be put aside and attention needs to be paid to basic functionality that's been in Proxmox since the beginning and was part of what made it an attractive product over other offers.

Thanks again for your kindness in offering an alternative solution though
 

frank lupo

Member
Dec 27, 2016
16
4
8
49
www.enterpriseve.com
Hello everybody,
the problem is that Proxmox VE is not structured to develop external tools to be integrated in the GUI and at the core level. Development is closed not to plugins. This does not improve the extension of the system.

Thank you all.
 

markmarkmia

New Member
Feb 5, 2018
23
0
1
44
Frank, thank you for your script. It is working well for me to allow for fast daily backups, I'm looking to make Ceph my primary storage because of this (and that Ceph can do snapshots) instead of iSCSI.

Do you foresee the ability to do continuous/synthetic fulls (never have to run a 'full' backup again, ideal for saving bandwidth for offsite backup repos)? Is that possible with the way Ceph does it's snapshots?
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
13,895
460
103
That's kind of you to be helpful and offer another solution, thank you...the problem is that backups are an integral part of Proxmox since its inception and frankly the problems with the speed need to be addressed within Proxmox itself instead of throwing the problems off to a third party script or whatever. They've made Ceph an officially supported backend for Proxmox, part of that means that backups need to be fixed to work as expected at the speeds expected with that supported backend as well.

I watch the pve-devel list. A lot of talk of a whole lot of features etc but no talk of fixing this basic functionality that is broken that I've seen. I know someone's pride is going to be hurt with this statement and they'll be in-sensed over it and chastise me for daring to say anything as if what I'm saying is offensive. Pride needs to be put aside and attention needs to be paid to basic functionality that's been in Proxmox since the beginning and was part of what made it an attractive product over other offers.

Thanks again for your kindness in offering an alternative solution though
vzdump speed from Ceph is now up to twice as fast as it was before the block size fixes. seems you missed this part in following in the pve-devel lists? you can expect more than 100 mb/s.

in one of our production clusters we had initially:

Code:
210: Aug 05 02:56:11 INFO: transferred 274877 MB in 4250 seconds (64 MB/s)
210: Aug 05 02:56:12 INFO: archive file size: 61.99GB
210: Aug 05 02:56:17 INFO: Finished Backup of VM 210 (01:11:15)
after the block size fixes, we see now:
Code:
210: 2018-03-15 02:29:15 INFO: transferred 274877 MB in 2634 seconds (104 MB/s)
210: 2018-03-15 02:29:16 INFO: archive file size: 76.77GB
210: 2018-03-15 02:29:20 INFO: Finished Backup of VM 210 (00:44:18)
 

frank lupo

Member
Dec 27, 2016
16
4
8
49
www.enterpriseve.com
Frank, thank you for your script. It is working well for me to allow for fast daily backups, I'm looking to make Ceph my primary storage because of this (and that Ceph can do snapshots) instead of iSCSI.

Do you foresee the ability to do continuous/synthetic fulls (never have to run a 'full' backup again, ideal for saving bandwidth for offsite backup repos)? Is that possible with the way Ceph does it's snapshots?
Hi,
there are new ideas. We'll update soon.

Best regards
 

symmcom

Renowned Member
Oct 28, 2012
1,078
26
68
Calgary, Canada
www.symmcom.com
any idea why the image file is not created with eve4pve-barc, config file are created but no image file @frank lupo
The config files that are missing images, are they stored on Ceph?

I use --vmid=all key. Initially, when started using barc, some of the VMs were not stored on Ceph. So for those, it only backed up the config files but not the image because it could not create snapshot.
 

symmcom

Renowned Member
Oct 28, 2012
1,078
26
68
Calgary, Canada
www.symmcom.com
eve4pve-barc is a backup app strictly for Ceph RBD. I don't believe it will do a backup when VM is stored on CephFS or any other storage. barc uses rbd export.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!