Yesterday I updated all nodes to the latest 4.4 and here is my feedback:
Through the night, backup was running and the result is slightly better than before.
For example:
One node contain two VMs with together about 600GB.
Time to backup:
The previous 4.4 version managed to backup both VMs in 7 hours 31 minutes.
The latest 4.4 managed it in 5 hours 28 minutes.
Transfer speed:
Previous 4.4: 1st VM 38MB/s; 2nd VM 36MB/s
latest 4.4: 1st VM 51MB/s; 2nd VM 52MB/s
Well its going in the right direction, but its still far away from what would be possible.
I could write to the storage with about 400-500MB/s...
Code:
Kernel Version: Linux 4.4.83-1-pve #1 SMP PVE 4.4.83-96 (Tue, 19 Sep 2017 10:30:12 +0200)
PVE Manager Version: pve-manager/4.4-18/ef2610e8
Ceph Version: 10.2.10Through the night, backup was running and the result is slightly better than before.
For example:
One node contain two VMs with together about 600GB.
Time to backup:
The previous 4.4 version managed to backup both VMs in 7 hours 31 minutes.
The latest 4.4 managed it in 5 hours 28 minutes.
Transfer speed:
Previous 4.4: 1st VM 38MB/s; 2nd VM 36MB/s
latest 4.4: 1st VM 51MB/s; 2nd VM 52MB/s
Well its going in the right direction, but its still far away from what would be possible.
I could write to the storage with about 400-500MB/s...
@fips
thanks for the feedback.
Your result looks better but still too slow what rbd export can do.
For example for my environment even 50MB/s is still not affordable approach, as we have several TBs and cluster is still growing.
Could you post the exact version of the latest PVE host where you got 50MB/s?
Thanks.
thanks for the feedback.
Your result looks better but still too slow what rbd export can do.
For example for my environment even 50MB/s is still not affordable approach, as we have several TBs and cluster is still growing.
Could you post the exact version of the latest PVE host where you got 50MB/s?
Thanks.
Code:
root@ceph:~# pveversion -v
proxmox-ve: 4.4-96 (running kernel: 4.4.83-1-pve)
pve-manager: 4.4-18 (running version: 4.4-18/ef2610e8)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.83-1-pve: 4.4.83-96
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-53
qemu-server: 4.0-113
pve-firmware: 1.1-11
libpve-common-perl: 4.0-96
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.9.0-5~pve4
pve-container: 1.0-101
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80
ceph: 10.2.10-1~bpo80+1We just upgrade (fresh install) to the latest version our Proxmox VE 5.034 cluster. Everything looks great, the new Proxmox and Ceph Luminous works well except for one thing. The well known slow vzdump legacy.
We run a 8 nodes Ceph cluster as VM storage and 5 nodes Proxmox compute nodes. The backup is made on a NFS server. We also have jumbo frames and the new Google BBR enabled. No bottleneck hardware involved.
Sadly the backup speed does not jump over 35MB/s.
We run a 8 nodes Ceph cluster as VM storage and 5 nodes Proxmox compute nodes. The backup is made on a NFS server. We also have jumbo frames and the new Google BBR enabled. No bottleneck hardware involved.
Sadly the backup speed does not jump over 35MB/s.
Last edited:
Is there any new feedback from users who upgraded Proxmox and Ceph and could increase backup speed?
I can tell for myself: Last weekend I upgraded to 5.1 and Ceph 12.2.4 and after 2 rounds of backups I can say backup speed didn't change, at least for me.
I can tell for myself: Last weekend I upgraded to 5.1 and Ceph 12.2.4 and after 2 rounds of backups I can say backup speed didn't change, at least for me.
I can say for us it's identical: As long as we've been using Proxmox with Ceph the backup speeds are between 30 - 50MB per second ... no change
We keep up with all updates
I am subscribed to the pve-devel list and haven't seen any work being done on this so it makes sense that the backup speeds haven't changed.
We keep up with all updates
I am subscribed to the pve-devel list and haven't seen any work being done on this so it makes sense that the backup speeds haven't changed.
Hi,
you can also use this external script, to backup with rbd snapshot, rbd-diff export feature.
https://github.com/EnterpriseVE/eve4pve-barc
Works very fine and a lot faster.
you can also use this external script, to backup with rbd snapshot, rbd-diff export feature.
https://github.com/EnterpriseVE/eve4pve-barc
Works very fine and a lot faster.
That's kind of you to be helpful and offer another solution, thank you...the problem is that backups are an integral part of Proxmox since its inception and frankly the problems with the speed need to be addressed within Proxmox itself instead of throwing the problems off to a third party script or whatever. They've made Ceph an officially supported backend for Proxmox, part of that means that backups need to be fixed to work as expected at the speeds expected with that supported backend as well.
I watch the pve-devel list. A lot of talk of a whole lot of features etc but no talk of fixing this basic functionality that is broken that I've seen. I know someone's pride is going to be hurt with this statement and they'll be in-sensed over it and chastise me for daring to say anything as if what I'm saying is offensive. Pride needs to be put aside and attention needs to be paid to basic functionality that's been in Proxmox since the beginning and was part of what made it an attractive product over other offers.
Thanks again for your kindness in offering an alternative solution though
I watch the pve-devel list. A lot of talk of a whole lot of features etc but no talk of fixing this basic functionality that is broken that I've seen. I know someone's pride is going to be hurt with this statement and they'll be in-sensed over it and chastise me for daring to say anything as if what I'm saying is offensive. Pride needs to be put aside and attention needs to be paid to basic functionality that's been in Proxmox since the beginning and was part of what made it an attractive product over other offers.
Thanks again for your kindness in offering an alternative solution though
Hello everybody,
the problem is that Proxmox VE is not structured to develop external tools to be integrated in the GUI and at the core level. Development is closed not to plugins. This does not improve the extension of the system.
Thank you all.
the problem is that Proxmox VE is not structured to develop external tools to be integrated in the GUI and at the core level. Development is closed not to plugins. This does not improve the extension of the system.
Thank you all.
Frank, thank you for your script. It is working well for me to allow for fast daily backups, I'm looking to make Ceph my primary storage because of this (and that Ceph can do snapshots) instead of iSCSI.
Do you foresee the ability to do continuous/synthetic fulls (never have to run a 'full' backup again, ideal for saving bandwidth for offsite backup repos)? Is that possible with the way Ceph does it's snapshots?
Do you foresee the ability to do continuous/synthetic fulls (never have to run a 'full' backup again, ideal for saving bandwidth for offsite backup repos)? Is that possible with the way Ceph does it's snapshots?
vzdump speed from Ceph is now up to twice as fast as it was before the block size fixes. seems you missed this part in following in the pve-devel lists? you can expect more than 100 mb/s.
in one of our production clusters we had initially:
Code:
210: Aug 05 02:56:11 INFO: transferred 274877 MB in 4250 seconds (64 MB/s)
210: Aug 05 02:56:12 INFO: archive file size: 61.99GB
210: Aug 05 02:56:17 INFO: Finished Backup of VM 210 (01:11:15)after the block size fixes, we see now:
Code:
210: 2018-03-15 02:29:15 INFO: transferred 274877 MB in 2634 seconds (104 MB/s)
210: 2018-03-15 02:29:16 INFO: archive file size: 76.77GB
210: 2018-03-15 02:29:20 INFO: Finished Backup of VM 210 (00:44:18)any idea why the image file is not created with eve4pve-barc, config file are created but no image file @frank lupo
The config files that are missing images, are they stored on Ceph?
I use --vmid=all key. Initially, when started using barc, some of the VMs were not stored on Ceph. So for those, it only backed up the config files but not the image because it could not create snapshot.
eve4pve-barc is a backup app strictly for Ceph RBD. I don't believe it will do a backup when VM is stored on CephFS or any other storage. barc uses rbd export.