Ceph backup

[PiotrD]

Hi,
I am experiencing slow backup speed when I backup ceph images in proxmox. I do not see any problems with network and ceph. I would like to know what type of ceph functionailty is used by proxmox to do backups ? ceph export ? I would like to debug ceph, but I do not know how proxmox does backups in ceph. Now backups are sth around 40MB/s and it should be much faster. I did test backups from ceph to local storage.

UPDATE:
I did small test:
standard proxmox backup took ~4 minutes
using rbd export and lzop was around ~1 min

However, I am not sure if these equals also I do not know if export is correct way to do live backup.

Kind regards,
Piotr D

 

[dietmar]

I am experiencing slow backup speed when I backup ceph images in proxmox. I do not see any problems with network and ceph. I would like to know what type of ceph functionailty is used by proxmox to do backups ? ceph export ? I would like to debug ceph, but I do not know how proxmox does backups in ceph. Now backups are sth around 40MB/s and it should be much faster. I did test backups from ceph to local storage.

We also observe that. Backup are done inside qemu, using the normal block driver to read the data (using a qemu backup job).
Qemu backup jobs read data by 64K block size. Maybe that is limiting performance?

 

[PiotrD]

We also observe that. Backup are done inside qemu, using the normal block driver to read the data (using a qemu backup job).
Qemu backup jobs read data by 64K block size. Maybe that is limiting performance?

I tried time qemu-img convert rbd:rbd/vm-101-disk-1 -O raw dump.raw

real 0m43.892s
user 0m21.199s
sys 0m14.446s

with lzop it should be around ~1.5 min. So it is faster than via GUI. Are you maybe using sth different than this ? I am probably missing something about this

 

[e100]

Qemu backup jobs read data by 64K block size. Maybe that is limiting performance?

Is this configurable? I suspect larger reads would perform much better on some of my servers.

 

[PiotrD]

Is this configurable? I suspect larger reads would perform much better on some of my servers.

And that should be my second question. Is this configurable?

 

[dietmar]

No, that is not configurable, an it is non-trivial to change it.
Besides, 64K reads should be fast, so there must be a bug inside ceph/rados library. Or maybe the check for allocated chunks are slow?

 

[Mitya]

I have 20MB/s for allocated space and 40 for unallocated (reported sparse).
Mayde there's easy way to enable readahead at lower level, while still read 64k in qemu's backup level?

 

[mo_]

maybe itd be worth it to collaborate with the Ceph guys on this. Especially since theres another issue involving backups to be solved (by the Ceph guys): cache pools. More to the point: if you have a cache pool in front of your Ceph storage and you take a backup, you essentially clear out the entire cache (that used to contain most commonly used objects) by replacing it with the backup. As such a specific read method for backups would be desirable, that would a) read in bigger blocksizes (4M preferrably) and b) "bypass" the cache for such a backup process. This last bit would have to make sure the data is consistent:

- initiate backup
- switch cache pool to read-only cache
- write dirty data to cold storage (data thats changed on the cache)
- take backup directly from cold storage
- switch cache pool back to rw cache

Backups in general are sadly really more of an afterthought for Ceph, as backing up Petabytes of data simply isnt desirable and in the end thats the amount of data Ceph has been designed around. Of course there are also much smaller setups (like the ones that would be sufficient to 'normal' datacenter virtualization) where backups ARE feasible. This is an area where Ceph could still improve quite a bit.

 

[PiotrD]

maybe itd be worth it to collaborate with the Ceph guys on this. Especially since theres another issue involving backups to be solved (by the Ceph guys): cache pools. More to the point: if you have a cache pool in front of your Ceph storage and you take a backup, you essentially clear out the entire cache (that used to contain most commonly used objects) by replacing it with the backup. As such a specific read method for backups would be desirable, that would a) read in bigger blocksizes (4M preferrably) and b) "bypass" the cache for such a backup process. This last bit would have to make sure the data is consistent:

- initiate backup
- switch cache pool to read-only cache
- write dirty data to cold storage (data thats changed on the cache)
- take backup directly from cold storage
- switch cache pool back to rw cache

Backups in general are sadly really more of an afterthought for Ceph, as backing up Petabytes of data simply isnt desirable and in the end thats the amount of data Ceph has been designed around. Of course there are also much smaller setups (like the ones that would be sufficient to 'normal' datacenter virtualization) where backups ARE feasible. This is an area where Ceph could still improve quite a bit.

+1 to that. Or maybe someone already reported it to ceph team ?

 

[Mitya]

Maybe just use rbd export/import command to backup/restore rbd images? It should be much faster than qemu's backup.

 

[liska_]

Hi,
is here anything new in this area? My backups run around 30MB/s from ceph storage but more than 100mb/s from nfs based storage to the same location. Rados bench shows read around 300mb/s. It is quite unconvenient.
Isn`t here any settings or anything else I can do to improve performance? I have found nothing regarding this matter on google as well.