Are there any plans to build in ceph as datastore backend to Proxmox Backup Server like it is already available on Proxmox VE?
Ceph as Storage Backend
- Thread starter codingspiderfox
- Start date
there are some users backing a datastore with cephfs, but obviously it won't be the most performant solution. many of the same caveats also apply to using an RBD image mapped and mounted on the PBS host. since it's a suboptimal choice, we likely won't implement any integration for such setups. but you can try them out yourself and report back with your experiences 
PBS uses lots of small files (chunks), storing those on cephfs means adding network latency to all the file operations, which in turn means worse performance. it's just an inherent difference between accessing a local filesystem vs accessing a network share (and in ceph's case, because of the redundancy, there's actually even more work behind the scenes than with, for example, an NFS or CIFS share).
Could you please go a bit into detail which actions access all these small files?
Wouldn't it be possible to do all these action locally on the PBS host and when the backup is done - pushing those small files to the ceph storage?
Wouldn't it be possible to do all these action locally on the PBS host and when the backup is done - pushing those small files to the ceph storage?
If you backup a 1 TB virtual disk with PBS, PBS won't store a single 1TB image file like vzdump, it will store it as atleast 250,000 (but probably more like 500,000) chunk files of max 4MB size.
Each time you do a restore, verify or GC of that virtual disk it will need to read/write all those 250,000+ files.
For example for the pruning/GC PBS will set/read the atime metadata of each of those chunk files to decide if it should be deleted or not.
Last edited:
Chunks are hashed and stored in index files. But that will only help when you for example create a new backup snapshot and want to know if that chunk already exists, so it doesn't need to be uploaded a second time. But it won't help with maintaince tasks like GC or verify or if you actually need to read those chunks when doing a restore.
Especially for the GC task you really want the metadata of the whole datastore to be stored on a local SSD so it can be done in minutes instead of hours or even days.
Especially for the GC task you really want the metadata of the whole datastore to be stored on a local SSD so it can be done in minutes instead of hours or even days.
I've just today configured PBS with a CephFS backend, and wanted to share my experience:
PBS is installed in a Debian LXC, on a PVE host, which has CephFS mounted from another Proxmox cluster (the host running PBS is different to the hardware running Ceph.
The hosts are connected via 10GbE. Ceph is running 4TB 7200rpm SAS disks.
As soon as I created the Datastore in PBS, it crippled my Ceph cluster to the point I was getting slow heartbeats on OSD's, and failed PG's across the cluster.
It eventually recovered, but not before slowing EVERYTHING down for ages.
My guess would be the creation of the hundreds-of-thousands of chunks across the storage in one go was too much for Ceph.
Once it recovered I was able to backup VM's and LXC's ok. I've not got benchmarks yet, but performance was acceptable (50GB virtual machine in under 10 minutes).
So can you use CephFS - yes. Should you - maybe. Just beware the huge initial I/O may harm Ceph performance whilst it creates all the chunks.
A better way might be to create the storage on a separate disk (SSD), then rsync the chunks over to CephFS before remounting the Datastore.
PBS is installed in a Debian LXC, on a PVE host, which has CephFS mounted from another Proxmox cluster (the host running PBS is different to the hardware running Ceph.
The hosts are connected via 10GbE. Ceph is running 4TB 7200rpm SAS disks.
As soon as I created the Datastore in PBS, it crippled my Ceph cluster to the point I was getting slow heartbeats on OSD's, and failed PG's across the cluster.
It eventually recovered, but not before slowing EVERYTHING down for ages.
My guess would be the creation of the hundreds-of-thousands of chunks across the storage in one go was too much for Ceph.
Once it recovered I was able to backup VM's and LXC's ok. I've not got benchmarks yet, but performance was acceptable (50GB virtual machine in under 10 minutes).
So can you use CephFS - yes. Should you - maybe. Just beware the huge initial I/O may harm Ceph performance whilst it creates all the chunks.
A better way might be to create the storage on a separate disk (SSD), then rsync the chunks over to CephFS before remounting the Datastore.