Ceph block device snapshots

[mgaudette112]

Hi,

I have built myself a little lab of five (5) noisy old HP DL360G7 clustered, and have been learning Ceph with it.

I actually I have a well-running 2 node cluster with a TrueNAS NFS storage. While I do use a Proxmox backup server to backup this cluster's VM, I also am running scheduled snapshots on TrueNAS's ZFS filesystem, as a secondary type of backup (and as a primary backup for a non-Proxmox filesystem on the TrueNAS).

I would like to do the same with Ceph/RDB on the homelab Proxmox cluster. But I can't find how to snapshot all images, or the entire Ceph pool at once. It feels almost certain such a command exists, but I can't find what that command might be.

All I found is

ceph osd pool mksnap <pool-name> <snap-name>

But as soon as a VM image is located in the pool, it is documented to be impossible to snapshot the pool itself. And vice versa

Warning
If you create a pool snapshot, you will never be able to take RBD image snapshots within the pool and it will be irreversible.

Can somebody point me in the right direction here?

 

[alexskysilk]

Can somebody point me in the right direction here?

https://docs.ceph.com/en/latest/rbd/rbd-snapshot/#:~:text=Snapshot Basics

 

[mgaudette112]

https://docs.ceph.com/en/latest/rbd/rbd-snapshot/#:~:text=Snapshot Basics

Thanks, I had seen that link but it explains how to take snapshots of images. While I could make a snapshot of each and every images (images correspond to a Proxmox VM disk as far as I can tell) I wanted a snapshot of all images.

A command that does an individual snapshot of all images works too...sort of like "rbd snap create testpool/all" would work too.

I am very surprised this isn't a native command. Do I need to script something myself?

 

[VictorSTS]

Making a snapshot of a RBD image without notifying the VM about it is of limited usefulness: if you use that snapshot would be exactly as if you had unplugged the power cord of the VM and plugged it back. All pending writes in buffers, in flight write ops in the applications, etc will not be in the snapshot. May even cause filesystem corruption.

IMHO you should install QEMU Agent and take a VM snapshot instead. The VM will be notified about the snapshot and QEMU Agent will freeze/thaw the disks, making pending writes to be flushed to disk before the snapshot.

 

[alexskysilk]

I am very surprised this isn't a native command.

If you can think of a reason this would be desirable, you can submit that to the ceph development team.

Do I need to script something myself?

If there is something you want to do that isnt provided in the existing toolset- by definition.

Making a snapshot of a RBD image without notifying the VM about it is of limited usefulness: if you use that snapshot would be exactly as if you had unplugged the power cord of the VM and plugged it back. All pending writes in buffers, in flight write ops in the applications, etc will not be in the snapshot. May even cause filesystem corruption.

This is the reason what you suggest is not desireable.

 

[mgaudette112]

Making a snapshot of a RBD image without notifying the VM about it is of limited usefulness: if you use that snapshot would be exactly as if you had unplugged the power cord of the VM and plugged it back. All pending writes in buffers, in flight write ops in the applications, etc will not be in the snapshot. May even cause filesystem corruption.

IMHO you should install QEMU Agent and take a VM snapshot instead. The VM will be notified about the snapshot and QEMU Agent will freeze/thaw the disks, making pending writes to be flushed to disk before the snapshot.

I get that, and I am using Proxmox Backup server for a more coherent set of backups. But limited usefulness is not the same as uselessness. It may be a bit overkill, but having a totally separate second mecanism for backups reassures me. A backup of a server from which the virtual power plug has been pulled is certainly better than no backup - if for example the PBS algorithm ends on buggy in some fringe case I happen to be a victim of.
It is admittedly more paranoia than lack of confidence in PBS. But why not have a belt and suspender approach if I have the storage space for it.

I understand this is not possible, and I accept that. I'm just surprised by that fact, I thought I just missed an obvious command.

Do I need to script something myself?

If there is something you want to do that isnt provided in the existing toolset- by definition.

No need to be sarcastic - my question was whether it was provided in the existing toolset and I had missed it.

 

[VictorSTS]

I get that, and I am using Proxmox Backup server for a more coherent set of backups. But limited usefulness is not the same as uselessness. It may be a bit overkill, but having a totally separate second mecanism for backups reassures me. A backup of a server from which the virtual power plug has been pulled is certainly better than no backup - if for example the PBS algorithm ends on buggy in some fringe case I happen to be a victim of.
It is admittedly more paranoia than lack of confidence in PBS. But why not have a belt and suspender approach if I have the storage space for it.

I understand this is not possible, and I accept that. I'm just surprised by that fact, I thought I just missed an obvious command.

Snapshots are not backups because by definition they require the base storage to be usable, but I get your point. Still, I insist that you should be taking snapshots of the VMs and not of the storage

Take a look at this tool https://github.com/Corsinvest/cv4pve-autosnap

 

[mgaudette112]

Snapshots are not backups because by definition they require the base storage to be usable, but I get your point. Still, I insist that you should be taking snapshots of the VMs and not of the storage

Take a look at this tool https://github.com/Corsinvest/cv4pve-autosnap

Yes the word backup was used loosely in my question, I realize that now. "copy of my data at a past point in time" seemed convoluted.

Not trying to beat a dead horse - but this document here (https://docs.ceph.com/en/quincy/rbd/rbd-mirroring/) does explain how to schedule image snapshots, mirror them to another ceph instance, etc. Individual images need to be enabled for scheduling (no "schedule *" or similar commands) but it seems possible (with of course the caveat that are mentioned above in this thread) to accomplish what I want. I try that, so how that works out. I'm still just a the homelab stage so the learning is the point.