Incremental backups & Ceph?
- Thread starter SanderM
- Start date
Hi,
I guess tom mean the tradition way for backups: use an backup-client inside the VM guest to backup the data.
I use bareos/bacula for that.
Udo
I guess tom mean the tradition way for backups: use an backup-client inside the VM guest to backup the data.
I use bareos/bacula for that.
Udo
for guest files ,i'm using baresos.
for disaster recovery, i'm backuping ceph rbd volume with rbd snapshots + export
http://ceph.com/dev-notes/incremental-snapshots-with-rbd/
for disaster recovery, i'm backuping ceph rbd volume with rbd snapshots + export
http://ceph.com/dev-notes/incremental-snapshots-with-rbd/
Okay. So for rbd snapshots+exports. Is there any script available for that already or will I have to write something myself?
And before I can use this I need to look into more detail why making a snapshot of a VM running CloudLinux always freezes the VM completely.
I tried that before and as soon as I create a snapshot of VM's running CloudLinux, they freeze and the snashot creation process seems to never end... Other OS's are not affected.
And before I can use this I need to look into more detail why making a snapshot of a VM running CloudLinux always freezes the VM completely.
I tried that before and as soon as I create a snapshot of VM's running CloudLinux, they freeze and the snashot creation process seems to never end... Other OS's are not affected.
Are you creating a VM snapshot of CloudLinux or a ceph snapshot of the disk of the CloudLinux VM?
I have written a script that will perform a nightly export of ceph images. On Saturday night, it takes a full export of the disk image from ceph. Every other day of the week it takes a differential from that full export. I accomplish this using snapshots. It exports to a NFS share.
I also wrote a restore script to help my operators restore the proper files back to ceph if needed. If they try to restore a differential, the script informs the user that it will first restore the full image and then the differential.
If you are interested I can post the script. It is just a BASH script.
Here you go. Please go easy on me, I am not great at BASH scripting 
But all feedback is welcome.
I plan to modify the restore script so it will act like vzdump with vma files. Right now it just restores the image to the ceph cluster. Soon the script will restore the vm.conf and all disks that were backed up with the ceph backup script. I also want to figure out how to have the script register in the gui under the task pane to show its backing up and give a status.
https://github.com/valeech/proxmox_ceph_backups
But all feedback is welcome. I plan to modify the restore script so it will act like vzdump with vma files. Right now it just restores the image to the ceph cluster. Soon the script will restore the vm.conf and all disks that were backed up with the ceph backup script. I also want to figure out how to have the script register in the gui under the task pane to show its backing up and give a status.
https://github.com/valeech/proxmox_ceph_backups
About the CloudLinux freezing: I just use the "Take snapshot" button from proxmox gui. It works for every single VM except those running CloudLinux inside. The dialog that opens when it starts to create a snapshot never finishes and the VM freezes completely. No idea why that is.
Maybe it helps if I install the qemu agent inside the cloudlinux VM?
About your script: awesome! Please post back if you update your script with the additional features you already suggested. I'm very very interested in that!!! THANKS
Maybe it helps if I install the qemu agent inside the cloudlinux VM?
About your script: awesome! Please post back if you update your script with the additional features you already suggested. I'm very very interested in that!!! THANKS
Thanks! We'll have a look into that.Here you go. Please go easy on me, I am not great at BASH scripting
I plan to modify the restore script so it will act like vzdump with vma files. Right now it just restores the image to the ceph cluster. Soon the script will restore the vm.conf and all disks that were backed up with the ceph backup script. I also want to figure out how to have the script register in the gui under the task pane to show its backing up and give a status.
https://github.com/valeech/proxmox_ceph_backups
The main problem here is that you need a working ceph cluster to restore
Thats the biggest benefit of PVEs VMA backups, they can be restored an any other PVE machine even with local storage (if the ceph cluster is completely destroyed i.e. what hopefully never happens
So your script is not interacting with the guest OS (through qemu-agent for example) and the backups are maybe (more) inconsistent compared to Proxmox's VM backup method?
This has nothing to do with the OS, the backup works on the RBD layer (ceph). Its more like a snapshot of the virtual disk.
Yes, but proxmox backup system is sending commands to qemu-agent (running inside the guest) in order to freeze the filesystem momentarily or something, in orde to make the snapshot more consistent before taking a backup. I don't know if this is worth it but this is something your script won't do, right?
I'm not sure if there's any way we could improve backup consistency by telling guest VM's to flush it data, somehow, before your scripts creates an rbd snapshot...
I'm not sure if there's any way we could improve backup consistency by telling guest VM's to flush it data, somehow, before your scripts creates an rbd snapshot...
Thanks! We'll have a look into that.
The main problem here is that you need a working ceph cluster to restore
Thats the biggest benefit of PVEs VMA backups, they can be restored an any other PVE machine even with local storage (if the ceph cluster is completely destroyed i.e. what hopefully never happens
Actually you don't need a working ceph cluster to restore. rbd export creates raw image files. Those images can them be mapped directly to VMs or converted using qemu to another format. I know this works with a full export. For differential exports, it looks like I will have to write my own tool to replay them onto the full image.
The mechanism I use in my script to get the differential exports to work is this:
create ceph snapshot of disk on Saturdayn (rbd snap create)
backup full snapshop (rbd export pool/disk@snap)
create ceph differential export Sun-Fri using the snapshot from Saturday as the source (rbd export-diff --from-snap)
If you changed the backup strategy to always full backup rather than differential, you would be able to mount all you backups directly.
You are right, my script does not do this currently. It should be pretty simple to quiesce the filesystems and then do the ceph snapshot just to make sure everything is good.
Thanks for clarifying! Raw sounds good, but the backups are fullsized filesystems then, right? What about compression?
Good question about compression. I installed pigz compressor on one of my nodes in my cluster. Its a parallel compressor based on gzip so it is really fast.
In order to add add compression, when exporting with the rbd command, instead of specifying an output file to the rbd command, specify a - (stdout) and then pipe that to the pigz command and specify the output filename there.
Example:
rbd export --rbd-concurrent-management-ops 20 pool/image - | pigz > /nfsshare/rbdexport.img.gz
pigz has a whole host of options you can give it to speed up the compression. I used the defaults and the compression added no significant time to the process. I will add this as a command-line option to the backup script.
I was able to export a 32G image down to a 5G file to a standalone proxmox host. I then unzipped it and mounted it directly to a new VM. The VM started right up, proving you can restore without a ceph cluster.
In order to add add compression, when exporting with the rbd command, instead of specifying an output file to the rbd command, specify a - (stdout) and then pipe that to the pigz command and specify the output filename there.
Example:
rbd export --rbd-concurrent-management-ops 20 pool/image - | pigz > /nfsshare/rbdexport.img.gz
pigz has a whole host of options you can give it to speed up the compression. I used the defaults and the compression added no significant time to the process. I will add this as a command-line option to the backup script.
I was able to export a 32G image down to a 5G file to a standalone proxmox host. I then unzipped it and mounted it directly to a new VM. The VM started right up, proving you can restore without a ceph cluster.
Sounds great, but pigz can burn all CPU cores, which can cause problems in in VM enviroment ;-)
You could test lzop, which PVE also uses for VMA format. Its very fast with medium compression, which is enough in most cases.
Mounting it as RAW ist very nice, indeed, and makes it really interesting ;-)
You could test lzop, which PVE also uses for VMA format. Its very fast with medium compression, which is enough in most cases.
Mounting it as RAW ist very nice, indeed, and makes it really interesting ;-)