[SOLVED] Official way to backup proxmox VE itself?

[Dunuin]

Isn't is possible to just create a snapshot of the whole rpool (pve installation) and use zfs send and receive to to backup to another destination?

Yes thats possible. But usually you want...:
A.) a full copy of your system disk so if you lose it you just need to restore it to a new disk and everything is working like before
B.) only the files you really care about. Basically the config files in "/etc" and your stuff from the "local" storage in "/var/lib/vz"
A is usually quite old because PVE shouldn't be running while backing it up on block level. So you have to boot into something like a clonezilla pen drive. B is more recent as you can do that daily/hourly while PVE is running. I try to combine both. So first a block level restore of the whole system disk and then a restore my config files from B over it.

 

[LnxBil]

Isn't is possible to just create a snapshot of the whole rpool (pve installation) and use zfs send and receive to to backup to another destination?

Sure. That's what I already pointed out in #7. @Dunuin's points are also true, yet A) is not hard to archieve in an UEFI way. Just regularly rsync your UEFI disk to your ZFS too and it'll be included in the off-site backup. You would also need a ZFS-capable rescue system in order restore your machine from your backup.

 

[Dunuin]

Sure. That's what I already pointed out in #7. @Dunuin's points are also true, yet A) is not hard to archieve in an UEFI way. Just regularly rsync your UEFI disk to your ZFS too and it'll be included in the off-site backup. You would also need a ZFS-capable rescue system in order restore your machine from your backup.

Jep, both should work. But there really should be a well tested data recovery plan. Replicating snapshots between hosts and rsyncing the ESP adds a lot of extra complexity compared to a blocklevel restore of a whole disk and you need to be able to handle in a pressure situation.

 

[aanjaneyam]

Sure. That's what I already pointed out in #7. @Dunuin's points are also true, yet A) is not hard to archieve in an UEFI way. Just regularly rsync your UEFI disk to your ZFS too and it'll be included in the off-site backup. You would also need a ZFS-capable rescue system in order restore your machine from your backup.

After I posted I found these two relevant thread which look promising.
Using ZFS Snapshots on rpool/ROOT/pve-1?

rescue mode with root on zfs trouble

However, these posts say that only rpool/ROOT/pve-1 needs to be snapshoted for full restore and the snapshot of rpool/ROOT/pve-1 can be taken while pve is running and the use send/receive back it up to a different pool. Sorry for novice question what is rpool/DATA dataset doing in the pve installation or what is its purpose in an boot OS cd which will be just used for OS purposes. Does rpool/DATA needs to be backed up as well?

 

[aanjaneyam]

Jep, both should work. But there really should be a well tested data recovery plan. Replicating snapshots between hosts and rsyncing the ESP adds a lot of extra complexity compared to a blocklevel restore of a whole disk and you need to be able to handle in a pressure situation.

For backup of snapshots I will be using a separate zfs pool on the same server and also back it up to a zfs pool on an external USB drive. I will then also upload the snapshots to a cloud service like s3 or even rsync.net which is now providing zfs suport rsync.net supports ZFS send and receive over SSH . I will not be directly backing up or restoring over the network.

As as precaution I will also use rsync to regularly backup the entire /etc folder and may be also /var/cache, /var/log etc. just in case.

 

[Dunuin]

Sorry for novice question what is rpool/DATA dataset doing in the pve installation

Storing the virtual disks of your VMs/LXCs you probably already got backuped via VZDump/PBS.

 

[aanjaneyam]

I have been looking at Clonezilla but this method isn't ideal for me as my Proxmox is installed on a mirrored zfs pool and thus Clonezilla only supports full-disk (200GB in my case) backups

I am on the lookout for an easy and not too space consuming way to backup of my Proxmox host.

 

[aanjaneyam]

Storing the virtual disks of your VMs/LXCs you probably already got backuped via VZDump/PBS.

I see. That's also what came to my mind at first but thought it is better to confirm.

 

[Dunuin]

I have been looking at Clonezilla but this method isn't ideal for me as my Proxmox is installed on a mirrored zfs pool and thus Clonezilla only supports full-disk (200GB in my case) backups

I am on the lookout for an easy and not too space consuming way to backup of my Proxmox host.

I boot a Debian from a pen drive and then use the proxmox-backup-client to backup the 3 system partitions on block level to my PBS. Thats incremental and using deduplication, so not that space consuming.

And you can also tell the PVE installer to only create something like a 32GB rpool. Later you could partition the unallocated space with another partition and use that for another ZFS pool. Clonezilla then could only backup the first 3 partitions so 33GB.

 

[maatsche]

Sure. That's what I already pointed out in #7. @Dunuin's points are also true, yet A) is not hard to archieve in an UEFI way. Just regularly rsync your UEFI disk to your ZFS too and it'll be included in the off-site backup. You would also need a ZFS-capable rescue system in order restore your machine from your backup.

Hi LnxBil,

Exactly what i have in mind.

Currently Im running a pve (8.1.3) cluster on 2 dedicated servers. Both servers have pve installed using zfs on 2xnvme in raid1 config. Even though i use pbs for all my lxc/vm backups I need to setup a remote backup for each pve server.

I have enough remote storage to also include lxc/vm volumes. In best case I like to have a complete disk backup that can be restored in case the server fails.

Btw are further backups incremental to the initial backup?

How would I go about restoring the server with a rescue media ?

Could you please describe your solution in more detail ?

Thank you and have a great day!

 

[LnxBil]

Btw are further backups incremental to the initial backup?

Yes. PBS and zfs send/receive are incremental.

How would I go about restoring the server with a rescue media ?
Could you please describe your solution in more detail ?

In general you just need some mechanism to start a rescue system. Most hosting companies have a rescue system in place. The ones I know are Ubuntu or Debian based so, you can just boot from it and install the ZFS userland and compile the kernel module. If you have your own servers with their own management interface and/or physical access, you can use the PVE install media, which contains a rescue system.

 

[Kingneutron]

I have been backing up my "main" proxmox node (LVM+ext4 root) with fsarchiver and tar. Haven't tried a restore yet tho, it's been running 24/7

https://github.com/kneutron/ansitest/blob/master/proxmox/bkpcrit-proxmox.sh

 

[gfngfn256]

I just would like to chronicle my experience(s) with PVE host backup.

I regularily use this script DerDanilo script with some modifications (removed --one-file-system (twice) from script, changed backup dir & max backups to 20). I find it useful for a complete record of all system changes made. I HAVE NEVER USED IT TO DIRECTLY RESTORE ANYTHING.

As far as complete block-level backup of host goes, I tried clonezilla with limited success, (I believe it doesn't handle pve-partitions well). I HAVE NEVER SUCCESSFULLY RESTORED A PVE HOST WITH clonezilla!

Instead I just boot from a live media (I use my own customized SystemRescue; SSH enabled, following commands stored in bash_history) and do the following:

mount /dev/xxx /mnt
#(mount storage device)

tmux
#(to enable leaving the process running)

dd if=/dev/YYY bs=32M status=progress conv=sync,noerror | gzip -c > /mnt/prxmx_host$(date +'%Y_%m_%d_%I_%M_%p').img.gz
#(YYY is PVE system os disk)

I HAVE USED THIS (ABOVE CODE - MODIFIED TO REVERSE) PROCESS FOR A COMPLETE SUCCSESSFUL MIGRATION TO A NEW HOST!

I make a complete block-level backup before any major update/system change.

I have a 512gb nvme system disk, (contains local & lvm, but hardly used as I store VM's CT's etc. on a different storage disk) and backup time takes thirty-something minutes (yes - that's downtime needed!), and produces a zipped file of approximately 7-8 GB.

I know I could try PBS for block-level plus deduplication etc. but cant be bothered to build another host for this. (I run all backups, vzdump to a local disk & rsync them to another external usb disk from time to time)

Anyway that's my host-backup/restore experience plus rant, and yes, for such a robust, sophisticated & well-build PVE system (thanks to the excellent Proxmox team!):
I REALLY WOULD EXPECT PVE TO HAVE AN IN-HOUSE SOLOUTION!

 

[Dunuin]

I REALLY WOULD EXPECT PVE TO HAVE AN IN-HOUSE SOLOUTION!

It's the first point on the roadmap for a very long time. I really hoped to see this when PVE8/PBS3 got announced. Maybe we are lucky and get it with PVE9/PBS4 in a few years.
I'm also curious how they will solve it. I personally would like to see some root filesystem snapshotting that can be planned like backup jobs and that are also taken before doing any upgrade for a easy rollback in case the new version screws something up. And a way to create a backup job that is backing up the PVE config files to a PBS or via VZDump. So like a config export/import feature. And a "Proxmox Backup and Rescue" ISO we could put on a USB stick/PXE and boot from it to select disks that should be backuped or restored to/from the PBS, rollback root filesystem snapshots, restore PVE configs backups and so on. The latter one would also be great to backup any physical host, not just PVE nodes. But I guess its not that easy as PVE isn't an appliance and everyone could be screwed that installs some additional packages or is doing some additional partitioning so the PVE isn'T stock anymore.

 

[gfngfn256]

It's the first point on the roadmap for a very long time.

Yes I can't wait for the crossed out black line!

I like your planned ideas.

Sorry to make cross-platform references, but (even?) Windows has had for years a restore-point system in place, although a full bootable system backup is still rather cumbersome, and can hardly be done cleanly in-house. I do a full Windows backup with the above (Linux) procedure!
However on Linux systems itself this "should" be much easier?

A possible solution in PVE would be to run a PVE (VM style) within a PVE (system) where the outer, higher system-layer is "untouchable". Then online backups could easily be made to the lower PVE (VM style). Something similar to the way many people run a test PVE VM, but without the overhead?

Anyways, enough ranting for now!

 

[LnxBil]

Sorry to make cross-platform references, but (even?) Windows has had for years a restore-point system in place, although a full bootable system backup is still rather cumbersome, and can hardly be done cleanly in-house. I do a full Windows backup with the above (Linux) procedure!
However on Linux systems itself this "should" be much easier?

As already pointed out (in #7), it can be with ZFS. Just snapshot and replicate your root dataset. We've been doing it for years. Restore is just boot from a LiveCD (e.g. PVE installer) and sync back.

For non-ZFS-based installs, ReaR could be used and have been used in the past for PVE, according to this german post. It creates a bootable restore medium and is on my todo list.

 

[gfngfn256]

For non-ZFS-based installs, ReaR could be used

Thats me, I've never used ZFS, and I don't think I'm going down that rabbit-hole just for backups.

Will look into ReaR, looks interesting, never heard of it before. Thanks for info.

Anyone successfully restored/migrated PVE bootable host with ReaR?


One last rant:
I don't understand the "[SOLVED]" tag in this post's title!

 

[Photon4]

Out of curiosity, why isn't a "simple" backup solution of the PVE a thing? I know it's not the same function, but TrueNAS has an option to backup the system config and restore it on a new install, and everything "just works" minus networking.

Is it just not possible for some technical reason? Thankfully I don't have a complicated PVE setup so just following my own documentation if I have to start over isn't really an issue, but I know it is for others.

 

[Dunuin]

Out of curiosity, why isn't a "simple" backup solution of the PVE a thing? I know it's not the same function, but TrueNAS has an option to backup the system config and restore it on a new install, and everything "just works" minus networking.

TrueNAS is only an appliance. There you are not supposed/allowed to touch anything that isn't integrated in the webUI.
PVE is a full OS any you can tamper with it as you like, install whatever you want and so on.
If I decide that I want to install a desktop environment and steam to play some games directly on the PVE host, that is possible. How should the backup feature know how to backup the save games of my games so I can continue playing those games once I restore the backup to some other server?
Its a stupid but still valid example.

You could reduce complexity by only backing up the contents of the config.db but then everyone with an advanced usecase would be screwed that requires to install additional packages or edit Debian config files via CLI. Atleast here, most work when setting up PVE is all the stuff that isn't covered by PVE itself. Monitoring agents, Log collectors, SIEM agent, mail server, custom scripts and systemd services, encryption, UPS client, ... all stuff that has to run on the PVE host and not in a VM/LXC.

If you want to cover all of that too, you would need to backup the whole root filesystem.

 

[gfngfn256]

Out of curiosity, why isn't a "simple" backup solution of the PVE a thing?

Search these forums for endless rantings/whining/workarounds on this subject.

My 2 cents: Make as little changes AP to the PVE host system & document (in a safe place) all changes you've had to make. EVRYTHING else should be in an LXC/VM - then its not to hard to reinstall a fresh system.