Is there a way to backup the PVE Host to the Proxmox Backup Server (PBS)?
- Thread starter logui
- Start date
Backing up the host is not super difficult, but depends a bit on some of the details of how your system is installed. For a host that uses ZFS as root, I do something along these lines:
You can obviously make all sorts of customizations, depending on your local needs. And if you don't use ZFS, then you need to figure out on your own, how to create atomic snapshots. If you have other ZFS filesystems that also need to be backed up, then don't forget to include them here.
Recovery is the big elephant in the room. That's potentially more difficult. If you just want to recover some part of the file hierarchy, that's easy enough to do. But if you want to bring up an entirely new machine, you have to decide what to do about boot strapping, and what to do about differences in hardware.
You could probably boot a Debian rescue disk, manually partition your drives and/or create appropriate ZFS pools and filesystems. Then use the proxmox-backup-client to restore the files. Afterwards, chroot into the new filesystem and configure your boot loader. You also might have to edit /etc/network/interfaces and possibly adjust /etc/systemd/network/*.link files, if you assign stable network interface names based on MAC address.
Editing /etc/hostname wouldn't be a bad idea either.
This is all a little tricky and depending on how good you are with Linux, it could take you a couple of hours to fully restore a system after a catastrophic failure. If you have access to spare hardware, I would try to practice this, or I would attempt to make a custom-built rescue image that you can boot into in order to automate this process. But the specifics will depend a lot on your local needs.
Bash:
#!/bin/bash
# Configure where to store the backup and what to exclude, as it changes
# regularly and only contains emphemeral data.
pbspasswd="/etc/pbs-passwd"
pbshost="pbs.lan"
pbsns="pve-host"
pbsvol="pve-sys"
exclude=( '/dev/shm' '/tmp' '/run' '/var/cache' '/var/lib/rrdcached'
'/var/tmp' '/var/lib/vz#' '/var/log#' )
# Convert the list of exclusions into command line arguments for the PBS
# client. Treat directories like /var/log special, as we want to keep some
# of the directory structure and permissions in the backup, but want to
# discard all of the actual data.
exdirs=
for ex in "${exclude[@]}"; do
exdirs="${exdirs} --exclude ${ex%#}/?*"
[ "${ex%#}" != "${ex}" ] &&
exdirs="${exdirs} --exclude ${ex%#}/**/?* --exclude !${ex%#}/**/"
done
# Take advantage of ZFS to create an atomic snapshot for backing up.
# We keep this snapshot around afterwards, as it is useful for quickly
# repairing accidentally damaged systems.
root="$(zfs list / | awk 'NR==2{ print $1 }')"
zfs destroy "${root}@backup" >&/dev/null || :
zfs snapshot "${root}@backup"
[ -d "/.zfs/snapshot/" ] || zfs set snapdir=visible "${root}"
# The credentials for accessing the PBS server should be stored in
# /etc/pbs-passwd
export PBS_REPOSITORY="$(sed -n 2p <"${pbspasswd}")@${pbshost}:${pbsvol}"
export PBS_PASSWORD_FILE="${pbspasswd}"
# Create a new namespace if it doesn't exist yet, then backup our snapshot
# to the PBS server. Exclude ephemeral data, as it just fills up the server
# and likely won't deduplicate well.
proxmox-backup-client namespace list 2>/dev/null | egrep "^${pbsns}$" >/dev/null ||
proxmox-backup-client namespace create "${pbsns}"
(set -f
proxmox-backup-client backup "proxmox-root.pxar:/.zfs/snapshot/backup" --ns "${pbsns}" \
--change-detection-mode=metadata ${exdirs})You can obviously make all sorts of customizations, depending on your local needs. And if you don't use ZFS, then you need to figure out on your own, how to create atomic snapshots. If you have other ZFS filesystems that also need to be backed up, then don't forget to include them here.
Recovery is the big elephant in the room. That's potentially more difficult. If you just want to recover some part of the file hierarchy, that's easy enough to do. But if you want to bring up an entirely new machine, you have to decide what to do about boot strapping, and what to do about differences in hardware.
You could probably boot a Debian rescue disk, manually partition your drives and/or create appropriate ZFS pools and filesystems. Then use the proxmox-backup-client to restore the files. Afterwards, chroot into the new filesystem and configure your boot loader. You also might have to edit /etc/network/interfaces and possibly adjust /etc/systemd/network/*.link files, if you assign stable network interface names based on MAC address.
Editing /etc/hostname wouldn't be a bad idea either.
This is all a little tricky and depending on how good you are with Linux, it could take you a couple of hours to fully restore a system after a catastrophic failure. If you have access to spare hardware, I would try to practice this, or I would attempt to make a custom-built rescue image that you can boot into in order to automate this process. But the specifics will depend a lot on your local needs.
Here's the script that I am currently using, it focusses on backing up the host configuration, not the payload (VMs, CT, etc.).
The file is located at
Content of that file:
Also, since I use encrypted backups, the encryption key (file) needs to be saved as
Bash:
#!/bin/bash
if [ -f /etc/pve/local/pve-backup.env ] ; then
source /etc/pve/local/pve-backup.env
else
echo "File /etc/pve/local/pve-backup.env missing" > /dev/stderr
exit 1
fi
/usr/bin/proxmox-backup-client backup root.pxar:/ \
--crypt-mode encrypt \
--keyfile /etc/pve/pve-backup.json \
--exclude /bin \
--exclude /boot \
--exclude /dev \
--exclude /lib \
--exclude /lib64 \
--exclude /local-zfs \
--exclude /lost+found \
--exclude /mnt \
--exclude /opt \
--exclude /proc \
--exclude /run \
--exclude /sbin \
--exclude /sys \
--exclude /tmp \
--exclude /usr \
--exclude /var/lib/lxcfs \
--include-dev /etc/pve \
--backup-type host \
--skip-lost-and-foundThe file is located at
/usr/local/sbin/pve-backup.sh and executed by cron daily. The referenced file /etc/pve/local/pve-backup.env is a simple key/value file, containing the Proxmox node specifics. That way I can use the same script on all my PVE nodes unchanged, while only having to adjust the host specific configuration in /etc/pve/local/pve-backup.env .Content of that file:
Bash:
export PBS_REPOSITORY=<API TOKEN USER>!<API TOKEN NAME>@<PBS HOST>:<DATASTORE>
export PBS_PASSWORD=<API TOKEN>
export PBS_FINGERPRINT=<PBS HOST FINGERPRINT>Also, since I use encrypted backups, the encryption key (file) needs to be saved as
/etc/pve/pve-backup.json. If you don't use encryption, simply remove the lines "--crypt-mode" and "--keyfile".
Last edited:
Thanks, backup works just fine. But what is the method to restore such a backup ?
Restoring is more complex, depends a lot more on the specifics of your situation, and that's most likely the reason why Proxmox doesn't officially have any support for backing up the host. It's not that it's difficult per se. But it's difficult to come up with a solution that works for everyone each and every time.
In my case, I've used it a few times to restore a host into a running Chromebook (yes, I know I'm strange; don't ask why). That's probably the hardest target to restore, as you can't even boot from rescue media. So, I had to jump through extra hoops.
In the end, I started out with the default Debian container that ChromeOS installed for me (in hindsight, a minimal Alpine container would probably be even better). I then manually installed and configured the Proxmox client software, FUSE support, and rsync. This allowed me to mount the backup as a file system. And I could then use rsync to copy it over my running system. When done, I rebooted, and executed rsync one more time, just in case anything didn't copy properly on the first run.
This worked surprisingly well. Much better than I had expected. The only thing that required manual repair were permissions on things like ping. I don't know whether they're stored incorrectly in PBS or if rsync dropped capabilities. But that's only a very small number of files and easy to fix.
If you are restoring to raw metal, things are presumably easier. Either boot from rescue media or into the initramfs in rescue mode.
If this is something you expect to do regularly, it wouldn't be difficult to script. But specifics depend a lot on your local requirements. I could see a PXE-based solution that restores from the network.
One of the things you'll have to take care of is the creation of disk partitions for your filesystem(s), and making sure that the ids for these partitions or drives are referenced in all the important places. That might mean the ZFS pool, or things like /etc/fstab and /etc/kernel/cmdline. Don't forget to run proxmox-boot-tool
In my case, I've used it a few times to restore a host into a running Chromebook (yes, I know I'm strange; don't ask why). That's probably the hardest target to restore, as you can't even boot from rescue media. So, I had to jump through extra hoops.
In the end, I started out with the default Debian container that ChromeOS installed for me (in hindsight, a minimal Alpine container would probably be even better). I then manually installed and configured the Proxmox client software, FUSE support, and rsync. This allowed me to mount the backup as a file system. And I could then use rsync to copy it over my running system. When done, I rebooted, and executed rsync one more time, just in case anything didn't copy properly on the first run.
This worked surprisingly well. Much better than I had expected. The only thing that required manual repair were permissions on things like ping. I don't know whether they're stored incorrectly in PBS or if rsync dropped capabilities. But that's only a very small number of files and easy to fix.
If you are restoring to raw metal, things are presumably easier. Either boot from rescue media or into the initramfs in rescue mode.
If this is something you expect to do regularly, it wouldn't be difficult to script. But specifics depend a lot on your local requirements. I could see a PXE-based solution that restores from the network.
One of the things you'll have to take care of is the creation of disk partitions for your filesystem(s), and making sure that the ids for these partitions or drives are referenced in all the important places. That might mean the ZFS pool, or things like /etc/fstab and /etc/kernel/cmdline. Don't forget to run proxmox-boot-tool
Last edited:
Code:
#!/bin/bash
if [ -f ~/backup.env ] ; then
source ~/backup.env
else
echo "File ~/backup.env missing" > /dev/stderr
exit 1
fi
/usr/bin/proxmox-backup-client backup root.pxar:/ \
--exclude /bin \
--exclude /boot \
--exclude /dev \
--exclude /lib \
--exclude /lib64 \
--exclude /local-zfs \
--exclude /lost+found \
--exclude /mnt \
--exclude /opt \
--exclude /proc \
--exclude /run \
--exclude /sbin \
--exclude /sys \
--exclude /tmp \
--exclude /zstor \
--exclude /usr \
--exclude /var/lib/lxcfs \
--include-dev /etc/pve \
--backup-type host \
--skip-lost-and-foundbackup.sh and backup.env located@ /root
backup.env is user, password and fingerprint copied from the <datastore> <your backup> <show connection info> in pbe
setup as a cron job if you dare
Last edited:
I faced the same question and built a little tool called TheReaniamtor for this purpose. It pulls the backups of the host configurations graphically via the API/SSH so that you don't have to do everything manually on the console.
The Reanimator
The Reanimator
I would imagine that it would be best if it worked a bit like in Home Assistant or the Ubiquiti hardware. The idea would be that you do a clean Proxmox install on a machine and then you get an option to either use it directly or to initialize it from a backup. That process could check if your hardware is compatible with the backup (enough storage for thze pools and such) and would only let you proceed if a restore is possible,
This might not be super helpful when restoring on a totally different system, but it would catch the typical scenario where you need to reinstall on the same hardware for some reason. And given how homogenous server farms often are, it would probably be better than nothing at all even for other scenarios.
The trouble with that is, that (since it's basically a customized Debian) ProxmoxVE is not a locked-down system but quite flexible, you can do everything you could also do with a regular Linux (although it's often enough not a good idea). So (different to HA, a router os etc) you can do way more things (e.g. installing drivers or tools to work with exotic hardware) but you can't just backup the configuration data and call it a day.
What you can do right know is to treat the OS install of ProxmoxVE like a regular Linux server and backup it with the same tools (but excluding the vm/lxc storage of course, no point in backing them up again if you already backup them with the integrated backup feature).
So any backup tool for linux (rsnapshot, rsync, borg, tar, restic, duplicati, kopai etc) would work. You could even use the proxmox-backup-client to backup to PBS, but then you will need the PBS for restore. Most regular Linux tools just need access to the backup store (be it on local disc, s3, a network share or a ssh/sftp server), which is why I prefer them for the PVE host backup.
Another variant (if you happen to use zfs) is zfs send/receive to replicate snapshots of your os to a remote location.
IIn a homogenous server farm you will usually put them together in clusters and in a cluster you don't care about single nodes. If one fails you can remove them from the cluster, wipe and reinstall them and add them again to the cluster. The cluster features of PVE will take care to replicate the pve-specific configuration to the new node. Host backup is more about single-node-setups and if you did a lot of customizing.