Backup/Restore procedure for Proxmox VE host

[MightySlaytanic]

Hi,
I‘d like to have your opinion or some reference to howtos or manuals about how to backup/restore Proxmox VE Host.

My setup:
1) Proxmox VE installed on Intel NUC on nvme0
2) Proxmox Backup Server in LXC container backed up by Proxmox VE schedule on NFS share
3) Proxmox VMs and CTs (apart from PBS) backed up to PBS that saves the data to /mnt/pve/sata_disk/proxmox_backups via bind-mount using the /dev/sda disk (that disk is automatically mounted by Proxmox VE since I’ve added it as disk->Directory within PVE GUI, without selecting the Add Storage flag since it is used by PBS and not PVE)
4) Daily cron job that archives (with tar cvpjf) /etc folder onto an external NFS share

In case of disaster, I have quite clear ideas about how to restore everything apart from PVE Host:
1) Reinstall PVE from Scratch and then a) restore config from /etc backup on the NFS share or b) reconfigure everything “by hand”?
2) Restore PBS LXC container from NFS share
3) Restore all the VMs and CTs via PBS

First of all, can you confirm me that VMs and CTs’ backups saved by an instance of PVE can be restored by a new install of PVE since they’re self-contained, i.e. they contain all the info necessary to be restored (no encryption in use)? The same for PBS backups restored by a new PBS installation?

My greater concerns are about the first step, i.e. reconfiguration of PVE. If I do a fresh install, can I simply re-install the packages I’ve added via apt install and then copy the /etc backup over the fresh install and reboot the machine and hope to have all the configurations (storages, schedules etc) back to my PVE?

Do I need to back up other file system folders or is /etc sufficient? Should I save the output of some commands like mount to rebuild the PVE host after the disaster?

Would it be feasible to dump (with DD) the entire /dev/mapper/pve-root to an external disk once every X days and in case of disaster do a fresh install (to re-create the same disk layout and partitions, supposing to have a disk with the same size), reboot from USB with a live Linux distro and then DD in the opposite direction? This would be surely more difficult for me since I don’t really know if it is possible to do that and how to do that (I don’t know if the live Linux would see it as /dev/mapper/PvE-root or if I need to do some LVM magic which is something I don’t really know at present time)

Thank you in advance for any help!

 

[Dunuin]

1) Reinstall PVE from Scratch and then a) restore config from /etc backup on the NFS share or b) reconfigure everything “by hand”?

Just replacing the whole /etc folder never worked here. I had to compare every file line by line and decide what lines I want to overwrite and what lines to keep.

First of all, can you confirm me that VMs and CTs’ backups saved by an instance of PVE can be restored by a new install of PVE since they’re self-contained, i.e. they contain all the info necessary to be restored (no encryption in use)? The same for PBS backups restored by a new PBS installation?

Depends. Some stuff like security groups, aliases, IP sets and so on used by the guests are not part of the guest backups. And vzdump backups aren't encrypted, PBS backups can be (optionally) and you should have multiple copies of your key or you won't be able to access your backups, compression and encryption is done client side.

Would it be feasible to dump (with DD) the entire /dev/mapper/pve-root to an external disk once every X days and in case of disaster do a fresh install (to re-create the same disk layout and partitions, supposing to have a disk with the same size), reboot from USB with a live Linux distro and then DD in the opposite direction? This would be surely more difficult for me since I don’t really know if it is possible to do that and how to do that (I don’t know if the live Linux would see it as /dev/mapper/PvE-root or if I need to do some LVM magic which is something I don’t really know at present time)

Also keep in mind that your PVE shouldn't be running (so its unmounted) while backup up your system disk. That makes it really annoying to backup the whole PVE system as you can't automate it. Because of that I usually only do it before doing a major version upgrade so I can restore it in case something won't work anymore, as there is no easy way to downgrade a PVE.
Have a look at a clonezilla USB stick for backing up and restore block devices: https://clonezilla.org/

Its also possible to backup block devices (like the PVE system disk) to a PBS using the proxmox-backup-client. But that only would make sense if your PBS would run on another server.

 

[mokaz]

That's what I'm using:
https://gist.github.com/mrpeardotnet/6bdc4b504f43ce57fa7eaee96d376edf

You will get all the needed configurations files, I'm usually restoring only what's needed after a clean install, certificates, hosts, interfaces etc..

 

[MightySlaytanic]

That's what I'm using:
https://gist.github.com/mrpeardotnet/6bdc4b504f43ce57fa7eaee96d376edf

You will get all the needed configurations files, I'm usually restoring only what's needed after a clean install, certificates, hosts, interfaces etc..

Thank you, looking at the script it does a backup of subdirs of /etc so I whould be fine with backing up the whole /etc plus my /root subdirs where I have some scripts and dumps. In case you have to restore some of the files, do you need to boot in some restricted mode or stop PVE services in order to replace the files with the ones from the backup?

Also keep in mind that your PVE shouldn't be running (so its unmounted) while backup up your system disk. That makes it really annoying to backup the whole PVE system as you can't automate it. Because of that I usually only do it before doing a major version upgrade so I can restore it in case something won't work anymore, as there is no easy way to downgrade a PVE.
Have a look at a clonezilla USB stick for backing up and restore block devices: https://clonezilla.org/

I know that it's not very clean, but I think I can dump the system disk even if it's running. In case of restore it would be like booting from a system where an abnormal power-down has occurred. BTW, I will do as you suggest with a USB live distro before major updates. Does the clonezilla automatically see the LVM PVs like pve-root?

 

[mokaz]

Well TBH with you I'm doing such things manually, meaning full reinstall, minimal networking up and then I'm gathering what I might be missing..
PVE is really an easy resetup, I'm usually up and running in a matter of 5 minutes.. The only important things to me are VMs, which I migrate over, backup through PBS before any other things etc etc.. so the host themselves I try to have them as switchable as possible..

 

[MightySlaytanic]

Well TBH with you I'm doing such things manually, meaning full reinstall, minimal networking up and then I'm gathering what I might be missing..
PVE is really an easy resetup, I'm usually up and running in a matter of 5 minutes.. The only important things to me are VMs, which I migrate over, backup through PBS before any other things etc etc.. so the host themselves I try to have them as switchable as possible..

You're right, I've documented in a page all the customization I've done (like enabling mails to gmail with TLS, monitoring UPS with upsmon, fix for e1000 driver) and I should be up and running in a matter of minutes with PVE and the storages mounted in order to recreate PBS LXC and use it to restore all the VMs. The /etc backup can be useful for reference or to restore some configs like upsmon etc (what is the config file containing info about the storages defined in PVE?).

BTW, I've I reconfigure PVE, with all its storages and network bridges etc, then when I restore the VMs from PBS they all have their configurations set up and embedded in the backup (config like the network bridge and tag to use), correct?

 

[Dunuin]

I know that it's not very clean, but I think I can dump the system disk even if it's running. In case of restore it would be like booting from a system where an abnormal power-down has occurred.

Yeah, search the forums for the hundreds of threads titled something like "Help!...Proxmox won't boot anymore after power outage" where a power outage corrupted the system disk because no UPS was used...
I really wouldn't rely on those backups. And what are backups good for if you can't rely on them?

 

[MightySlaytanic]

Yeah, search the forums for the hundreds of threads titled something like "Help!...Proxmox won't boot anymore after power outage" where a power outage corrupted the system disk because no UPS was used...
I really wouldn't rely on those backups. And what are backups good for if you can't rely on them?

You're right. I have an UPS to avoid possible issues and have a clean shutdown. So, I'll keep my scheduled monthly dump of pve-root as last resort and I'll do a offline dump before major changes as you suggest (BTW I think a reconfiguration from scratch could be the way to go)
BTW, some days ago I've tested the automatic shutdown with upsmon and I've seen that Proxmox started the shutdown process after reaching the critical level of battery. It never shutdown completely but from the console I could only see a black screen with the cursor on top. I know that the VMs that were running can all be shutdown in few seconds (if I shutdown them they come down in less than 20 seconds) but after booting back into the PVE system the task showed that there was the 180 seconds timeout in shutdown of the VMs. How can I debug what happens when proxmox shutsdown?

 

[tonycav]

Hi, I am writing here because I have a similar question.
My configuration is:
120GB SSD: PVV
500GB NVME: (Windows 10 VM + Turkney Server File LXC)
1TB: Shared ZFS
I backup NVME + ZFS disk every day
Once a week (Monday) I will clone the 120GB drive with clonezilla
If I don't make changes to my system and don't download updates, can my starting SSD be replaced with the cloned one?
Thank you for your time

 

[Dunuin]

If I don't make changes to my system and don't download updates, can my starting SSD be replaced with the cloned one?

Yes, but any changes done to PVE will be lost, as the configs files are in the /etc folder on that disk. So I would also do a daily backup of the "/etc" folder, so you don't lose a week of changes. Then you could restore the clonezilla image first and restore the more recent "/etc" folder afterwards.

 

[tonycav]

Thanks a lot for the valuable information

 

[tonycav]

Yes, but any changes done to PVE will be lost, as the configs files are in the /etc folder on that disk. So I would also do a daily backup of the "/etc" folder, so you don't lose a week of changes. Then you could restore the clonezilla image first and restore the more recent "/etc" folder afterwards.

Sorry, but can I also copy the /etc folder if VM and LXC are running?

 

[MightySlaytanic]

Sorry, but can I also copy the /etc folder if VM and LXC are running?

There should be no issues in backing up /etc while VMs and CTs are running. I run a nightly cron jobs that backs up the /etc folder along with other ones to an NFS share mounted by PVE. The script produces a tgz file for every folder, replacing "/" with "_" and before starting the backup it checks if the folder is correctly mounted (it looks for a .FOLDER_EXISTS hidden file):

cat /root/scripts/backup_config_to_nas.sh

#!/bin/bash

NFS_FOLDER="/mnt/pve/synology-backup-pve"
TARGET_FOLDER="${NFS_FOLDER}/snippets"
CHECK_FILE="${NFS_FOLDER}/.FOLDER_EXISTS"

if ! [ -f ${CHECK_FILE} ] ; then
  echo "NFS Share ${NFS_FOLDER} not mounted!" 1>&2
  exit -1
fi

DIRS_TO_BACKUP="/etc /root /opt"

for dir in ${DIRS_TO_BACKUP} ; do tar cpjf ${TARGET_FOLDER}/$(echo $dir | sed -e "s:^/::g" | sed -e "s:/:_:g").tgz $dir ; done

-------------------------------

cat /etc/cron.d/maintenance

# BACKUP CONFIG AND SCRIPTS
0 2 * * * root    /root/scripts/backup_config_to_nas.sh >/dev/null 2>&1

 

[Dunuin]

And in case you are using a Proxmox Backup Server (if not I would highly recommend doing it as backups are way faster and it saves a crapload of space) you can also use the proxmox-backup-client to save the /etc folder on the PBS.

 

[MightySlaytanic]

There should be no issues in backing up /etc while VMs and CTs are running. I run a nightly cron jobs that backs up the /etc folder along with other ones to an NFS share mounted by PVE. The script produces a tgz file for every folder, replacing "/" with "_" and before starting the backup it checks if the folder is correctly mounted (it looks for a .FOLDER_EXISTS hidden file):

cat /root/scripts/backup_config_to_nas.sh

#!/bin/bash

NFS_FOLDER="/mnt/pve/synology-backup-pve"
TARGET_FOLDER="${NFS_FOLDER}/snippets"
CHECK_FILE="${NFS_FOLDER}/.FOLDER_EXISTS"

if ! [ -f ${CHECK_FILE} ] ; then
  echo "NFS Share ${NFS_FOLDER} not mounted!" 1>&2
  exit -1
fi

DIRS_TO_BACKUP="/etc /root /opt"

for dir in ${DIRS_TO_BACKUP} ; do tar cpjf ${TARGET_FOLDER}/$(echo $dir | sed -e "s:^/::g" | sed -e "s:/:_:g").tgz $dir ; done

-------------------------------

cat /etc/cron.d/maintenance

# BACKUP CONFIG AND SCRIPTS
0 2 * * * root    /root/scripts/backup_config_to_nas.sh >/dev/null 2>&1

I’ve forgot to tell that I’m overwriting the previous backup everyday because I have daily snapshots on my target Synology NAS that exposes the NFS folder, otherwise it is better to save to archives with the current data in the name and have a cleanup job to remove old backups. This is important if you need to recover a backup from some days ago

 

[tonycav]

I’ve forgot to tell that I’m overwriting the previous backup everyday because I have daily snapshots on my target Synology NAS that exposes the NFS folder, otherwise it is better to save to archives with the current data in the name and have a cleanup job to remove old backups. This is important if you need to recover a backup from some days ago

many many thanks for your script and for your advice
I am moved and happy

 

[tonycav]

And in case you are using a Proxmox Backup Server (if not I would highly recommend doing it as backups are way faster and it saves a crapload of space) you can also use the proxmox-backup-client to save the /etc folder on the PBS.

Thanks a lot, I'm thinking of using P.B.S
I would like to take things slow

 

[chudak]

Is there the list of files that are to be restored from `etc/` backup available anywhere?

 

[Dunuin]

All files in /etc are used. You are backing up a full-fledged Linux OS, not an appliance. And also keep in mind that you can customize PVE with whatever you want, so every server might look different.

But the most important files used by PVE packages should be:

1. /etc/pve/
2. /etc/vzdump.conf
3. /etc/hosts
4. /etc/network/interfaces
5. /etc/resolv.conf
6. /var/lib/vz/
7. /var/lib/pve-cluster/config.db
8. + what ever you manually change

 

[chudak]

All files in /etc are used. You are backing up a full-fledged Linux OS, not an appliance. And also keep in mind that you can customize PVE with whatever you want, so every server might look different.

But the most important files used by PVE packages should be:

1. /etc/pve/
2. /etc/vzdump.conf
3. /etc/hosts
4. /etc/network/interfaces
5. /etc/resolv.conf
6. /var/lib/vz/
7. /var/lib/pve-cluster/config.db
8. + what ever you manually change

I don't have /var/lib/*

But I don't run clusters. Can I live w/o it?

I am still hoping someone can shed some light on my other post about failed pve7to8 (https://forum.proxmox.com/threads/failed-upgrade-to-pve-8-need-help.130314/)

But assuming I am not recoverable, how do I go about installing pve 8 via ssh session (if it's possible)?