[SOLVED] Official way to backup proxmox VE itself?

[Kingneutron]

https://github.com/kneutron/ansitest/tree/master/proxmox

Have a look at the bkpcrit-proxmox and bkpsys-2fsarchive scripts. There's also a restore script under Virtbox:

https://github.com/kneutron/ansitest/blob/master/VIRTBOX/RESTORE-fsarchive-root.sh

This should work fine for LVM+ext4; schedule both to run every Friday and before making ANY system changes, including config file edits and package upgrades.

You should probably run the proxmox installer to recreate your disk setup, then boot into a rescue environment and use the script to restore from the .fsa file. After rebooting, your whole proxmox environment should be back to the way it was.

 

[Photon4]

TrueNAS is only an appliance. There you are not supposed/allowed to touch anything that isn't integrated in the webUI.
PVE is a full OS any you can tamper with it as you like, install whatever you want and so on.
If I decide that I want to install a desktop environment and steam to play some games directly on the PVE host, that is possible. How should the backup feature know how to backup the save games of my games so I can continue playing those games once I restore the backup to some other server?
Its a stupid but still valid example.

You could reduce complexity by only backing up the contents of the config.db but then everyone with an advanced usecase would be screwed that requires to install additional packages or edit Debian config files via CLI. Atleast here, most work when setting up PVE is all the stuff that isn't covered by PVE itself. Monitoring agents, Log collectors, SIEM agent, mail server, custom scripts and systemd services, encryption, UPS client, ... all stuff that has to run on the PVE host and not in a VM/LXC.

If you want to cover all of that too, you would need to backup the whole root filesystem.

That makes sense, thanks for the info!

My 2 cents: Make as little changes AP to the PVE host system & document (in a safe place) all changes you've had to make. EVRYTHING else should be in an LXC/VM - then its not to hard to reinstall a fresh system.

Thankfully, I don't have my instance doing too much and I already made my documentation so I can recreate things as needed. I'm replacing my current Proxmox setup running on a NUC with the new Minisfourm MS-01 which is what lead me down the rabbit hole trying to migrate over if possible or fresh install. The other rabbit hole I'm looking down is a single disk OS install or RAID 1 with a complicated replacement process... Haven't decided on that one though.

 

[Kingneutron]

https://github.com/kneutron/ansitest/tree/master/proxmox

Have a look at the bkpcrit-proxmox and bkpsys-2fsarchive scripts. There's also a restore script under Virtbox:

https://github.com/kneutron/ansitest/blob/master/VIRTBOX/RESTORE-fsarchive-root.sh

This should work fine for LVM+ext4; schedule both to run every Friday and before making ANY system changes, including config file edits and package upgrades.

You should probably run the proxmox installer to recreate your disk setup, then boot into a rescue environment and use the script to restore from the .fsa file. After rebooting, your whole proxmox environment should be back to the way it was.

Update - I have been working all morning, testing a restore of my primary PVE host into a VM and put out some proxmox-specific restore scripts.

https://github.com/kneutron/ansitest/tree/master/proxmox

Caveat: The restore won't boot grub for some reason, so keep a copy of Super Grub Disc around and that should boot it. Once up and running you can login as root and ' grub-install /dev/blah '

NOTE - if you test-restore your PVE into a VM and there are zpool(s) and mount points that exist on the host but not in-vm, you should:


systemctl disable zfs-import@zpoolnamehere

in-vm and it should not hold up the reboot anymore. Either that or just recreate the zpool.

Also make sure /etc/fstab is not trying to mount things that don't exist in-vm.

You may also want to disconnect the VM's network interface when test-booting so you don't get IP conflicts.

Please feel free to test these scripts and let me know if there are any issues.

You should know that you'll be able to restore your PVE non-zfs root reliably and get back to a running state in a DR situation, what steps need to be taken to accomplish that goal, and the ETA to accomplish same. 1-2 hours is not unreasonable if you have to change out a disk or have a large root backup / slow network; if you get done before that, so much the better.

 

[Kingneutron]

Update - I have been working all morning, testing a restore of my primary PVE host into a VM and put out some proxmox-specific restore scripts.

https://github.com/kneutron/ansitest/tree/master/proxmox

Caveat: The restore won't boot grub for some reason, so keep a copy of Super Grub Disc around and that should boot it. Once up and running you can login as root and ' grub-install /dev/blah '

BTW I believe I've fixed the problem with grub reinstall, latest script has been uploaded to github

 

[Kingneutron]

The other rabbit hole I'm looking down is a single disk OS install or RAID 1 with a complicated replacement process... Haven't decided on that one though.

For a homelab, you shouldn't need a mirror/RAID1 disk for the OS. Backup your root frequently, have a spare disk handy in case you need to restore/recreate. With proper backups and a TESTED DR procedure, you should be able to get back up and running in ~2 hours or less.

RAID1 for the OS is more for datacenters and prod-critical environments where they can't afford downtime.

 

[b101]

For a homelab, you shouldn't need a mirror/RAID1 disk for the OS. Backup your root frequently, have a spare disk handy in case you need to restore/recreate. With proper backups and a TESTED DR procedure, you should be able to get back up and running in ~2 hours or less.

RAID1 for the OS is more for datacenters and prod-critical environments where they can't afford downtime.

lol, a lab is a lab.

I don't think your point of not using xyz redundancy scheme(or at all) makes any sense what so ever just because it's a lab. I truly believe a lab's purpous(whether it may be office or home) is to test any and all conditions to gain knowledge.

 

[bentrevor]

very informative thread. I have 2 questions.
1. i only have 1 host, with proxmox installed on 1 zfs pool, and a second zfs pool for storage attached to it. is my best option for snapshotting the host

zfs snapshot rpool

?
2. for vms / containers, is it better to snapshot those with proxmox backup on the rpool, or zfs snapshot those as well?

 

[Dunuin]

zfs snapshot rpool

"rpool" is empty with the important datasets "rpool/data" and "rpool/ROOT/pve-1" on top. You will have to do recursive snapshots so those datasets will be snapshotted too. And always keep in mind that snapshots are no backups. One you accidentally destroy the pool (this is happening often...) or it corrupts for some reason, you will be without any proper backups.

2. for vms / containers, is it better to snapshot those with proxmox backup on the rpool, or zfs snapshot those as well?

I wouldn't use snapshot if you want to keep them for more than a few day. Best would be to get some old PC (a free dumpster one with dual core and 2-4GB RAM would be fine, then drop a small HDD or better a SSD in it) and install PBS on it. If that is not an option you could connect a dedicated backup HDD or NAS to your PVE and do VZDump backups there.

 

[bentrevor]

"rpool" is empty with the important datasets "rpool/data" and "rpool/ROOT/pve-1" on top. You will have to do recursive snapshots so those datasets will be snapshotted too. And always keep in mind that snapshots are no backups. One you accidentally destroy the pool (this is happening often...) or it corrupts for some reason, you will be without any proper backups.


I wouldn't use snapshot if you want to keep them for more than a few day. Best would be to get some old PC (a free dumpster one with dual core and 2-4GB RAM would be fine, then drop a small HDD or better a SSD in it) and install PBS on it. If that is not an option you could connect a dedicated backup HDD or NAS to your PVE and do VZDump backups there.

Thank you. Further question, can I use that recursive zfs snapshot to rollback individual VMs / containers e.g

zfs rollback rpool/root/data/subvol_example_vm@example_snapshot

? Would that be functionally equivalent to using the backups in the proxmox gui to rollback?

 

[Dunuin]

Thank you. Further question, can I use that recursive zfs snapshot to rollback individual VMs / containers e.g

zfs rollback rpool/root/data/subvol_example_vm@example_snapshot

? Would that be functionally equivalent to using the backups in the proxmox gui to rollback?

This would only roll back the virtual disks and confuse PVE as the VMs config files would have to be edited too which is only done by managing snapshots via PVE GUI/API/CLI.
Either create those VM/LXC snapshots manually via webUI or use some scripts like cv4pve-autosnap that is using the PVE API for automated snapshotting/pruning: https://github.com/Corsinvest/cv4pve-autosnap

And again, snapshots should only be used in combination with proper backups. So either backups or backups + snapshots but not just snapshots.

 

[kssxs]

BTW I believe I've fixed the problem with grub reinstall, latest script has been uploaded to github

Hi there,

I'm new to this Proxmox thing, but am going to be setting it up on physical hardware in the next couple of weeks. I've played with it a bit in a VM on Windows, so understand the basics.

Part of my setup concerns backups and being able to restore the host to full working status, same as before the failure, after a failure or misconfiguration, etc. I realize that until an official backup solution is given to us, we have to make do with community-driven initiatives.

Backups would be done for UEFI-based VM's (OPNsense/PiHole/Unifi).

I had the following questions:

1) Your site has a bunch of scripts. Am I right in assuming the ones that I should be looking at using are named:

a) bkpcrit-proxmox.sh
b) bkpsys-2fsarchive
c) RESTORE-fsarchive-root-proxmox--EFI.sh

Is this list complete, or are there other scripts that should be used to create/restore any of those backups? I'm assuming you could run these scripts from the 'Shell' within Proxmox itself, correct?

The idea would be to use these scripts as well as the built-in VM backup in Proxmox for the VM's.

2) Is Veeam a better backup strategy for the PVE Host (and again the built-in VM backup in Proxmox for the VM's themselves)?

Not to throw shade at your scripts, heh, I'm just looking for something simple and easy to do. My backups would be stored on a NAS on the same network so I'd want to map a path over to the NAS share before running the backup.

Any help and answers you can provide would be greatly appreciated!

Thank you!

 

[gfngfn256]

Your site has a bunch of scripts

AFAIK None of the scripts are PVE sanctioned, tested or endorsed.

PVE has a robust full backup & restore system for all its VM's & LXC's - you could also use PBS for this - but not required.

What's missing is a Host backup: i.e. the Host OS system does not have a backup system of itself. Any scripts you find are not a replacement for this. They attempt to "save" configs & settings etc. of your host - which supposedly should be able to help you when you later reinstall your Host OS & attempt to reconfigure it "as it was". I have not found much about users who actually did this - successfully.

For my personal mileage, see my above post.

 

[Kingneutron]

AFAIK None of the scripts are PVE sanctioned, tested or endorsed.

PVE has a robust full backup & restore system for all its VM's & LXC's - you could also use PBS for this - but not required.

What's missing is a Host backup: i.e. the Host OS system does not have a backup system of itself. Any scripts you find are not a replacement for this. They attempt to "save" configs & settings etc. of your host - which supposedly should be able to help you when you later reinstall your Host OS & attempt to reconfigure it "as it was". I have not found much about users who actually did this - successfully.

For my personal mileage, see my above post.

> 2) Is Veeam a better backup strategy for the PVE Host (and again the built-in VM backup in Proxmox for the VM's themselves)?

Only you can tell, I haven't tested it personally bc my scripts work for me. I successfully bare-metal backed up my PVE host with fsarchiver and modified my existing restore scripts to restore to a Proxmox VM, and documented the process as comments in the script. I put the restore in an isolated host-only network so as not to have an IP address conflict and it boots just fine.

Also successfully used that restore to test an LVM resize. Admittedly it's not exactly plug-and-play, you have to know what you're doing as a Linux sysadmin, but it should be reproducible if you follow the instructions. EDIT: Please feel free to test, and if the process doesn't work for you, kindly provide constructive feedback so the script/documentation can be improved.

As an alternative, I also encourage people to try other methods - such as the Veeam agent for Linux. It's up to each individual to find a reliable backup/restore method that works for them; I found a reliable one that has worked for me dozens of times P2P, P2V and V2P and documented + shared my scripts with the world.

If I were a proxmox app developer, I'd be looking at integrating an fsarchiver restore into the PVE installer. It not only parallel compresses your filesystem, it also saves the UUID of the partition and restores that as well! And it can convert an ext4 FS to XFS on the fly for restores. Some really nice features with that program.

FYI, Comments from the PVE restore VM notes:
[[

NOTE - leave this on hostonly network to avoid IP conflicts


2024.0420 + added 2xVHD for zpools


2024.0419 successfully tested fsarchiver restore of LVM+ext4 root from qotom-proxmox


To match physical host, Recreate /dev/vda as follows:


Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  biosboot
   2            2048         2099199   1024.0 MiB  EF00  EFI
   3         2099200       167772160   79.0 GiB    8E00  proxmox # LVM
   4       167772162       169410561   800.0 MiB   8200  Linux swap
   5       169410562       291506175   58.2 GiB    8E00  Linux LVM
   8       291506176       497027071   98.0 GiB    A504  FreeBSD ZFS


Actual test restore only requires this for root:


Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  biosboot
   2            2048         2099199   1024.0 MiB  EF00  EFI
   3         2099200       167772160   128.0 GiB   8E00  proxmox

Run the Proxmox PVE installer from ISO / USB to recreate the LVM+ext4 first with appropriate disk and root FS sizes

Then run RESTORE-fsarchive-root-proxmox--EFI.sh


=======================================
Full restore instructions:


Boot systemrescuecd


cd /tmp
Fire up ' mc ' Midnight Commander


Tab to right pane, Esc+9 (or F9) and SFTP to where your .fsa backup file(s) are


SCP the appropriate restore script over to local /tmp and ' chmod +x ' it


EDIT THIS SCRIPT (look for EDITME) and change the appropriate values to match your restore environnment


Follow the sshfs HOWTO in the comment at the beginning of this script to mount your backup file storage


cd /mnt/restore
/tmp/$0 backupfilename.fsa # Start the restore

From here you can still chroot into the /mnt/tmp2 dir and disable zfs imports, edit /etc/fstab, et al

' systemctl disable zfs-import@zpoolnamehere ' # before reboot


Shutdown, remove rescue media and it should reboot into Proxmox VE
If not, boot Super Grub Disc and that should do it


Once you have a PVE login prompt, you can login as root and reinstall grub
' grub-install /dev/blah '


and then do a test reboot to make sure everything comes up as expected.

]]

 

[kssxs]

Thanks guys...

I'm coming in from the Windows side of things, so when dealing with Linux/*BSD etc, I try my best, but am limited by my knowledge and what I can readily find on the Internet. Most of the things you find, as I'm sure you all know, don't necessarily line up with what you're running on the box in front of you, and the learning curve and time needed can be steep.

The concern I'm trying to mitigate for myself is having an Internet router setup in front me, doing an upgrade, and having it take down my connection because the update went badly. I need a quick backup and restore for that situation.

So, keeping in mind that there is no 'official' or 'sanctioned' solution for backup that can be restored by someone without the Linux Sysadmin-level of knowledge, I'll have to take the all or nothing approach. That means learning to use Clonezilla (as I've read others have done to create a one-time static backup before upgrades) to create an image that can be restored just as easily.

Regardless, I appreciate your feedback, and maybe at some point in time I'll have enough knowledge to be able use your scripts successfully for my specific use-case.

Thanks again!

 

[gfngfn256]

I'm coming in from the Windows side of things .................. a quick backup and restore for that situation .............. 'official' or 'sanctioned' solution for backup

So tell me - in your Windows world - what's the quick backup & restore method? Yes you've got Restore points, Rescue mode etc. but quick backup & restore for a non-bootable situation doesn't exist. I live in that Windows world condition too. An OS gone wrong, BSOD - go search all over the internet - "reinstall Windows"! You know the game. Yes, there are some third-party solutions for complete disk backups out there that sometimes work (Windows offline & they are basically Linux dd imaging of the disk OS too) but no: Windows also doesn't have an 'official' or 'sanctioned' solution for complete backup.

The challenge of a full & restorable backup solution on a main System OS disk - is the online/offline situation, so basically you cannot make a true backup of any System OS that is running, this holds true on ANY OS; Windows or Linux.

That means learning to use Clonezilla

Firstly Clonezilla is pretty simple. But secondly you don't need it. Just search "how to make a complete compressed disk image in Linux with the dd command" - its not very hard. (On a side note - I don't use Clonezilla for a Proxmox system - he doesn't understand the partition tables correctly).

Let's face it - you are on this site because you want to install/setup/run/config/fix a full Hypervisor Computer System - so learning how to make a complete disk image should be child's play compared to that.

Another side point: In my personal experience, and I use Windows on a daily basis - Linux is far more resilient & forgiving.

 

[Kingneutron]

> basically you cannot make a true backup of any System OS that is running, this holds true on ANY OS; Windows or Linux

You should check out Veeam free agent for Windows and AOMEI, both do in-situ bare-metal backup and restore for Windows using shadow copies (without needing to reboot for backup) - and can be used for P2V into a VM or P2P to different hardware, bc they use driver injection. Both create bootable ISOs and can also restore backups over Samba shared drive. Give it a try, you might like it. Restore your main Win environment into a VM and you have a portable bootable backup that could even run on a cluster.

In my experience, I've used fsarchiver more than dozens of times to bare-metal backup and restore my Linux rootfs (ext4 / XFS) for P2V, P2P and V2P with absolutely no issues. YMMV if you're running a DB, might want to shut that down first but again, you don't need to reboot into a recovery environment for the backup - just for the restore.

 

[gfngfn256]

Give it a try, you might like it.

Its always been on my Roadmap - but never got there!

Restore your main Win environment into a VM

I have that already - I run 2 Windows VMs created purely with dd full disk images from bare-metal Windows. You still need to know what you are doing from a setup-perspective (VM config, drivers & Windows Authentication etc.), IMO its not for the novice user, but surely resourceful.

 

[Dunuin]

You should check out Veeam free agent for Windows

Have you tried it the Veeam Backup & Replication server? I'm using the Veeam Windows agent here backing up to SMB but this one is basically a no-go without that Veeam Backup & Replication server. I don't want to rely on backups where the client is able to wipe all my backups because it needs to be able to overwrite/delete old backup images for compacting/merging/pruning maintainance tasks. So no ransomware protection. And while the backups don't take that long, the maintainance tasks do and the client has to do them. Always annoying if I can't shut that Windows client down because it needs another 3 hours for that after the weekly backup. Here it would also be useful if a server could do that on its own at night.
Looks like that could be solved by using that Veeam Backup & Replication server but I had a look at the minimum system requirements and those are pretty bad. If I remember it correctly, it needs Windows (no full Linux support) and lots of RAM and CPU (actually more demanding than the Windows client I want to backup...). So sounds like it is only worth it with lots of client and not if you only need to backup 2 or 3 machines at home.

Did you try bacula or borg for bare metal windows/linux client backups?
Wanted to try those in the near future as their backup server is less demanding and could run on linux.
Any recommendation?

 

[Kingneutron]

So no ransomware protection

Nope, haven't tried VBR, I use ZFS Samba-shared drive with snapshots and make multiple backups for ransomware protection. And I don't run Windows for the most part outside of a host-only-network restricted VM where all network comms go through Pihole + Squid proxy using an SSH tunnel with port forwarding.

 

[Dunuin]

I use ZFS Samba-shared drive with snapshots and make multiple backups for ransomware protection.

I'm not using snapshots for the Veeam share because it wastes way too much space as compacting the incremental backups will write all that data again and remove the old copy just that then nothing will every be freed up. I don't want to waste multiple TBs for snapshots to store 1TB of backups when I store snapshots for some months for ransomware protection. Even if the backed up data isn't changing that much between backups, those active full backups or compacting tasks will write all that full data again and again. Only temporarily, but because of snapshotting it is then permanent until you destroy all those snapshots.

Maybe you aren't doing those optional maintainance tasks in the advanced options like "Full Backup File Maintainance: Defragment and compact full backup file" and "Storage-level corruption guard: Perform backup file health check" or "Active full backup: Create active full backups periodically"?