Bare Metal backup of entire Hypervisor for DR

[cburkins]

After suffering recent failure of my Proxmox VE boot disk, I'm contemplating a better (but simple) Disaster Recovery Strategy.

Option 1: Boot from a LiveOS, use "dd" to backup my (now unmounted) Proxmox boot disk (safest, but requires prep)

Option 2: Shutdown Proxmox VE application, use "dd" to backup the running OS, startup Proxmox (can't figure out how to shutdown Proxmox VE from CLI)

Any other ideas I'm missing ? I could just use "dd" to backup the live Proxmox VE, but that seems dangerous (too many open files). Also thought about an LVM snapshot to handle the open files, but that seems complicated.

I think this becomes much simpler if I could figure out one of these:

• How to prevent Proxmox VE from starting-at-boot (could reboot the VE host with Proxmox suppressed, backup with "dd", then restart with VE enabled)
• How to shutdown Promox from CLI

 

[Dunuin]

You can also boot into a clonezilla image and use that instead of dd. And yes, without snapshotting you shouldn't backup a running PVE system.
In case you are running a PBS you could also do a block level backup of the system disk by using the proxmox-backup-client booted from a Debian USB stick.

 

[cburkins]

You can also boot into a clonezilla image and use that instead of dd. And yes, without snapshotting you shouldn't backup a running PVE system.
In case you are running a PBS you could also do a block level backup of the system disk by using the proxmox-backup-client booted from a Debian USB stick.

Thanks for the confirmation.... I'll investigate clonezilla to see if it's any better than dd (I know dd is rudimentary, but it works).

 

[cburkins]

Anybody know how to shutdown Proxmox VE from the command-line ? There are a surprising number of unit files....

root@cube:~# systemctl list-unit-files --state=enabled | grep ^pve
pve-cluster.service          enabled enabled
pve-firewall.service         enabled enabled
pve-guests.service           enabled enabled
pve-ha-crm.service           enabled enabled
pve-ha-lrm.service           enabled enabled
pve-lxc-syscalld.service     enabled enabled
pvebanner.service            enabled enabled
pvedaemon.service            enabled enabled
pvefw-logger.service         enabled enabled
pvenetcommit.service         enabled enabled
pveproxy.service             enabled enabled
pvescheduler.service         enabled enabled
pvestatd.service             enabled enabled
pve-daily-update.timer       enabled enabled

 

[Dunuin]

I think you don't mean to shutdown the whole server but to just stop all PVE related services?

You could do a graceful server shutdown by running shutdown -h now

 

[cburkins]

I think you don't mean to shutdown the whole server but to just stop all PVE related services?

That's exactly right. If I can quiesce the filesystem, reduces the risk of open files during a backup.... Not perfect for sure, but I don't really need perfect here.... "Good" is good enough.

 

[Dunuin]

But you are backing up the whole linux OS not just some PVE packages. Without a flush and snapshot 5his might be quite risky.

What I do is shutting down the whole server and backing up the system disks while PVE isn't booted. And that only every couple of months or before upgrading to a new major/minor PVE release.

Then I also run a weekly backup of all the config files which can easily done while PVE is running.

In case of an emergency I then could write the quite old blocklevel backup to a new disk or server to get a working but outdated PVE. Then I can do a apt update && apt full-upgrade and restore the more recent config files from the filelevel backup overwroting the old config files.

 

[cburkins]

But you are backing up the whole linux OS not just some PVE packages. Without a flush and snapshot 5his might be quite risky.

What I do is shutting down the whole server and backing up the system disks while PVE isn't booted. And that only every couple of months or before upgrading to a new major/minor PVE release.

Then I also run a weekly backup of all the config files which can easily done while PVE is running.

In case of an emergency I then could write the quite old blocklevel backup to a new disk or server to get a working but outdated PVE. Then I can do a apt update && apt full-upgrade and restore the more recent config files from the filelevel backup overwroting the old config files.

Agreed, the risk is real. Just sort of surprised there isn't a simple way to shutdown PVE and start it back up.

I like your model (infrequent block-level backups). Is there a known list of config files that need to be restored during a recovery ?

 

[Dunuin]

Agreed, the risk is real. Just sort of surprised there isn't a simple way to shutdown PVE and start it back up.

Its its a full OS with all bells and whistels and not just an appliance. You can‘t easily prevent a OS completely from writing to disk unless its designed in a way to be able to run from a read-only root filesystem. Even if you would stop all PVE services, all the hundreds/thousands of other debian packages and the kernel would still continue writing to the disk.

I like your model (infrequent block-level backups). Is there a known list of config files that need to be restored during a recovery ?

Primarily the /etc/pve folder. But I backup the entire /etc (contains for example the /etc/network/interfaces, /etc/hosts, /etc/vzdump.conf, /etc/resolv.conf), /root (contains the SSH keys and so on), /home (homefolder of custom PAM users) and /var (contains the ISOs, snippets, templates, the PMXCFS DB and so on) folders.

 

[cburkins]

Its its a full OS with all bells and whistels and not just an appliance. You can‘t easily prevent a OS completely from writing to disk unless its designed in a way to be able to run from a read-only root filesystem. Even if you would stop all PVE services, all the hundreds/thousands of other debian packages and the kernel would still continue writing to the disk.

Primarily the /etc/pve folder. But I backup the entire /etc (contains for example the /etc/network/interfaces, /etc/hosts, /etc/vzdump.conf, /etc/resolv.conf), /root (contains the SSH keys and so on), /home (homefolder of custom PAM users) and /var (contains the ISOs, snippets, templates, the PMXCFS DB and so on) folders.

Agree with you on the risk, for sure.

Thx for sharing your backup methodology. Do you restore files one-by-one, or just restore the entire /etc/pve directory over the existing /etc/pve directory ?

 

[Dunuin]

Isn't that easy. Tried that once and PVE then wasn't working anymore. Usually when I restore my old configs I open all files, compare them line by line and only copy the files/lines that I actually want to replace.
Especially when restoring to new hardware you will have to edit some files like /etc/network/interfaces anyway to match the new hardware.

Unfortunately PVE doesn't got a function to import/export configs yet.

 

[cburkins]

I'm still kind of curious if there's a way to tell PVE to not start at boot time. Feels handy for backups for sure, but also if I'm just troubleshooting something, and I want to look around on the PVE Host without the complexity of PVE running....