Proxmox host and client: backup and restore steps

[MrBruce]

Hi!

I also read this before I post. I still new with Proxmox but I like it!
I have a latest Proxmox server using only 1 physical (home) server with Linux software raid, mostly for:

samba: file sharing: movies, music (for raspberry pi music players) and for windows notebooks for movies, music all about 200GB
nfs and tftp: raspberry pi network boot and usage without disk. For 4 raspberry pi with sensors, posting data to mqtt
mysql: for zabbix 5 monitoring, for home assistant and for sensor data
redis: for emoncms
mqtt: for raspberry pi for sensor data manage
influxdb: for sensor data
rrdtool: for sensor data visualization
grafana: for sensor data visualization
emoncms (PHP 7.0.30): for power usage data stats and visualization
zoneminder (PHP: 7.4): for security, with 3 web usb webcam
zabbix 5 (PHP: 7.4): monitoring devices and alarms
VM: home assistant: home automation, sensor data triggers
qbittorrent: for downloading

It's my "all in one" home server.

Using containers and VM. I prefer the smallest downtime, smallest backup size and most safest way, most easiest way if anything happening and need to restore from backup.

Please help me how create the proper way of backups, I tell what I plan to do and please approve my idea or tell better or more optimal. Please tell all important steps what needed to backup and restore. I will test it!
I had another Linux server with 1TB backup HDD (gigabit Ethernet connection), my Proxmox server have 2x1TB HDD too, it's ok for me.

I still thinking about mysql: what version to use? Percona or MariaDB? I prefer Percona.

I planned to backup:

P 0. Making a whole server backup with Acronis True Image or Clonezilla from HDD 1 of 2. It takes a long time and long downtime, as base backup. It will contains all partitions. What other way for that?

P 1. simple files (file server): rsync to backup HDD (using network connection to connect to backup HDD). After first copy it will be fast, because it will sync only the changes.

P 2. VM: currently only 1 VM: home assistant. Seems like the snapshot mode the best for me. Or you recommend me a suspend mode? Any incremental way?

P 3. Containers, if no database server inside: snapshot mode? Or stop mode? Any incremental way?

P 4. Container with mysql: Currnetly mysql data directory outside of container files, so if I backup, I backup only container files without mysql data. Percona have Percona XtraBackup, what is seems like optimal for me, helps me make a proper mysql data dump without mysql stop. Any better idea?

P 5. backing up a Proxmox settings and host: reboot with USB boot + make image backup (like Acronis True Image or Clonezilla) but it makes downtime for long. How do this better? Without downtime?

If I lost my all data from server (both HDD died in raid 1), how to restore, please correct and update my steps:

PLAN V1:

S -1. getting at least same size 2 HDD as before: 2x 1TB installed again.

S 0. restore data from all full image backup from HDD 1 of 2, but it's the oldest, but i get the HDD structure for raid, etc.

S 1. restore image backup from Proxmox host as I do in P 5.

S 2. booting from restored HDD 1, and waiting for rebuild this HDD 2 by Linux Software raid 1.
Because of Linux software raid, I need to update grub to boot from HDD 2? Or what needed? What command?
Now my Proxmox host are the latest and Raid 1 working again, but my all containers and vms are out of date.

S 3. restore latest vm from latest backup

S 4. restore all container from latest backup

S 5. restore all files from latest backup, rsync back from backup

S 6. start all containers and vms, done? I forgot something?

Please help me to correct this steps and backup ways.

PLAN V2: For me, the most easiest way to shutdown Proxmox server, boot from USB (Clonezilla) and create a full image backup to external HDD or to other Linux server.
If any problem, just restore from this image and all done. Or also need to update grub for boot in HDD2?
But it makes maybe long downtime. But maybe faster if I try to make incremental backup, but I don't know it can work in image backup? Also problem I need to do this manually. Possible any automated way of this?

PLAN V3: Or adding 3. HDD to server but not mounted to Proxmox. Late night a planned reboot booting a full backup small partition image (Clonezilla?) what automatically making a full or incremental backup from HDD 1 to HDD 3. It's the smallest downtime possible but for hours also stop. Easy to recover. Problem: 3. HDD inside or 3. HDD need work all times. It will work if Clonezilla will be in 3. HDD first partition and 3. HDD 2. partition will be like EXT4 as data partition?

PLAN V3.5: Same ideas as before, just the backup server at scheduled time, same scheduled time as the Proxmox server are turn on, and backing up to powered on server and after done, shutting down it with command. So, at every Monday 2:00 turn on this backup Linux PC with backup storage, also Proxmox server scheduled stop all services in 1:59 and reboot for backup at 2:00 and 2:05 after boot done in Clonezilla automatically start backup to 2:00 powered on server. After backup done, sending remote shutdown to powered on PC. After booting back to Proxmox and starting back all services. Possible?

PLAN V4: My most stupid idea but like no downtime and easy backup: i first install a Xen (xcp-ng) server and adding 1 big vm using all resources and hdd space. Inside this vm just installing Proxmox with vm + containers. And if I need a full Proxmox backup, just making a snapshot with xcp-ng, and done. Also I save the xcp-ng installation first. If and problem, I restore the xcp-ng backup, starting xcp-ng and restore from snapshot, and all Proxmox settings, vms, containers are back. It will be so slow, right? But possible?

PLAN V5: tell me better please.

For me the PLAN V3.5 seems like the best way, because easiest to recover if any problems because 1 image contains all. Problem: long downtime and need 2. server proper setup and also don't know how possible to do this times reboot from other partition, starting auto backup, etc. Also Raspberry Pi's what is using this Proxmox as file system will be crash, so also need to same time stop and turn on, but don't know how. So better to no downtime solution but so difficult to restore data?

 

[MrBruce]

Invisible topics?

 

[tom]

Please ask simple questions and you will get answers here.

Second, before asking "all", read the documentation and search/read this forum for similar topics - most is covered already in several threads and posts.

 

[MrBruce]

OK, simple question: how can I make a full backup from Proxmox host + VMs + Containers (with mysql server) without downtime? How can I restore from backup? Thank you.

 

[tom]

OK, simple question

I see 5 questions in your post, here are some links for you.

VM and CT backup and Restore:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_vzdump

Proxmox VE host backups are not yet implemented, here is a forum thread about this topic:
https://forum.proxmox.com/threads/best-practice-for-proxmox-self-backup.38382/

 

[pavels]

OK, simple question: how can I make a full backup from Proxmox host + VMs + Containers (with mysql server) without downtime? How can I restore from backup? Thank you.

I suggest to post such confused stories on https://www.reddit.com/r/Proxmox/

 

[MrBruce]

I suggest to post such confused stories on https://www.reddit.com/r/Proxmox/

Thank you, but your answer means = no solution for Proxmox full backup without downtime.

 

[tom]

Thank you, but your answer means = no solution for Proxmox full backup without downtime.

You can backup your VMs in snapshot mode, no downtime.

 

[MrBruce]

You can backup your VMs in snapshot mode, no downtime.

Yes, I know it.

But how backup a Proxmox Linux host + partitions?
How backup containers with databases, like Mysql?

Thank you.

 

[Lephisto]

Hi Bruce,

I understand the brevity of the Proxmox guys. You initial post was pretty confused and more a case for an IT Consultant that gives you a basic understanding of how virtualisation stacks work and what backup concepts there are.

Basically there are two ways in which you should backup data:

1. Traditional Filebackup: Use a Backup Tool of your choice (Borg, Bacula, rsync, you name it) do Backup important files of your VM's to an external Storage which is running independant from your Proxmox Cluster. Combine it with a hook that dumps out your Databases, so you can always restore a consistent state of it

2. Snapshot Backups: Additionally, to achieve a decent RTO you can take Snapshot Backups of your running VM's and Containers: Basically it takes a Clone of your Disks and compresses and writes it somwhere out of your cluster so you can restore it. This can be sone online, while the VM is running with just a little hickup of a few milliseconds while fsfreeze'ing. Though, there are some caveats. Basically taking snapshots is like pulling the power of a running server, and hoping the filesystem was not damaged during that. To reduce the risk of it, there's the qemu-guest-agent, that does a fsfreeze-freeze before, and fsfreeze-thaw after the snapshot. This signals the guest vm to sync inflight data to disk, so you reduce the risk of filesystem corruptions. Still, on the application level you might end up in corrupted databases, because the database agent is not aware of that (allthough, there are hooks for qemu-guest-agent that cann tell mysql databases to write it's files to a consistent state to reduce the risk even further and make applications snapshot aware). (It's a different story with windows, there qemu-guest-agent triggers vss, a windows solution for taking snapshots while machines are running). The VM configurations are saved with Snapshots, too, so in case of a complete machine fail you just setup a clean Proxmox and import the VM's again.

You should really go both approaches, it's always nice to have a second Backup source, just in case you have a systemic failure on one method. Check your backups regulary, make test restores and see if it works. Having snapshots is always nice for a good RTO, restoring a failed system is much faster then reinstalling and migrating the Data from Filebased backups. Still, it's handy to have the Filebackups, just in case you run into inconsitencies.

Additionally: Don't Backup on a Harddisk in the same machine. That's not a Harddisk. At least have a separate Machine for that (simple NAS mounted w/ cifs or nfs), which will at least save you from issues of your powersupplies. Imagine a defective powersupply frying all your components? Nice backup, then. There are also people that say that backups to harddisk are not enough, and you should choose an offline medium (tape).

Still, I strongly suggest to search for professional help, if you intend to use your setup for business purposes.

regards,
Bastian

 

[MrBruce]

Hi, thank you for answer.

I do this for my home server. Not for business. I just prefer a complete solution. Now try to build a Proxmox cluster with 3 node + ceph, and I will turn off 3. node after backup done, and 3. node will be only for weekly sync.
For home usage, 2 pc in cluster more than enough.

my planned setup:

proxmox node: raid 1 hdd with 2x hdd + ceph: 1x 1tb hdd / node
3 node, 1. for primary usage, 2. for data sync, 3. for weekly power on.
I hope I can set live migration from 1 to 2 but not for 3. Yes, I need read more about, I know.