Advice on using Restic with VM backups

[rbl]

Just started to use Proxmox for home lab on Lenovo mini PC.

Other servers in my network have cron jobs that use Restic to sftp to backup to backup server running Samba and sshd. Have experimented with Minio on the file server for immutable backups.

So, now I need to back up Proxmox VMs. I have used the vzdump script to start a restic backup over sftp to the backup server. This is working, but vzdump makes an archive file. Restic looks to be including the whole file in its repo on each backup:

[0:00] 100.00%  4 / 4 index files loaded
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.log
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.vma
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.vma.notes
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.log
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.vma
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.vma.notes

Files:           3 new,     3 removed,     0 changed
Dirs:            0 new,     0 removed
Others:          0 new,     0 removed
Data Blobs:  10032 new,  9993 removed
Tree Blobs:      5 new,     5 removed
  Added:   6.893 GiB
  Removed: 6.887 GiB

This is on two vz dump snapshots a few minutes apart.

What can I do to get smaller restic backups? Should I expand the archive file somehow before running restic?
Are there other approaches I should consider?

 

[Chris]

Just started to use Proxmox for home lab on Lenovo mini PC.

Other servers in my network have cron jobs that use Restic to sftp to backup to backup server running Samba and sshd. Have experimented with Minio on the file server for immutable backups.

So, now I need to back up Proxmox VMs. I have used the vzdump script to start a restic backup over sftp to the backup server. This is working, but vzdump makes an archive file. Restic looks to be including the whole file in its repo on each backup:

[0:00] 100.00%  4 / 4 index files loaded
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.log
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.vma
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.vma.notes
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.log
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.vma
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.vma.notes

Files:           3 new,     3 removed,     0 changed
Dirs:            0 new,     0 removed
Others:          0 new,     0 removed
Data Blobs:  10032 new,  9993 removed
Tree Blobs:      5 new,     5 removed
  Added:   6.893 GiB
  Removed: 6.887 GiB

This is on two vz dump snapshots a few minutes apart.

What can I do to get smaller restic backups? Should I expand the archive file somehow before running restic?
Are there other approaches I should consider?

Hi, if your goal is to make file level backups of VMs using restic, then you will have to do this from within the VM itself. vzdump and in general the Proxmox VE host sees only the block device.

 

[rbl]

Chris,
Thanks for the quick reply. I suspect I am rushing into this without doing enough learning first - a common problem for me as the home lab is not a priority until it stops working.

I have read the Backup section of the manual, but on a first read I'm having some trouble distinguishing between the storage architecture for the host system and that for VMs. It was not clear to me if a backup includes the VM RAM or not. Let me explain my situation:

My host looks like:



My guest (only one at the moment) looks like:

admin1@utilsrv2:~$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 392M 672K 391M 1% /run
/dev/sda1 ext4 31G 5.5G 24G 19% /
tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 392M 0 392M 0% /run/user/1000
/dev/sdb1 ext4 30G 1.6G 27G 6% /mnt/lokistore

(/dev/sdb1 is a mapped USB drive for log storage)

My goals are:

1. disaster recovery - if the PC dies, get another PC, install proxmox, manually install my tweaks, restore VMs to "a day ago" state.
2. file loss - If I mistakenly delete some files, or a service goes crazy, recover a set of files from N days or weeks ago.
3. (eventually) ransomware recovery - if a ransomware virus/trojan/etc. gets into this machine, be able to recover to "before encryption" or "before compromise" state. Similar to (1) but with immutable backups.

Restic with a S3 storage will do this at a guest level, but with some manual work. On a simple ext4 file system, I try to be careful and pause services to get a clean state while Restic is doing the backup overnight.

I suppose I could do this more easily at the guest level by using ZFS or BTRFS as the guest file system and taking a disk snapshot.

One of my hopes in using Proxmox was that a solution to this situation would be easier. Are there any recommendations or best practices I should consider? Or do you have some links I could read? This seems like a generic problem, but the only well-known solutions I've come across involve commercial products like Veeam, which is more money that I want to spend on a home server.

 

[Chris]

Chris,
Thanks for the quick reply. I suspect I am rushing into this without doing enough learning first - a common problem for me as the home lab is not a priority until it stops working.

I have read the Backup section of the manual, but on a first read I'm having some trouble distinguishing between the storage architecture for the host system and that for VMs. It was not clear to me if a backup includes the VM RAM or not. Let me explain my situation:

My host looks like:

View attachment 68114

My guest (only one at the moment) looks like:

admin1@utilsrv2:~$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 392M 672K 391M 1% /run
/dev/sda1 ext4 31G 5.5G 24G 19% /
tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 392M 0 392M 0% /run/user/1000
/dev/sdb1 ext4 30G 1.6G 27G 6% /mnt/lokistore

(/dev/sdb1 is a mapped USB drive for log storage)

My goals are:

1. disaster recovery - if the PC dies, get another PC, install proxmox, manually install my tweaks, restore VMs to "a day ago" state.
2. file loss - If I mistakenly delete some files, or a service goes crazy, recover a set of files from N days or weeks ago.
3. (eventually) ransomware recovery - if a ransomware virus/trojan/etc. gets into this machine, be able to recover to "before encryption" or "before compromise" state. Similar to (1) but with immutable backups.

Restic with a S3 storage will do this at a guest level, but with some manual work. On a simple ext4 file system, I try to be careful and pause services to get a clean state while Restic is doing the backup overnight.

I suppose I could do this more easily at the guest level by using ZFS or BTRFS as the guest file system and taking a disk snapshot.

One of my hopes in using Proxmox was that a solution to this situation would be easier. Are there any recommendations or best practices I should consider? Or do you have some links I could read? This seems like a generic problem, but the only well-known solutions I've come across involve commercial products like Veeam, which is more money that I want to spend on a home server.

Well, all your requested features can be covered without any additional workarounds by using Proxmox VEs backup functionality in combination with a Proxmox Backup Server instance https://pbs.proxmox.com/docs/
This does however require an additional external host for the PBS.

 

[rbl]

Thanks for the info, I will look into PBS.

 

[ilium007]

I was hoping to back up the PBS chuckstore to online storage using Restic. Did you ever get this configured?

 

[rbl]

No, I gave up on it. Occasional backups in Proxmox, nightly Restic incrementals in the VM. PBS was too much to learn/spend for this little home lab.

 

[ilium007]

No, I gave up on it. Occasional backups in Proxmox, nightly Restic incrementals in the VM. PBS was too much to learn/spend for this little home lab.

I got mine running. Restic backing up the PBS chunk store. Works great!