Advice on using Restic with VM backups

R

rbl

New Member
May 13, 2024
6
0
1
Just started to use Proxmox for home lab on Lenovo mini PC.

Other servers in my network have cron jobs that use Restic to sftp to backup to backup server running Samba and sshd. Have experimented with Minio on the file server for immutable backups.

So, now I need to back up Proxmox VMs. I have used the vzdump script to start a restic backup over sftp to the backup server. This is working, but vzdump makes an archive file. Restic looks to be including the whole file in its repo on each backup:

Code: 
[0:00] 100.00%  4 / 4 index files loaded
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.log
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.vma
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.vma.notes
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.log
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.vma
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.vma.notes

Files:           3 new,     3 removed,     0 changed
Dirs:            0 new,     0 removed
Others:          0 new,     0 removed
Data Blobs:  10032 new,  9993 removed
Tree Blobs:      5 new,     5 removed
  Added:   6.893 GiB
  Removed: 6.887 GiB

This is on two vz dump snapshots a few minutes apart.

What can I do to get smaller restic backups? Should I expand the archive file somehow before running restic?
Are there other approaches I should consider?
 
rbl said:
Just started to use Proxmox for home lab on Lenovo mini PC.

Other servers in my network have cron jobs that use Restic to sftp to backup to backup server running Samba and sshd. Have experimented with Minio on the file server for immutable backups.

So, now I need to back up Proxmox VMs. I have used the vzdump script to start a restic backup over sftp to the backup server. This is working, but vzdump makes an archive file. Restic looks to be including the whole file in its repo on each backup:

Code: 
[0:00] 100.00%  4 / 4 index files loaded
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.log
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.vma
-    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-09_04_25.vma.notes
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.log
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.vma
+    /var/lib/vz/dump/vzdump-qemu-100-2024_05_13-16_04_45.vma.notes

Files:           3 new,     3 removed,     0 changed
Dirs:            0 new,     0 removed
Others:          0 new,     0 removed
Data Blobs:  10032 new,  9993 removed
Tree Blobs:      5 new,     5 removed
  Added:   6.893 GiB
  Removed: 6.887 GiB

This is on two vz dump snapshots a few minutes apart.

What can I do to get smaller restic backups? Should I expand the archive file somehow before running restic?
Are there other approaches I should consider?
Click to expand...
Hi, if your goal is to make file level backups of VMs using restic, then you will have to do this from within the VM itself. vzdump and in general the Proxmox VE host sees only the block device.
 
Chris,
Thanks for the quick reply. I suspect I am rushing into this without doing enough learning first - a common problem for me as the home lab is not a priority until it stops working.

I have read the Backup section of the manual, but on a first read I'm having some trouble distinguishing between the storage architecture for the host system and that for VMs. It was not clear to me if a backup includes the VM RAM or not. Let me explain my situation:

My host looks like:

Screenshot 2024-05-14 140719.png

My guest (only one at the moment) looks like:

admin1@utilsrv2:~$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 392M 672K 391M 1% /run
/dev/sda1 ext4 31G 5.5G 24G 19% /
tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 392M 0 392M 0% /run/user/1000
/dev/sdb1 ext4 30G 1.6G 27G 6% /mnt/lokistore

(/dev/sdb1 is a mapped USB drive for log storage)

My goals are:

1. disaster recovery - if the PC dies, get another PC, install proxmox, manually install my tweaks, restore VMs to "a day ago" state.
2. file loss - If I mistakenly delete some files, or a service goes crazy, recover a set of files from N days or weeks ago.
3. (eventually) ransomware recovery - if a ransomware virus/trojan/etc. gets into this machine, be able to recover to "before encryption" or "before compromise" state. Similar to (1) but with immutable backups.

Restic with a S3 storage will do this at a guest level, but with some manual work. On a simple ext4 file system, I try to be careful and pause services to get a clean state while Restic is doing the backup overnight.

I suppose I could do this more easily at the guest level by using ZFS or BTRFS as the guest file system and taking a disk snapshot.

One of my hopes in using Proxmox was that a solution to this situation would be easier. Are there any recommendations or best practices I should consider? Or do you have some links I could read? This seems like a generic problem, but the only well-known solutions I've come across involve commercial products like Veeam, which is more money that I want to spend on a home server.
 
rbl said:
Chris,
Thanks for the quick reply. I suspect I am rushing into this without doing enough learning first - a common problem for me as the home lab is not a priority until it stops working.

I have read the Backup section of the manual, but on a first read I'm having some trouble distinguishing between the storage architecture for the host system and that for VMs. It was not clear to me if a backup includes the VM RAM or not. Let me explain my situation:

My host looks like:

View attachment 68114

My guest (only one at the moment) looks like:

admin1@utilsrv2:~$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 392M 672K 391M 1% /run
/dev/sda1 ext4 31G 5.5G 24G 19% /
tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 392M 0 392M 0% /run/user/1000
/dev/sdb1 ext4 30G 1.6G 27G 6% /mnt/lokistore

(/dev/sdb1 is a mapped USB drive for log storage)

My goals are:

1. disaster recovery - if the PC dies, get another PC, install proxmox, manually install my tweaks, restore VMs to "a day ago" state.
2. file loss - If I mistakenly delete some files, or a service goes crazy, recover a set of files from N days or weeks ago.
3. (eventually) ransomware recovery - if a ransomware virus/trojan/etc. gets into this machine, be able to recover to "before encryption" or "before compromise" state. Similar to (1) but with immutable backups.

Restic with a S3 storage will do this at a guest level, but with some manual work. On a simple ext4 file system, I try to be careful and pause services to get a clean state while Restic is doing the backup overnight.

I suppose I could do this more easily at the guest level by using ZFS or BTRFS as the guest file system and taking a disk snapshot.

One of my hopes in using Proxmox was that a solution to this situation would be easier. Are there any recommendations or best practices I should consider? Or do you have some links I could read? This seems like a generic problem, but the only well-known solutions I've come across involve commercial products like Veeam, which is more money that I want to spend on a home server.
Click to expand...
Well, all your requested features can be covered without any additional workarounds by using Proxmox VEs backup functionality in combination with a Proxmox Backup Server instance https://pbs.proxmox.com/docs/
This does however require an additional external host for the PBS.
 
No, I gave up on it. Occasional backups in Proxmox, nightly Restic incrementals in the VM. PBS was too much to learn/spend for this little home lab.
 
ilium007 said:
I got mine running. Restic backing up the PBS chunk store. Works great!
Click to expand...
I'm curious.

Can you tell us about deduplication, as PBS does its deduplication first and restic does his after what's the result ?
Is the restic backup smaller that the PBS one ? Larger ? Same size ?
As far as I know the two dedup methods are different.
If it's the same size, this could mean that restic dedup can't go farther on chunking and dedup so..

And more genral wuestion, what is you strategy if PBS goes down ? What are the key elements of PBS that you backup to get it back on feet as if it was the same PBS (encryption keys and... ) ?
 
  • Like
Reactions: Johannes S and UdoB
ilium007 said:
I got mine running. Restic backing up the PBS chunk store. Works great!
Click to expand...
Did you already tested a restore? Because I would expect simmiliar issues like the ones people had with rclone:
Z

Thread 'Datastore synced with Rclone broken?'

I'm trying to set up a remote sync to a friend's RPI for my Proxmox Backup Server. I've been using this script that uses Rclone with an encrypted remote (Crypt) on a schedule:
Bash: 
#!/bin/bash

# A simple script to back up PBS to Crypt remote

_lock_file=/root/.tertiary_backup.lock

_schedule_time=1:30
_max_duration="7.5h"
_cutoff_mode="soft"

_datastore_dir_name=Primary_Backup
_datastore_dir_path=/mnt/datastore/$_datastore_dir_name
_config_dir=$_datastore_dir_path/host_backup

_remote_user=user
_remote_ip=tailscale-ip
_remote_port=22
_remote_name=Tertiary_Backup_Crypt...
https://forum.proxmox.com/threads/pbs-appears-not-to-write-to-disk.157751/

For all what's worth my recommendation would be to use restic for ProxmoxVE/ProxmoxBackupServer OS backup or large amounts of bulk data (like your media libary or the stuff you have on your NAS) and PBS for VMs/LXCs.
 
  • Like
Reactions: UdoB
Johannes S said:
Did you already tested a restore? Because I would expect simmiliar issues like the ones people had with rclone:
Z

Thread 'Datastore synced with Rclone broken?'

I'm trying to set up a remote sync to a friend's RPI for my Proxmox Backup Server. I've been using this script that uses Rclone with an encrypted remote (Crypt) on a schedule:
Bash: 
#!/bin/bash

# A simple script to back up PBS to Crypt remote

_lock_file=/root/.tertiary_backup.lock

_schedule_time=1:30
_max_duration="7.5h"
_cutoff_mode="soft"

_datastore_dir_name=Primary_Backup
_datastore_dir_path=/mnt/datastore/$_datastore_dir_name
_config_dir=$_datastore_dir_path/host_backup

_remote_user=user
_remote_ip=tailscale-ip
_remote_port=22
_remote_name=Tertiary_Backup_Crypt...
Click to expand...

What issues have people had with an rclone restore? My test restores have worked fine. Been using Restic and PVE/PBS for years now.
 
VartKat said:
Can you tell us about deduplication, as PBS does its deduplication first and restic does his after what's the result ?
Is the restic backup smaller that the PBS one ? Larger ? Same size ?
Click to expand...

Restic does not seem to to make much difference to the already de-duped PBS data, maybe 1GB smaller in Restic for the same data:

Code: 
root@pve01:~# du -s /mnt/backup0/
56074654    /mnt/backup0/


Code: 
➜ restic stats a3899631
repository caecd736 opened (version 2, compression level auto)
[0:00] 100.00%  52 / 52 index files loaded
scanning...
Stats in restore-size mode:
     Snapshots processed:  1
        Total File Count:  94981
              Total Size:  55.056 GiB
 
ilium007 said:
What issues have people had with an rclone restore? My test restores have worked fine. Been using Restic and PVE/PBS for years now.
Click to expand...
Did you even read the links? Their backups got broken since the sync with rclone messed up the consistency of the datastore.
I wouldn't expect that restic behaves different because technically a restic backup and restore of a datastore is a kind of sync too.

What would work (but propably not safe much space) is to use PVEs native backup function (without PBS) to create vzdump files and backup them afterwards with restic.

And as you already figured out: Since PBS and restic both are doing deduplication there isn't much to gain from a combination of both of them.
 
Last edited:
  • Like
Reactions: UdoB
So changing backup strategy to only using PVE backups (no PBS) and using Restic to upload full dump files and use its dedup results in far greater backup sizes which will double cost for cloud stored backups (I'm using Borgbase as a hosted backup repository).

Code: 
b0c5356c  2025-05-23 02:00:08  pve01.sapling  resticprofile  /etc                            50.850 GiB
                                                             /home
                                                             /mnt/pve/backup01
                                                             /root
                                                             /usr/local/bin
                                                             /var/lib/pve-cluster
                                                             /var/lib/pve-cluster/config.db
                                                             /var/lib/vz/template
                                                             /var/spool/cron

aa7a0102  2025-05-23 08:11:23  pve01.sapling  resticprofile  /etc                            99.597 GiB
                                                             /home
                                                             /mnt/pve/backup01
                                                             /root
                                                             /usr/local/bin
                                                             /var/lib/pve-cluster
                                                             /var/lib/pve-cluster/config.db
                                                             /var/lib/vz/template
                                                             /var/spool/cron

Files on disk (local restic repo - this is replicated to BorgBase via 'restic copy xxxx' daily)

Code: 
sudo du -hs /tank/rest-server/pve01.sapling
98G     /tank/rest-server/pve01.sapling

About 1gb of dedup locally.
 
I'll continue to use Restic (with the awesome resticprofile wrapper) to back up the local PBS repo to Borgbase. I am now running a script before a backup (using the resticprofile "run-before" run hook https://creativeprojects.github.io/resticprofile/configuration/run_hooks/index.html) to shutdown PBS is there are no active tasks, otherwise wait until PBS is free (script below). After the Restic backup is complete I then start PBS again using the resticprofile run-after (as well as run-after-fail).

/usr/local/bin/stop-proxmox-backup.sh

Bash: 
/usr/local/bin/stop-proxmox-backup.sh
#!/bin/bash

# Configurable parameters
MAX_ATTEMPTS=12         # Total number of checks before giving up
SLEEP_SECONDS=300       # Wait time between checks in seconds

# Function to check if any tasks are running
has_running_tasks() {
    proxmox-backup-manager task list | grep -q "│ running │"
}

attempt=1
while [ $attempt -le $MAX_ATTEMPTS ]; do
    echo "Attempt $attempt of $MAX_ATTEMPTS: Checking for running tasks..."

    if has_running_tasks; then
        echo "Active tasks detected. Waiting $SLEEP_SECONDS seconds before retrying..."
        sleep $SLEEP_SECONDS
        ((attempt++))
    else
        echo "No running tasks detected. Stopping proxmox-backup service..."
        systemctl stop proxmox-backup.service
        sleep 30
        exit 0
    fi
done

echo "Failed to stop service: Tasks are still running after $MAX_ATTEMPTS attempts."
exit 1

I will do a full PBS repo restore from Borgbase over the next few days to test again.
 
Last edited:
I think in that case it makes more sense to use restic as file-level backup inside the VM than such non-supported hacks who work until they don't. But it's your data so ymmv
 
Ok champ. The chunk store is just folders of fixed size files. Files on disk that are being 1:1 backed up after shutting down the PBS service. How is backing up the PBS datastore files a 'hack'?

What's the alternative? Building a VPS with terabytes or storage and running another PBS server? How does that compare to USD $80/year for 1TB at Borgbase for a hosted Restic rest server?

Hetzner: cheapest VPS USD $5.05/month plus USD $56.32/month for 1TB storage.
Digital Ocean: VPS USD $6.00/month plus USD $100/month 1TB block storage

But yeah, whatever, you do you. I'll stick with my 'hack' thats worked fine for years.
 
Last edited:
ilium007 said:
Ok champ. The chunk store is just folders of fixed size files. Files on disk that are being 1:1 backed up after shutting down the PBS service. How is backing up the PBS datastore files a 'hack'?
Click to expand...

Because there is no guarantee of it's consistency, see linked threads for reference.

ilium007 said:
What's the alternative? Building a VPS with terabytes or storage and running another PBS server?
Click to expand...

netcups vservers are quite affordable ( I pay around 12 Euro for mine including 300 GB extra storage). Inetts cloud PBS service cost 0.02 Euro per GB or 20 Euro per TB.

If this is still to much you could also combine a Hetzner vserver with a cifs attached Hetzber storagebox from the same datacenter. I wouldn't recommend this due to the performance implications of a network data store but it's an option.

Or alternatively ditch vm backups completely for offsite backup and do file-level backups with restic from inside of the vm. Another benefit of that approach would be that in case of a damaged datastore or restic repisitory you would still have the data in a different Format.


ilium007 said:
But yeah, whatever, you do you. I'll stick with my 'hack' thats worked fine for years.
Click to expand...

I can't stop you but I want to give other readers my opinion so they can decide what risks they are willing to take with non-supported setups. I mean support for clozd storage is on PBS Roadmap for years, which for me is an indicator that it's actualky harder to implement than one would like.
I mean if it would be easy to do this with rclone or restic PBS developers would propably already done this because that would adresse an often requested feature for a relative small effort.
But as said: To each their own, I'm quite conerative with my "better safe than sorry"-approach
 
  • Like
Reactions: UdoB
Again, the datastore is just files. Restic chunks, checksums, dedupes, encrypts, copies. I don’t know what you’re so afraid of. What can possibly happen to a bunch of files that are checksummed and verified as exact duplicates?

I’ve ready the forum posts. I’m not using rclone (with crypt backend) so I’m not going to comment on that.

I also do file level Restic backups from within proxmox guests.

I do local proxmox backups to PBS.

Offsite Restic is a last resort recovery option for the entire PBS repo.

It’s worked fine like this for years, and I’ve got a lot of years experience in enterprise devops.

Again, you do you Amigo.
 
Last edited:
You must log in or register to reply here.
Top Bottom