Backup PBS to cloud provider with incremental chunks

[thimplicity]

Hi,
I have not found a solution online for this. I would like to backup my PBS backups to OneDrive and be able to only upload incremental changes just as PBS does it. The execution does not seem to be a problem, I can upload the files (incl. chunks) to OneDrive via rclone. I woud like to do that so that I do not have to upload full backups every time if I do not use PBS but a standard proxmox backup job. At the same time, this solution does not seem to be used often, at least on the usual-suspects sites I checked.

Is uploading the chunks not a working solution?

 

[Johannes S]

Hello,

this most likely won't work or in other words: You can do this, but then your backups will get broken which explains why nobody uses this:
https://forum.proxmox.com/threads/datastore-synced-with-rclone-broken.154709/
https://forum.proxmox.com/threads/pbs-appears-not-to-write-to-disk.157751/

You have following options:

• Using a PBS cloud provider like this one: https://www.tuxis.nl/de/proxmox-backup-server/ Tuxis don't sell to private customers outside the Netherlands or in general to customers outside the EU but they might have resellers (I know of at least one German reseller). And they have a free tier for 150 GB so that might be enough for you.
• Get a Debian vserver from a hoster of your choice (Hetzner, netcup, catando, whatever) and setup PBS on it. This is surprising affordable I'm paying under ten Euro at the moment for a vserver at Netcup + additional storage for the PBS datastore.
• Ditch PBS all together and do the regular vzdump backups of Proxmox. Backup them with restic, borgbackup or some other simmiliar tool to the cloud provider of your choice. Note: In this case you might be better off in splitting up your vms/lxc (and thus their backups):
  • The VM/LXC OS lives on one virtual disk and is backuped with Proxmox vzdump or PBS
  • The data lives on another virtual disk and is backuped to a cloud storage with restic, borgbackup

The last option might be of interest for you since restic has a rclone backend enabling it to backup to everything supported by rclone.

I myself doing something of a mixed approach: My notebook, my NAS (which is a VM on Proxmox VE) data and the Proxmox hosts are backuped with restic to a hetzner storagebox and a external disk drive. My VMs and containers are backuped to a local PBS which is synced to the remote PBS on Netcup. At the moment I don't split between the OS and data of my VMs (except the NAS) and containers so it's less complicated to oversee everything. But I might if I run out of space at the vserver (storagebox is cheaper). The benefit is, that I can grow the storagespace on both places in small steps (grow as you go) if I need more space without breaking the bank. If at some point the storagebox and netcup together (even with a split between VMs OS and data) cost more than a dedicated server I will ditch them and replace with a dedicated server. If my future budget woudn't allow this I would propably ask some friends whether I might put a low-power-Server at their place and use that for my backups.

Hope that helps, Johannes.

 

[thimplicity]

Hello,
...

Hope that helps, Johannes.

Thanks for the very thorough answer. I think I will go for a split approach:

• Local backups with PBS - one directly on my homelab server with a sync to the PBS backup server twice a week. This way I have incremental backups.
• Remote backups with vzdump that will be uploaded to OneDrive. In case the local stuff taps out.

@Johannes S - Why did you use restic and not rclone or kopia?

 

[Johannes S]

Thanks for the very thorough answer. I think I will go for a split approach:

• Local backups with PBS - one directly on my homelab server with a sync to the PBS backup server twice a week. This way I have incremental backups.

You could and should still have a more frequent backup to PBS due to deduplication you can have a lot of snapshots without needing much space.

• Remote backups with vzdump that will be uploaded to OneDrive. In case the local stuff taps out.

If this fit's your need that's fine but be aware that they will need more storage than the pbs backups.

@Johannes S - Why did you use restic and not rclone or kopia?

I read in a Blog on it and liked the possibility to do cloud backups without the need for a dedicated server ( sftp is enough), ease of restore with fusermount and that I can have a lot of snapshots in relative small cloud storagespace thanks to deduplication.

 

[thimplicity]

I read in a Blog on it and liked the possibility to do cloud backups without the need for a dedicated server ( sftp is enough), ease of restore with fusermount and that I can have a lot of snapshots in relative small diskspace thanks to deduplication.

Looks like snapshots is the main differentiating factor between restic and rclone

 

[Johannes S]

Looks like snapshots is the main differentiating factor between restic and rclone

Not just that, the whole approach how it stores data everything is quite different. There are some older talks and podcasts with the main developer ( most are German this one is in English: https://changelog.com/podcast/434 ). It results actually in a lot of simmilarities with Proxmox backup server which might explain why I prefer both for my backups.

I'm not alone: CERN uses their own customized version of restic as backend for their internal backup software cback. They did some Papers on it with quite impressive statistics how much data they process with it. Higly recommended reading!

 

[thimplicity]

Not just that, the whole approach how it stores data everything is quite different. There are sone older taljs and podcasts with the main developer ( most are German this one is in English: https://changelog.com/podcast/434 ). It results actually in a lot of simmilarities with Proxmox backup server which might explain why I prefer both for my backups.

Thanks - will check it out. Maybe I will do restic instead of rclone to OneDrive then. Another change in approach, probalby the 5th in 3 days, lol

 

[thimplicity]

You could and should still have a more frequent backup to PBS due to deduplication you can have a lot of snapshots without needing much space.

Trying to save some power with not having the backup server running 24/7

My VMs and containers are backuped to a local PBS which is synced to the remote PBS on Netcup.

Can you share how this setup works exactly? Do you run that PBS yourself on a VPS?

 

[Johannes S]

Trying to save some power with not having the backup server running 24/7

Ok, that's understandable. For me it would be highly unpractical that I would have to turn on the PBS in case I want to restore a backup.

So if energy costs are of concern I would do something like this:

• Setup a VM on your ProxmoxVE with ProxmoxBackupServer, create as much backups as you like. Create a user and apikey for the physical PBS. It's permissions should be set that the physical and offsite PBS can pull backups but NOT remove or otherwise alter them so in case of a ransomware or hacker attack the damage is limited as much as possible: https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery
• Create a sync job on the physical PBS, which pulls the backups from the PBS VM: https://pbs.proxmox.com/docs/managing-remotes.html
  Setup permissions that PVE, PBS VM and offsite PBS can restore/pull Backups from the physical PBS but not alter or remove them
• Create a cronjob/systemd event on PVE which wakes up the physical PBS via WakeOverLan some minutes before the syncjob starts (5-15 minutes should be enough depending on the startup time). Create a cronjob/systemd event on the physical PBS which turn it off after finishing the sync- and any other jobs (if you schedule them right and your disks are not too slow one to two hours should be enough)
• Setup a offsite pbs which pull backups from the PBS VM. Setup permissions that PVE and the PBS VM/physical PBS can pull or restore backups but not overwrite or remove them.

On the other hand: If your energy costs are really high due to your homelab it might actually be cheaper to rent a dedicated server right from the start. I rembember some guy in r/homelab on reddit sold his homelab hardware and switched to a hetzner dedicated server for exactly that reason.

Can you share how this setup works exactly? Do you run that PBS yourself on a VPS?

Exactly! Since tuxis free tier is limited to 150 GB ( my PBS datastore was right at 200 GB from the start thanks to some windows vm created from the OEM install on the MINI-PCs I built my cluster on) and they don't sell to private customers outside the Netherlands I needed another solution. I noticed that netcups vservers are quite affordable (I pay around 9 Euro für a vserver with 250 GB storage plus around 250 GB additional object storage). They don't allow to install your own Linux distribution though, you must pick one of their templates. One of them was Debian so I used that template and afterwards installed PBS manually like written in the manual:
https://pbs.proxmox.com/docs/installation.html#install-proxmox-backup-server-on-debian

Afterwards I configured everything to my needs. Most important was to setup a strict firewall (I used ufw as iptables frontend) and a wireguard VPN, so my servers (including the vservers) can only communicate via VPN with eachother. The firewall is configured that only ssh connections are allowed without VPN (of course I can also use ssh inside the VPN) and fail2ban takes care of bots/script kiddes trying to guess my password for ssh. I also disabled password authentification for ssh in favour of publickey authentification. The PBS on the vserver can pull my backups from my local PBS but not remove anything on it. And vice versa my local PBS and ProxmoxVE can pull or restore backups from the offsite server but not remove anything.
My schedule for keeping the backups is like this: On my local PBS I like to have more backups, so I can restore a VM or container if I mess something up. The schedule is like this at the moment: 6-12 Backups per VM/container per day, 30 daily backups and twelve monthly (so I have one year of history, at the moment I'm in my third month after first install.). For the sync I created a custom nameset "remote", a local syncjob sync every backup snapshots to this namespace. This namespace has a prune job, which take care, that it contains only daily snapshots. These daily snapshots are then pulled by my offsite pbs. On the offsite PBS the prune schedule is 30 daily backups, twelve monthly. The idea is, that in case of an emergency I can still restore from an earlier date, but I don't need to sync every hourly backup for it.
As additional emergency backup I do a weekly regular vzdump (Proxmox native backup function) to an external drive and my hetzner storagebox. These backups are not so space-efficent like PBS thus they are limited and for emergency use only.
I hope I havn't lost you during my ramblings?

 

[thimplicity]

@Johannes S - Thanks so much!