Proxmox backup VM over object storage B2 / S3

pftech

Member
Nov 15, 2012
12
2
23
Italy - Torino
Hi there,
I'm been working finding a way to backup VM over an object storage like S3 or B2 Backblaze...
I've finally found a solution that is quite stable and perform well.

If someone is interested I'm doing in this way:
- VM backup over local storage
- made a script that with rclone move backup at the end of the process (using script hookup on vzdump.conf) to the object storage (I'm using B2 Backplace that perform very fast but it works perfectly over S3 compatible, tested over Scaleway).
- delete VM backup on local storage to free disk space

Then I've done a script running at cron weekly that delete in the object storage, oldest backup, keeping just the most recent ones.

What is nice is that I can restore a VM directly mounting the object storage using rclone mount or s3fs (rclone mount is better in performance) in a local folder than mapping in Proxmox storage directly (as directory) and restore which backup I need to, without copying backup file locally first. Using a 1 Gbps internet connection I can restore directly from B2 Backblaze at more than 500 Mbps speed! That's pretty fast for an object storage.

If someone needs help or wants give me suggestions to improve this idea, please let me know!

Best regards,
Paolo
 
  • Like
Reactions: devilkin and boopzz

pftech

Member
Nov 15, 2012
12
2
23
Italy - Torino
Hi Tobi,
I'm using this script for moving local backup, after they're done, over B2, directly with rclone, without mounting, since it's even better in performance!

if [ "$1" == "backup-end" ]; then
rclone move /var/lib/vz/dump/ backblaze:bucket-name/dump
fi

You should run this script adding this line at the end of /etc/vzdump.conf :
script: /root/move_backup_b2.sh

(that's your file with the script above)

Now I'm testing some sort of script that find oldest file than xx days over B2 and then remove them!


Regards,
Paolo
 

thb

New Member
Apr 8, 2020
2
0
1
24
Hi pftech, thanks for the script - works great! I figured out a nice way to automatically remove the backups - you can use B2's built-in lifecycle rules.

In my case, I have daily backups running every 24 hours, so I set up the following lifecycle rules:


File Path
fileNamePrefix
Days Till Hide
daysFromUploadingToHiding
Days Till Delete
daysFromHidingToDeleting
dump/71

This means every file inside of the dump/ folder will be hidden after 7 days, and then deleted a day after that. So I essentially have a rolling window of the last week's backups available.

Hope this helps!
 

pftech

Member
Nov 15, 2012
12
2
23
Italy - Torino
Hi pftech, thanks for the script - works great! I figured out a nice way to automatically remove the backups - you can use B2's built-in lifecycle rules.

In my case, I have daily backups running every 24 hours, so I set up the following lifecycle rules:


File Path
fileNamePrefix
Days Till Hide
daysFromUploadingToHiding
Days Till Delete
daysFromHidingToDeleting
dump/71

This means every file inside of the dump/ folder will be hidden after 7 days, and then deleted a day after that. So I essentially have a rolling window of the last week's backups available.

Hope this helps!

Thanks! I'll take a look if it works! I thought about that before, but I didn't think that it should work to delete old files...
I done a backup every week since VM files are too large and it takes about 2 days over 1 Gbps connection (backup files about 2 Tb!). Now I'm searching a way to transfer backups over B2, without using rclone, since it takes too long to start transferring large files (about 6-8 hours before it starts transferring with a backup file of 900 Gb!).

Regards!
 

thb

New Member
Apr 8, 2020
2
0
1
24
Thanks! I'll take a look if it works! I thought about that before, but I didn't think that it should work to delete old files...
I done a backup every week since VM files are too large and it takes about 2 days over 1 Gbps connection (backup files about 2 Tb!). Now I'm searching a way to transfer backups over B2, without using rclone, since it takes too long to start transferring large files (about 6-8 hours before it starts transferring with a backup file of 900 Gb!).

Regards!

Yup, the trick is that if you only set "daysFromHidingToDeleting" then it will only remove previous versions of the same file. But if you set "daysFromUploadingToHiding", then it will hide it even if there's only one version, and from there you can make it delete the hidden files automatically.

Docs (where I found this out): https://www.backblaze.com/b2/docs/lifecycle_rules.html

Also my backups are only around ~50GB, which is why I do it daily (in the middle of the night on my 1Gbps link). Impressive how much you're backing up :) But B2 is so cheap that it makes sense, heh.
 

devilkin

New Member
May 11, 2020
22
3
3
Hi there,
I'm been working finding a way to backup VM over an object storage like S3 or B2 Backblaze...
I've finally found a solution that is quite stable and perform well.
Do you have the script available anywhere? I'm currently using duplicati to send it off to B2, but it feels like your solution is way better.
 
Mar 19, 2018
146
20
38
Old thread, but I thought I'd chime in, although I only have a small amount of info to offer here.

I had tried to use b2's Lifecycle policy myself to delete old files, and failed. In the end I wrote my own script using the b2 cli tool to do so.

So the info @thb provided about how to successfully do it using Lifecycle rules is really interesting - thanks.

However, b2's Lifecycle rules can't take account of the fact that you may have a stored local backup of a file that's also being synced (not just moved) to b2. e.g. in my case I have some backups of VM templates that are stored locally and never deleted, which are also stored on b2 via the sync process.

If you allow b2's Lifecycle rules to delete a file after X days, then the next time you do a sync you'll be syncing that local backup to b2 again. So every X days, you'll be re-syncing that locally stored file, over and over. Sure, if it is only small then it makes little difference. But if it is huge, it makes a big difference. Hours and hours of difference, potentially.

Another thing I wanted to comment on was speed. @pftech, I use s3cmd to sync files rather than rclone.

Now rclone looks rather interesting as an alternative to using s3cmd. Much more powerful. Much more flexible. AND it is a simple binary, so I could potentially run it directly on a node rather than in a Container on a node (unlike, at least previously, s3cmd which is a whole mess of python).

What I found with s3cmd was that it was very flexible in terms of allowing you to set how many threads, and chunk size, and bandwidth limit, and by experimenting with all three settings I was able to get good transfer speeds, with minimal pre-transfer delay, without causing too high a load on the server. It took days of experimenting to get a result I was reasonably happy with. Some of my backups are 500G, others are 100 to 200G, and it was those larger ones that needed me to find the best settings. A larger chunk size than the default was critical.

The native b2 cli tool wasn't as flexible, and I was getting terrible transfer speeds and the load was way to high, no matter what I did. Maybe more experimentation would have helped, but I spent way too much time over it to start with.

I can see rclone has various options in terms of chunk sizes and bandwidth limits, but threads seem to be related to downloads as opposed to uploads? I'll have to investigate further, and maybe do some experiments to see if I can duplicate what I have in s3cmd + b2 with rclone.
 

pftech

Member
Nov 15, 2012
12
2
23
Italy - Torino
Hi Faris,
I must admit that I don't use B2 to store Proxmox backup anymore! As soon as arrived Proxmox Backup Server, I'm using it definitely... it's years ahead and even cheaper (I'm using 2 storage servers with PBS installed on).

Anyway rclone has almost the same tweaks that you can do with s3cmd, I remember that I've tried also s3cmd but I found rclone more flexible and faster in transfer too. The problem that I've abandoned B2 is that files retention is too long and with the size of our backup, I'd reached about 400 Tb in about 30 days and it's definitely too much!!!

With Proxmox Backup Server and deduplication, you save tons of space and it's really fast!!! I've abandoned B2 more than an year ago!

Best regards
 
Mar 19, 2018
146
20
38
With Proxmox Backup Server and deduplication, you save tons of space and it's really fast!!! I've abandoned B2 more than an year ago!

Thanks @pftech. I'm looking at PBS at the moment, but using Tuxis as a remote PBS service, as opposed to running my own. The cost of even a single dedicated PBS server with enough storage is far greater than the cost of b2 for me currently (we have only 10TB of data stored). I imagine that even though Tuxis is four times more expensive than b2 per GByte, deduplication and incremental backups will result in a similar cost to full backups all the time to b2, and be much faster (and of course individual file restore!!!).

Currently, my concern with PBS is that due to the way the code works at the moment, any problems on the network between my servers and Tuxis, or any problems at the Tuxis end, can result in unacceptable VM problems (e.g. total freeze requiring hard reboot). Until this can be resolved, I think I'm stuck with b2. But your comments on rclone make me want to look into it asap, until the amazing genius devs can persuade PBS to play nice when there are network or PBS server performance issues.
 
Mar 19, 2018
146
20
38
What is Tuxis? Is it a storage service?
Yup.

The Beta was discussed in the PBS section of the forum (sticky, first topic in the list).

The release service itself is here: https://www.tuxis.nl/en/proxmox-backup-server/

They offer 150Gb free so you can test it out. I have been doing so, and it has worked perfectly and painlessly.

My concerns about what would happen if there's a network issue or a problem at their end during a backup remain, however.
It is essentially a roadblock for me. I cannot risk a production VM freezing during a backup process.
Until PBS is refined in whatever way is needed to prevent this from happening, I cannot risk using it :-(
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!