Proxmox backup VM over object storage B2 / S3

[pftech]

Hi there,
I'm been working finding a way to backup VM over an object storage like S3 or B2 Backblaze...
I've finally found a solution that is quite stable and perform well.

If someone is interested I'm doing in this way:
- VM backup over local storage
- made a script that with rclone move backup at the end of the process (using script hookup on vzdump.conf) to the object storage (I'm using B2 Backplace that perform very fast but it works perfectly over S3 compatible, tested over Scaleway).
- delete VM backup on local storage to free disk space

Then I've done a script running at cron weekly that delete in the object storage, oldest backup, keeping just the most recent ones.

What is nice is that I can restore a VM directly mounting the object storage using rclone mount or s3fs (rclone mount is better in performance) in a local folder than mapping in Proxmox storage directly (as directory) and restore which backup I need to, without copying backup file locally first. Using a 1 Gbps internet connection I can restore directly from B2 Backblaze at more than 500 Mbps speed! That's pretty fast for an object storage.

If someone needs help or wants give me suggestions to improve this idea, please let me know!

Best regards,
Paolo

 

[pftech]

Hi Tobi,
I'm using this script for moving local backup, after they're done, over B2, directly with rclone, without mounting, since it's even better in performance!

if [ "$1" == "backup-end" ]; then
rclone move /var/lib/vz/dump/ backblaze:bucket-name/dump
fi

You should run this script adding this line at the end of /etc/vzdump.conf :
script: /root/move_backup_b2.sh

(that's your file with the script above)

Now I'm testing some sort of script that find oldest file than xx days over B2 and then remove them!


Regards,
Paolo

 

[thb]

Hi pftech, thanks for the script - works great! I figured out a nice way to automatically remove the backups - you can use B2's built-in lifecycle rules.

In my case, I have daily backups running every 24 hours, so I set up the following lifecycle rules:

File Path fileNamePrefix	Days Till Hide daysFromUploadingToHiding	Days Till Delete daysFromHidingToDeleting
dump/	7	1

This means every file inside of the dump/ folder will be hidden after 7 days, and then deleted a day after that. So I essentially have a rolling window of the last week's backups available.

Hope this helps!

 

[pftech]

Hi pftech, thanks for the script - works great! I figured out a nice way to automatically remove the backups - you can use B2's built-in lifecycle rules.

In my case, I have daily backups running every 24 hours, so I set up the following lifecycle rules:

File Path fileNamePrefix	Days Till Hide daysFromUploadingToHiding	Days Till Delete daysFromHidingToDeleting
dump/	7	1

This means every file inside of the dump/ folder will be hidden after 7 days, and then deleted a day after that. So I essentially have a rolling window of the last week's backups available.

Hope this helps!

Thanks! I'll take a look if it works! I thought about that before, but I didn't think that it should work to delete old files...
I done a backup every week since VM files are too large and it takes about 2 days over 1 Gbps connection (backup files about 2 Tb!). Now I'm searching a way to transfer backups over B2, without using rclone, since it takes too long to start transferring large files (about 6-8 hours before it starts transferring with a backup file of 900 Gb!).

Regards!

 

[thb]

Thanks! I'll take a look if it works! I thought about that before, but I didn't think that it should work to delete old files...
I done a backup every week since VM files are too large and it takes about 2 days over 1 Gbps connection (backup files about 2 Tb!). Now I'm searching a way to transfer backups over B2, without using rclone, since it takes too long to start transferring large files (about 6-8 hours before it starts transferring with a backup file of 900 Gb!).

Regards!

Yup, the trick is that if you only set "daysFromHidingToDeleting" then it will only remove previous versions of the same file. But if you set "daysFromUploadingToHiding", then it will hide it even if there's only one version, and from there you can make it delete the hidden files automatically.

Docs (where I found this out): https://www.backblaze.com/b2/docs/lifecycle_rules.html

Also my backups are only around ~50GB, which is why I do it daily (in the middle of the night on my 1Gbps link). Impressive how much you're backing up But B2 is so cheap that it makes sense, heh.

 

[devilkin]

Hi there,
I'm been working finding a way to backup VM over an object storage like S3 or B2 Backblaze...
I've finally found a solution that is quite stable and perform well.

Do you have the script available anywhere? I'm currently using duplicati to send it off to B2, but it feels like your solution is way better.

 

[Faris Raouf]

Old thread, but I thought I'd chime in, although I only have a small amount of info to offer here.

I had tried to use b2's Lifecycle policy myself to delete old files, and failed. In the end I wrote my own script using the b2 cli tool to do so.

So the info @thb provided about how to successfully do it using Lifecycle rules is really interesting - thanks.

However, b2's Lifecycle rules can't take account of the fact that you may have a stored local backup of a file that's also being synced (not just moved) to b2. e.g. in my case I have some backups of VM templates that are stored locally and never deleted, which are also stored on b2 via the sync process.

If you allow b2's Lifecycle rules to delete a file after X days, then the next time you do a sync you'll be syncing that local backup to b2 again. So every X days, you'll be re-syncing that locally stored file, over and over. Sure, if it is only small then it makes little difference. But if it is huge, it makes a big difference. Hours and hours of difference, potentially.

Another thing I wanted to comment on was speed. @pftech, I use s3cmd to sync files rather than rclone.

Now rclone looks rather interesting as an alternative to using s3cmd. Much more powerful. Much more flexible. AND it is a simple binary, so I could potentially run it directly on a node rather than in a Container on a node (unlike, at least previously, s3cmd which is a whole mess of python).

What I found with s3cmd was that it was very flexible in terms of allowing you to set how many threads, and chunk size, and bandwidth limit, and by experimenting with all three settings I was able to get good transfer speeds, with minimal pre-transfer delay, without causing too high a load on the server. It took days of experimenting to get a result I was reasonably happy with. Some of my backups are 500G, others are 100 to 200G, and it was those larger ones that needed me to find the best settings. A larger chunk size than the default was critical.

The native b2 cli tool wasn't as flexible, and I was getting terrible transfer speeds and the load was way to high, no matter what I did. Maybe more experimentation would have helped, but I spent way too much time over it to start with.

I can see rclone has various options in terms of chunk sizes and bandwidth limits, but threads seem to be related to downloads as opposed to uploads? I'll have to investigate further, and maybe do some experiments to see if I can duplicate what I have in s3cmd + b2 with rclone.

 

[pftech]

Hi Faris,
I must admit that I don't use B2 to store Proxmox backup anymore! As soon as arrived Proxmox Backup Server, I'm using it definitely... it's years ahead and even cheaper (I'm using 2 storage servers with PBS installed on).

Anyway rclone has almost the same tweaks that you can do with s3cmd, I remember that I've tried also s3cmd but I found rclone more flexible and faster in transfer too. The problem that I've abandoned B2 is that files retention is too long and with the size of our backup, I'd reached about 400 Tb in about 30 days and it's definitely too much!!!

With Proxmox Backup Server and deduplication, you save tons of space and it's really fast!!! I've abandoned B2 more than an year ago!

Best regards

 

[Faris Raouf]

With Proxmox Backup Server and deduplication, you save tons of space and it's really fast!!! I've abandoned B2 more than an year ago!

Thanks @pftech. I'm looking at PBS at the moment, but using Tuxis as a remote PBS service, as opposed to running my own. The cost of even a single dedicated PBS server with enough storage is far greater than the cost of b2 for me currently (we have only 10TB of data stored). I imagine that even though Tuxis is four times more expensive than b2 per GByte, deduplication and incremental backups will result in a similar cost to full backups all the time to b2, and be much faster (and of course individual file restore!!!).

Currently, my concern with PBS is that due to the way the code works at the moment, any problems on the network between my servers and Tuxis, or any problems at the Tuxis end, can result in unacceptable VM problems (e.g. total freeze requiring hard reboot). Until this can be resolved, I think I'm stuck with b2. But your comments on rclone make me want to look into it asap, until the amazing genius devs can persuade PBS to play nice when there are network or PBS server performance issues.

 

[pftech]

What is Tuxis? Is it a storage service?

 

[Faris Raouf]

What is Tuxis? Is it a storage service?

Yup.

The Beta was discussed in the PBS section of the forum (sticky, first topic in the list).

The release service itself is here: https://www.tuxis.nl/en/proxmox-backup-server/

They offer 150Gb free so you can test it out. I have been doing so, and it has worked perfectly and painlessly.

My concerns about what would happen if there's a network issue or a problem at their end during a backup remain, however.
It is essentially a roadblock for me. I cannot risk a production VM freezing during a backup process.
Until PBS is refined in whatever way is needed to prevent this from happening, I cannot risk using it :-(

 

[coxmcse]

I have had much success with using a synology appliance with nfs mounts for proxmox backup storage. With the stored backups, then ship the backups off site encrypted to s3 bucket with https://www.synology.com/en-us/dsm/7.1/software_spec/glacier_backup

 

[penfold07]

Thank you for this script Faris. We are also using an adaption of it at my work. I've modified it a bit though for a couple of reasons.
1. I still want to have local copies of backups as well as Backblaze and 2. We have archival vm's we need to keep but won't be on much so won't change a lot - but they are large - multiple TB's so don't want to copy them to Backblaze and keep them local only.

What I do is have two backup locations locally on the Hypervisor and two separate sets of backups. One is moved to Backblaze using the rclone move and the other stays local only.