New Proxmox as a replacement for ESXi

[Johannes S]

Im starting to see how powerful it is. Im really liking it. I might have missed it but is there a guide to mirror the VMs to say Synology or offsite storage like backblaze?

S3 is not supported at the moment, allthough it might work with a third-party-hack. I wouldn't do this tbh.
I described the options in an earlier post:

J

Post in thread 'Backup PBS to cloud provider with incremental chunks'

Nov 25, 2024

Hello,

this most likely won't work or in other words: You can do this, but then your backups will get broken which explains why nobody uses this:
https://forum.proxmox.com/threads/datastore-synced-with-rclone-broken.154709/
https://forum.proxmox.com/threads/pbs-appears-not-to-write-to-disk.157751/

You have following options:

• Using a PBS cloud provider like this one: https://www.tuxis.nl/de/proxmox-backup-server/ Tuxis don't sell to private customers outside the Netherlands or in general to customers outside the EU but they might have resellers (I know of at...

• Johannes S

• 

Basically the best option is to use a pbs as a service provider ( like cloud-pbs, tuxis.nl or inett ) or rent a vserver to run pbs.
Using Rclone for sycing to s3 targets is known to break datastores.

 

[LerryV2]

if I can just push it to Synology from PBS then I can move it to backblaze after that. Ill check those links out and see if I can get that working, thanks.

I was able to connect to Synology via a NFS share and a scheduled backup via PVE.

 

[Johannes S]

if I can just push it to Synology from PBS then I can move it to backblaze after that. Ill check those links out and see if I can get that working, thanks.

if I can just push it to Synology from PBS then I can move it to backblaze after that. Ill check those links out and see if I can get that working, thanks.

I was able to connect to Synology via a NFS share and a scheduled backup via PVE.

This is possible yes, but not recommended either. At least it doesn't break data. Network shares are not known for great performance though with larger data sets. One guy did some benchmarking on it:
https://forum.proxmox.com/threads/datastore-performance-tester-for-pbs.148694/

Please note the discussion on the validity of his approach (part of Proxmox developers had some critical points in this thread: https://forum.proxmox.com/threads/developer-question-about-chunk-folder.148167/ ), but imho the basic picture is clear: Network shares are not a good idea for saving large amount of data with PBS, even on a local network. Over a WAN (like some people do) it's a no for enterprise usage.

 

[LerryV2]

This is possible yes, but not recommended either. At least it doesn't break data. Network shares are not known for great performance though with larger data sets. One guy did some benchmarking on it:
https://forum.proxmox.com/threads/datastore-performance-tester-for-pbs.148694/

Please note the discussion on the validity of his approach (part of Proxmox developers had some critical points), but imho the basic picture is clear: Network shares are not a good idea for saving large amount of data with PBS, even on a local network. Over a WAN (like some people do) it's a no for enterprise usage.

Even if your just using it to backup an VM? Seems like if it takes a snapshot and it backs it up, as long as Im not taking a snap faster then it takes to transfer over it shouldnt really matter how long it takes as long as it gets there in one piece. Ill read what the tests say, Im curious.

 

[Johannes S]

Even if your just using it to backup an VM? Seems like if it takes a snapshot and it backs it up, as long as Im not taking a snap faster then it takes to transfer over it shouldnt really matter how long it takes as long as it gets there in one piece. Ill read what the tests say, Im curious.

With only one VM it's most likely not a problem the trouble starts with large data sets since PBS works by splitting it's data in a lot of small files (called chunks) for deduplication, so after an initial backup of the chunks contents only new or changed data is uploaded:
https://pbs.proxmox.com/docs/technical-overview.html

Just you get an idea this is from a log of my latest garbage collection job:

2025-01-11T02:00:25+01:00: Original data usage: 19.087 TiB
2025-01-11T02:00:25+01:00: On-Disk usage: 199.521 GiB (1.02%)
2025-01-11T02:00:25+01:00: On-Disk chunks: 157025
2025-01-11T02:00:25+01:00: Deduplication factor: 97.96
2025-01-11T02:00:25+01:00: Average chunk size: 1.301 MiB

I have around 390 snapshots from the last three months after setting up the PBS. Together they amount to around 19TB if restored. Thanks to deduplication they are only around 200 GB on disk and compromised of 157025 chunks. Each garbage collection will read and alter the metadata of each chunk to decide which chunks can be removed because no snapshot reference it anymore. Now my environment isn't even big (homelab), but I would expect that even more data and thus chunks would be present in a corporate or enterprise environment (way more lxcs and vms, thus more snapshots over a longer time , thus more chunks to work with).

PBS is phantastic but it's really not made for network file systems like CIFS or NFS. The mentioned benchmark tool is quite enlightening since it's developer used it for benchmarks on the local machine: He set up several storages in different filesystems and NFS/CIFS shares, all on the same machine, so his tool works with localhost. With other words the network performance wouldn't be an issue. The only differences would be due to the nature of the used way to access the storage.

 

[LerryV2]

Then what is a good way to push the backups off the server onto backblaze? We have 20 or so VMs.

 

[Johannes S]

Then what is a good way to push the backups off the server onto backblaze? We have 20 or so VMs.

At the moment there isn't really a good option sadly for this specific goal. S3 is on the Roadmap for some years but afik there is no work on it right now. Your best bet would be to run an offsite PBS on a virtual or dedicated server or going with a PBS hosting Provider like tuxis.nl or Inett. Then you could use PBS integrated sync functions for syncing the backups between the PBS instances. Maybe office@proxmox.com knows of such a company in your country.

Another Option might be to use zfs send/receive to replicate the datastore to an offsite location e.g. a TrueNAS/FreeBSD server in a data center . Obviouvsly this would need ZFS on the datastore and the remote server. rsync.net provides zfs as a service: https://www.rsync.net/products/zfsintro.html

 

[LnxBil]

Basically the best option is to use a pbs as a service provider ( [...] inett )

Nice that the word about inett is already spreading

Seems like if it takes a snapshot and it backs it up

Snapshots are not part of the backup. Take a backup instead of a snapshot if you want a kind of a snapshot in your backup. (again problems with the terminology here. Snapshot on PVE is not the same snapshot in PBS)

Then what is a good way to push the backups off the server onto backblaze?

I don't know what backblaze is, but we use PBS sychronization and ZFS send/receive to get our primary PBS send to other locations. Of course everything encrypted. This works like a charm and @Johannes S already mentioned this twice.
It makes no sense to me to synchronize just the files if the underlying ZFS can already do the heavy litfting. You will not be able to create a consistent copy with rsync or rclone if you have constant backups running on your PBS (which is very fast and very easy to do), you could hack together a zfs snapshot method, yet this is totally up to you.

Another Option might be

a third option could be to use the new "external disk" feature of the newest PBS incarnation in order to replicate the data to NFS and only store one copy. I haven't tried this, yet I could imagine it could work.

a fourth option could be to generate a synthetic full backup of a PBS VM snapshot and store it on the NFS share, yet I don't know if you can create/extract such a vzdump directly from PBS.