Do I need every empty folder in .chunks for a restore?

[defekt]

Hi folks ,

I am currently in the process of setting up a test pbs, as I would like to use it for backups in the future. I also want to use offsite backups, the current idea is to move everything over via rclone. During a test I noticed that Rclone ignores all empty folders in .chunks by default but there are thousands. Can I ignore the empty folders or do I need them for a later restore of the VM's?

 

[Johannes S]

Yikes, the .chunks folder contains the actual data of your backups and PBS except it's not modified by any other Tool.
Beside this rclone is known to break datastores, so don't do this.

J

Post in thread 'Backup PBS to cloud provider with incremental chunks'

Nov 25, 2024

Hello,

this most likely won't work or in other words: You can do this, but then your backups will get broken which explains why nobody uses this:
https://forum.proxmox.com/threads/datastore-synced-with-rclone-broken.154709/
https://forum.proxmox.com/threads/pbs-appears-not-to-write-to-disk.157751/

You have following options:

• Using a PBS cloud provider like this one: https://www.tuxis.nl/de/proxmox-backup-server/ Tuxis don't sell to private customers outside the Netherlands or in general to customers outside the EU but they might have resellers (I know of at...

• Johannes S

• 

 

[defekt]

Well f+ck me then I guess.

How to solve Offsite backups then? Besides your solution with an way to expensive hoster.

 

[Johannes S]

Well f+ck me then I guess.

How to solve Offsite backups then? Besides your solution with an way to expensive hoster.

How large is your datastore? I pay around 12 Euro for my netcup vserver + 250 GB additional storage space. It's a Debian I installed PBS on.
Another option would be to buy two external usb drives and use them as removable datastores in PBS. If you change between them on a regular base and store one of it outside (e.G. in your office or a friends/family member place) this would be enough. Or you have a Mini-PC with PBS in your family member/friends place.
Or (like already said in my older post): Use PBS only for your local backups and use Proxmox native vzdump function (vma-archives) together with cloud storage. This works fine but doesn't have deduplication. Depending on your data amount this doesn't need to be a problem. Hetzners storagebox is around four Euro for 1 TB and eleven Euro for 5 TB backup space: https://www.hetzner.com/de/storage/storage-box/

Personally I don't mind paying around 20 euros per month for peace of mind.

And the cloud pbs providers (tuxis.nl, inett, cloud-pbs etc) ain't too expensive either, their typical price is around 20 Euro for one TB storage space.

 

[TurboActive]

Well f+ck me then I guess.

How to solve Offsite backups then? Besides your solution with an way to expensive hoster.

If you wanna use rclone you can use the --create-empty-src-dirs --s3-directory-markers flags to create empty directory markers. The version of rclone available via apt is 1.60 and does not support this feature. You can however download newer versions from rclone.org that support this feature. The only thing you need then is a s3 compatible storage provider. The uploading of the 65,536 takes a while however and likewise when you want to download the data this is gonna take a while. To speed up the syncing you can run

find .chunks -type d -empty -print0 | tar --null -cvf empty_directories.tar --files-from=-

inside your vm directory. This will create a tar archive with all empty directories in .chunks. Now you can upload it using rclone without the flags. Then when you need to restore the offsite backup you simply run

tar -xf empty_directories.tar

inside the vm directory to restore the empty directories

 

[Johannes S]

If you wanna use rclone you can use the --create-empty-src-dirs --s3-directory-markers flags to create empty directory markers. The version of rclone available via apt is 1.60 and does not support this feature. You can however download newer versions from rclone.org that support this feature. The only thing you need then is a s3 compatible storage provider. The uploading of the 65,536 takes a while however and likewise when you want to download the data this is gonna take a while. To speed up the syncing you can run

find .chunks -type d -empty -print0 | tar --null -cvf empty_directories.tar --files-from=-

inside your vm directory. This will create a tar archive with all empty directories in .chunks. Now you can upload it using rclone without the flags. Then when you need to restore the offsite backup you simply run

tar -xf empty_directories.tar

inside the vm directory to restore the empty directories

Did you test your procedure including restore?