Hetzner Storage Box as datastore

N

Nanobug

Member
May 14, 2021
22
3
8
35
Hello Proxmox Community,

As my offsite backup solution, I decided to go for a Hetzner Storage Box and mount it on my old Intel NUC.
I've used Rclone for it, create a systemd for it to auto mount it.
I also tried using CIFs for it with the same result.
I can see the .chunks and .lock directoy and file, and I can create files and folders from the terminal.

I'm running version 3.3.3 on my PBS server right now.

But when I access it in the WebUI it says Datastore is not available:
1743677364766.png

And from the Dasboard it's Permission denied (os error 13):
1743677430679.png

My Rclone config is:

Code: 
[storagebox-01]
type = sftp
host = uXXX.your-storagebox.de
user = uXXX
port = 23
key_file = /root/.ssh/id_rsa
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

My systemd service is:

Code: 
[Unit]
Description=Rclone mount for Hetzner Storage Box
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
ExecStart=/usr/bin/rclone mount storagebox-01: /mnt/storagebox-01 --config /root/.config/rclone/rclone.conf --vfs-cache-mode writes
ExecStop=/bin/fusermount -u /mnt/storagebox-01
Restart=always
RestartSec=10
MountFlags=slave

[Install]
WantedBy=multi-user.target

Am I trying to something here that can't be done, or am I missing something?
 
Nanobug said:
As my offsite backup solution, I decided to go for a Hetzner Storage Box and mount it on my old Intel NUC.
I've used Rclone for it, create a systemd for it to auto mount it.
Click to expand...

Bad idea:
Z

Thread 'Datastore synced with Rclone broken?'

I'm trying to set up a remote sync to a friend's RPI for my Proxmox Backup Server. I've been using this script that uses Rclone with an encrypted remote (Crypt) on a schedule:
Bash: 
#!/bin/bash

# A simple script to back up PBS to Crypt remote

_lock_file=/root/.tertiary_backup.lock

_schedule_time=1:30
_max_duration="7.5h"
_cutoff_mode="soft"

_datastore_dir_name=Primary_Backup
_datastore_dir_path=/mnt/datastore/$_datastore_dir_name
_config_dir=$_datastore_dir_path/host_backup

_remote_user=user
_remote_ip=tailscale-ip
_remote_port=22
_remote_name=Tertiary_Backup_Crypt...

Nanobug said:
I also tried using CIFs for it with the same result.
I can see the .chunks and .lock directoy and file, and I can create files and folders from the terminal.
Click to expand...

Another bad idea, the PBS manual recommends using fast local storage ( aka datacenter SSDs for a reason) for a reason:

https://pbs.proxmox.com/docs/installation.html#recommended-server-system-requirements

PBS splits the data in a lot small files ( chunks ) for it's space saving deduplication magic. This needs high IOPS for sufficient performance, a network share isn't good for this.

See this simulation of PBS modus operandi for a lot of different datastore options:
https://forum.proxmox.com/threads/datastore-performance-tester-for-pbs.148694/

Basically it turned out that even if the network shares are on the same host for the test (so you know that any performance differences are due the used kind of datastore (nfs, cifs/smba / iscsi/ ext4 /zfs etc) and not the latency/performance penallty of the network) performance with network shares will be way worse than with local storage connected as native filesystem (aka ext4/zfs etc).

One should note that (in another thread) there was quite a lively debate between the author of this benchmark and the PBS developers on the validity of some of his assumptions:
https://forum.proxmox.com/threads/developer-question-about-chunk-folder.148167


Nontheless they agreed with his findings, that using a network storage isn't good for performance and should be avoided. That's exactly the reason why PBS recommends using enterprise SSDs for PBS datastores. One compromise might be to setup a ZFS mirrorwith relative slow HDDs and combine them with a special device mirror on fast ssds. This will still be slower than a pure SSD datastore but will speed up stuff like garbage collection a lot.

Your best bet for offsite backup with PBS would be to use it's sync feature, e.g. a cheap vserver ( e.G. on netcup) with a PBS install or a PBS cloud provider like Inett or tuxis.nl

Another Option might be to use some external usb discs as removable dataastores which you switch on a regular schedule and store them offsite if not used
 
ness1602 said:
If i'm not mistaken Hetzner box doesnt allow acl?
Click to expand...
I don't think it does either.


Johannes S said:
Bad idea:
Z

Thread 'Datastore synced with Rclone broken?'

I'm trying to set up a remote sync to a friend's RPI for my Proxmox Backup Server. I've been using this script that uses Rclone with an encrypted remote (Crypt) on a schedule:
Bash: 
#!/bin/bash

# A simple script to back up PBS to Crypt remote

_lock_file=/root/.tertiary_backup.lock

_schedule_time=1:30
_max_duration="7.5h"
_cutoff_mode="soft"

_datastore_dir_name=Primary_Backup
_datastore_dir_path=/mnt/datastore/$_datastore_dir_name
_config_dir=$_datastore_dir_path/host_backup

_remote_user=user
_remote_ip=tailscale-ip
_remote_port=22
_remote_name=Tertiary_Backup_Crypt...



Another bad idea, the PBS manual recommends using fast local storage ( aka datacenter SSDs for a reason) for a reason:

https://pbs.proxmox.com/docs/installation.html#recommended-server-system-requirements

PBS splits the data in a lot small files ( chunks ) for it's space saving deduplication magic. This needs high IOPS for sufficient performance, a network share isn't good for this.

See this simulation of PBS modus operandi for a lot of different datastore options:
https://forum.proxmox.com/threads/datastore-performance-tester-for-pbs.148694/

Basically it turned out that even if the network shares are on the same host for the test (so you know that any performance differences are due the used kind of datastore (nfs, cifs/smba / iscsi/ ext4 /zfs etc) and not the latency/performance penallty of the network) performance with network shares will be way worse than with local storage connected as native filesystem (aka ext4/zfs etc).

One should note that (in another thread) there was quite a lively debate between the author of this benchmark and the PBS developers on the validity of some of his assumptions:
https://forum.proxmox.com/threads/developer-question-about-chunk-folder.148167


Nontheless they agreed with his findings, that using a network storage isn't good for performance and should be avoided. That's exactly the reason why PBS recommends using enterprise SSDs for PBS datastores. One compromise might be to setup a ZFS mirrorwith relative slow HDDs and combine them with a special device mirror on fast ssds. This will still be slower than a pure SSD datastore but will speed up stuff like garbage collection a lot.

Your best bet for offsite backup with PBS would be to use it's sync feature, e.g. a cheap vserver ( e.G. on netcup) with a PBS install or a PBS cloud provider like Inett or tuxis.nl

Another Option might be to use some external usb discs as removable dataastores which you switch on a regular schedule and store them offsite if not used
Click to expand...
Thank you for the reply.

I don't mind it not being fast, it's just the off site backup. I do have a local backup that is more than fast enough for my needs.

But I guess I'd need to look into Object Storage instead. Because I really don't mind not getting the best performance, since it's purely for the extra and offsite backups.
 
Nanobug said:
I don't think it does either.



Thank you for the reply.

I don't mind it not being fast, it's just the off site backup. I do have a local backup that is more than fast enough for my needs.
Click to expand...

You missed the most important point: Sycing datastores with rclone is known to mess up backups. That the target is a wan network share is just the icing on the xake.

Nanobug said:
But I guess I'd need to look into Object Storage instead. Because I really don't mind not getting the best performance, since it's purely for the extra and offsite backups.
Click to expand...

PBS doesn't support object storage like S3 at the moment, you could the normal backup function of PVE though. But then you will need more space compared to PBS.
 
Johannes S said:
You missed the most important point: Sycing datastores with rclone is known to mess up backups. That the target is a wan network share is just the icing on the xake.



PBS doesn't support object storage like S3 at the moment, you could the normal backup function of PVE though. But then you will need more space compared to PBS.
Click to expand...

Fair enough.
But if I have a local server, I can just sync it to an Rclone mount or something, right?
 
Nanobug said:
But if I have a local server, I can just sync it to an Rclone mount or something, right?
Click to expand...
I wouldn't risk it gor the reasons in the linked thread on rclone issues but if you can live with the risc of loosing your backups, I can't stop you
 
I guess I'll figure out another way to make the offsite backup then. I'm not going to rent a full server at Hetzner or something for it, then I might as well build a separate server and place it at my friends house.

Thanks for replying though :)
 
  • Like
Reactions: Johannes S
Nanobug said:
Is it Rclone in general, or just Rclone to Hetzner?
Maybe BorgBackup works instead?
Click to expand...
It's the same for all 3rd-party Tools for sycing the PBS datastores gor the reasons outlined by Proxmox developers @fabian and @Chris in the referenced thread:
fabian said:
missing chunks are possible if your manual sync processes files in the wrong order.. that's why we have a built-in sync that ensures indices and chunks are processed in a logical fashion ;)
Click to expand...
Chris said:
True, if there where concurrent operations on the datastore (backup, prune, gc,...) while the sync was ongoing, can you exclude that @Zeash ?
Click to expand...
Z

Thread 'Datastore synced with Rclone broken?'

I'm trying to set up a remote sync to a friend's RPI for my Proxmox Backup Server. I've been using this script that uses Rclone with an encrypted remote (Crypt) on a schedule:
Bash: 
#!/bin/bash

# A simple script to back up PBS to Crypt remote

_lock_file=/root/.tertiary_backup.lock

_schedule_time=1:30
_max_duration="7.5h"
_cutoff_mode="soft"

_datastore_dir_name=Primary_Backup
_datastore_dir_path=/mnt/datastore/$_datastore_dir_name
_config_dir=$_datastore_dir_path/host_backup

_remote_user=user
_remote_ip=tailscale-ip
_remote_port=22
_remote_name=Tertiary_Backup_Crypt...


You can use something like borg or restic for backups from inside the vms/lxcs or for sycing vzdumps/vma files created with PVEs native backups function to your cloud storage. The vzdumps will take more space than PBS though sibcd they are full backups without deduplication.


My own backup setup for my honelab without breaking the Bank works like this:

I have a hetzner storagebox for the data on my notebook and NAS, I backup them with restic ( borg would work too of course). Costs: Around 13 Euro for five TB. Weekly snapshots made with Hetzners auto-snaspshots ensure that a attacker can't remove all of my data even if they manage to get my ssh credentials for the storage Box.
For my vms and lxcs I have a local PBS and rent a netcup vserver ( around 10-12 Euro per month for 300 GB storage ). On the vserver I installed PBS, the netcup PBS has a daily pull sync kon which syncs the backups from my local PBS. Both PBS are allowed to pull from the other but can't write or remove anything from the other side for ransomware protection.

Additionally I do weekly vzdumps of my vms/lxcs which are synced to the storagebox. I also have an external disc on which I sync all my backups whenever I remember to do this ( I really should do this more...).

For my Budget this is a great solution ( my data is save for around 25-30 Euro per month), but I'm thinking about switching my remote PBS to Inetts Cloud PBS so I don't have to do the maintenance anymore ( 0.02 Euro per GB shouldn't kill the Bank, netter should 20 Euro for 1 TB, as I said for bulk data I gave my NAS back uped to the storagebox).

If your Budget doesn't allow this your best bet would be to buy multiple external discs, adding them as removable datastores to PBS and switch them on a regular schedule:
- Two discs for daily and weekly backups, two more monthly backups.
- The discs you don't use at he moment are stored offsite ( your pkace of work, a friends or family members place)

The idea is that you have always one disc for your current dsily and weekly backups but change it every week and store it outside of your place if you don't need it. This ensures that in case of Desaster ( hacker, fire, flooding) you still have an older vopy to restore everything from.

Hope that helps ;)
 
I use Hetzner Storage Box with PBS without problems, but each scenario is different. The key point is to use either CIFS or SSHFS as the mount protocol (I use CIFS everywhere). When you use cifs, remember to add paramters to change/force UID and GID to "backup" user

The second thing you must consider is how far from the data center you want to use these mount points. From my tests, if you use the Storage Box outside of a Hetzner data center, it works, but sometimes when I ran a garbage collection (GC), it would hang (due to high latency).

My recommendation is to use the Storage Box from within the Hetzner infrastructure: either mount it in a dedicated VPS (even a cheap one) or as a VM if you have your own dedicated servers.

For some of my clients, I use a Hetzner Storage Box as offsite backup storage, like this: local PBS inside the client’s office › push sync › VPS in Hetzner with the Storage Box mounted via CIFS. Everything works like a charm (purging, pruning, verifying).
 
You must log in or register to reply here.
Top Bottom