Backup not possible to Hetzner StorageBox

[cradling_stack339]

Hi,
I am using a Hetzner StorageBox as a Datastore in PBS, mounting it as follows:

//u******-sub1.your-storagebox.de/u******-sub1/pbs /mnt/storagebox cifs iocharset=utf8,rw,uid=34,gid=34,credentials=/root/.smbcredentials,file_mode=0660,dir_mode=0770,seal 0 0

In PBS, i created a user with Roles DatastoreBackup and DatastoreReader for /datastore/storagebox
In the Datastore I also created a namespace.
I added the Datastore to PVE with the user with no problems.
When i try to backup a container I get this error:

()
{{guestname}}
INFO: starting new backup job: vzdump 104 --remove 0 --notes-template '{{guestname}}' --mode stop --notification-mode auto --node pve --storage PBS-backup
INFO: Starting Backup of VM 104 (lxc)
INFO: Backup started at 2024-05-12 18:35:30
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: caddy
INFO: including mount point rootfs ('/') in backup
INFO: creating Proxmox Backup Server archive 'ct/104/2024-05-12T16:35:30Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=encrypt --keyfd=16 pct.conf:/var/tmp/vzdumptmp132092_104/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 104 --backup-time 1715531730 --entries-max 1048576 --repository proxmox_backup@pbs@***.***.***.***:storagebox --ns pve-backup
INFO: Starting backup: [pve-backup]:ct/104/2024-05-12T16:35:30Z
INFO: Client name: pve
INFO: Starting backup protocol: Sun May 12 18:35:30 2024
INFO: Using encryption key from file descriptor..
INFO: Encryption key fingerprint: 4d:e0:d1:7a:92:d2:68:b9
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp132092_104/etc/vzdump/pct.conf' to 'proxmox_backup@pbs@***.***.***.***:8007:storagebox' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'proxmox_backup@pbs@***.***.***.***:8007:storagebox' as root.pxar.didx
INFO: root.pxar: had to backup 107.99 MiB of 107.99 MiB (compressed 41.251 MiB) in 9.97s
INFO: root.pxar: average backup speed: 10.83 MiB/s
INFO: Uploaded backup catalog (54.189 KiB)
INFO: Error: error during syncfs: No such file or directory (os error 2)
ERROR: Backup of VM 104 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=encrypt' '--keyfd=16' pct.conf:/var/tmp/vzdumptmp132092_104/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 104 --backup-time 1715531730 --entries-max 1048576 --repository proxmox_backup@pbs@***.***.***.***:storagebox --ns pve-backup' failed: exit code 255
INFO: Failed at 2024-05-12 18:35:45
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

Error Message in PBS:

...
2024-05-12T18:35:42+02:00: successfully added chunk db51efedc7b041655427648c8c21b85233bd5e766e7bd660002275bfd1effe4b to dynamic index 1 (offset 86564980, size 4404302)
2024-05-12T18:35:42+02:00: successfully added chunk bf6079555e52aa3260366237109a4ad15f2005fc59981fd9e5bfae6d4888dc2a to dynamic index 1 (offset 90969282, size 4134312)
2024-05-12T18:35:42+02:00: successfully added chunk 4afe83ce030d32e2ee61a15b76b9791edfd65bb838c40036e17ac579d6cbd8a7 to dynamic index 1 (offset 95103594, size 4920712)
2024-05-12T18:35:42+02:00: successfully added chunk e55c37f88e6eaad8e9d7ecb5823ab371a1422014c3e502119deb54f53828d512 to dynamic index 1 (offset 100024306, size 13211282)
2024-05-12T18:35:42+02:00: POST /dynamic_close
2024-05-12T18:35:43+02:00: Upload statistics for 'root.pxar.didx'
2024-05-12T18:35:43+02:00: UUID: **
2024-05-12T18:35:43+02:00: Checksum: **
2024-05-12T18:35:43+02:00: Size: 113235588
2024-05-12T18:35:43+02:00: Chunk count: 36
2024-05-12T18:35:43+02:00: Upload size: 113235588 (100%)
2024-05-12T18:35:43+02:00: Duplicates: 0+34 (94%)
2024-05-12T18:35:43+02:00: Compression: 38%
2024-05-12T18:35:43+02:00: successfully closed dynamic index 1
2024-05-12T18:35:43+02:00: POST /dynamic_chunk
2024-05-12T18:35:43+02:00: upload_chunk done: 55490 bytes, 1e6843b09851275965c96c6231d8995fb027437b7b00d636221fe30fa3f83795
2024-05-12T18:35:43+02:00: PUT /dynamic_index
2024-05-12T18:35:43+02:00: dynamic_append 1 chunks
2024-05-12T18:35:43+02:00: successfully added chunk 1e6843b09851275965c96c6231d8995fb027437b7b00d636221fe30fa3f83795 to dynamic index 2 (offset 0, size 55490)
2024-05-12T18:35:43+02:00: POST /dynamic_close
2024-05-12T18:35:44+02:00: Upload statistics for 'catalog.pcat1.didx'
2024-05-12T18:35:44+02:00: UUID: ***
2024-05-12T18:35:44+02:00: Checksum: ***
2024-05-12T18:35:44+02:00: Size: 55490
2024-05-12T18:35:44+02:00: Chunk count: 1
2024-05-12T18:35:44+02:00: Upload size: 55490 (100%)
2024-05-12T18:35:44+02:00: Compression: 48%
2024-05-12T18:35:44+02:00: successfully closed dynamic index 2
2024-05-12T18:35:44+02:00: POST /blob
2024-05-12T18:35:44+02:00: add blob "/mnt/storagebox/ns/pve-backup/ct/104/2024-05-12T16:35:30Z/index.json.blob" (484 bytes, comp: 484)
2024-05-12T18:35:44+02:00: POST /finish
2024-05-12T18:35:45+02:00: POST /finish: 400 Bad Request: error during syncfs: No such file or directory (os error 2)
2024-05-12T18:35:45+02:00: backup ended and finish failed: backup ended but finished flag is not set.
2024-05-12T18:35:45+02:00: removing unfinished backup
2024-05-12T18:35:45+02:00: TASK ERROR: backup ended but finished flag is not set.

Am I missing something? It worked in a previous installation, but I dont remember the fstab entry or the roles the user has to have on PBS.

Thanks

 

[ggoller]

Hi!
seems like the syncfs call fails. Try to check if some of the `sync-level` options work (https://pbs.proxmox.com/docs/storage.html#tuning).

 

[cradling_stack339]

Thanks a lot, setting the sync-level to file worked

 

[cradling_stack339]

false alarm, it only worked for one small container.
However, when I create a local Datastore on PBS, it works. So it seems to be an issue with the SMB mount.
I tried out a number of settings.

- I played with SMB mounting options, removing seal, file_mode and dir_mode
- mounting the storagebox directly instead of a subfolder
- using the root user to perform backups
- using root namespace
- all sync level settings

i always get the same error. either the backup stops after some minutes with an error or it continues on forever without uploading chunks. I will try again with WebDav.

 

[ggoller]

Hi!
sadly I couldn't reproduce this. You could check out the `nostrictsync` mount option as well [0] or set the sync level to `None` [1].

[0]: https://www.kernel.org/doc/html//v6.7-rc1/admin-guide/cifs/usage.html
[1]: https://pbs.proxmox.com/docs/storage.html#tuning

 

[cradling_stack339]

I installed the PBS LXC from tteck's scripts and it worked.
Mounted the SMB directly to the host and passed it through to the container via bind mount.
I must have messed up something in the VM installation, I don't know what.