Screwed up (mea culpa)

P

p-user

Member
Jan 26, 2024
95
7
13
I have screwed up the .chuncks directory of my local cache for pbs, for the S3 backup.

I deleted all the chuncks in the cache, as there were a few inconsistencies. I hoped that I could resync back the S3 chuncks back to the cache or something.
However now an S3 refresh gave me this error:

2026-09-14T12:40:26+02:00: Using datastore cache with capacity 56966 for store daDup
2026-09-14T12:40:26+02:00: Fetching object ns/Backup/
2026-09-14T12:40:26+02:00: TASK ERROR: failed to create target file "/mnt/datastore/cache/tmp_iI6b18/ns/Backup/": Is a directory (os error 21)

Since there is not tmp_* directory in the cache I don't understand what the problem is here. As far as I can tell the S3 storage is still complete.

ANy help or pointers to a solution is welcome. Please note that: a) I screwed up, b) it is not a production backup, so starting new is an option, it's just a was of the data on the S3 storage (nearly 400 GB).

Is there anyway of getting the cache and the S3 storage lined up again?

Kind regards,

Albert
 
could you provide more details? which PBS version are you using, with which version was the datastore created? how about your S3 setup?
 
Thanks, the pbs version is 4.2.5 and the S3 datastore was created more or less exactly one year ago. So it was created at least with version 4 of pbs.

The cache is a USB SSD mounted to /mnt/datastore/cache and has no smart errors.

I've got 5 namespaces, 4 are used as a destination for sync jobs from the local datastore, and one is used using a proxmox client. As that is a filebackup of my mailserver I have that one create backups encrypted. Here's is my S3 configuration:

root@pbs:/etc/proxmox-backup# cat s3.cfg
s3-endpoint: Tuxis
access-key ######################
endpoint nl.dadup.eu
path-style true
put-rate-limit 10
port 443
provider-quirks skip-if-none-match-header
region default
secret-key ###################################

The datastore configuration is:
root@pbs:/etc/proxmox-backup# cat datastore.cfg
datastore: PBSStorage
comment Local Storage
gc-schedule sat 18:15
notification-mode legacy-sendmail
notify gc=error,prune=error,sync=error,verify=error
path /mnt/datastore/PBSStorage
verify-new true

datastore: daDup
backend bucket=backup,client=Tuxis,type=s3
comment Tuxis daDup Datatstore
gc-schedule sat 18:15
notification-mode legacy-sendmail
notify gc=error,prune=error,sync=error,verify=error
path /mnt/datastore/cache

Please note that a local nvme disk is mounted under /mnt/datastore/PBSStorage as local datastore. Since I want to control on the verification of backups on the S3 datastore I have not configured verify-new on the S3 datastore.

root@pbs:~# ls -la /mnt/datastore/cache/
total 1076
drwxr-xr-x 6 backup backup 4096 Sep 14 12:40 .
drwxr-xr-x 5 root root 4096 Oct 4 2025 ..
drwxr-x--- 1 backup backup 1060864 Sep 14 11:37 .chunks
-rw-r--r-- 1 backup backup 362 Sep 12 18:17 .gc-status
-rw-r--r-- 1 backup backup 0 Sep 13 2025 .lock
drwx------ 2 backup backup 16384 Sep 13 2025 lost+found
drwxr-xr-x 7 backup backup 4096 Sep 14 11:47 ns

Is there any more info you need?

Kind regards,
Albert
 
p-user said:
Fetching object ns/Backup/
Click to expand...
This seems wrong. Can you inspect the content of that "directory" on the provider side? Is it empty? empty namespaces normally contain a marker object.
 
When I use the CS Browser (where you can browse your S3 storage) I see the following for ns/Backup:

1789401045228.png

Does that seem correct? When I look into the mounted datastorage on the pbs then I don't see the .namespace file (for both the cache as the actual datastorage on disk.
 
Unfortunately,
root@pbs:~# proxmox-backup-manager version --verbose
proxmox-backup 4.2.0 running kernel: 7.0.14-17-pve
proxmox-backup-server 4.2.5-1 running version: 4.2.5
proxmox-kernel-helper 9.2.0
proxmox-kernel-7.0.14-17-pve-signed 7.0.14-17
proxmox-kernel-7.0 7.0.14-17
proxmox-kernel-7.0.14-16-pve-signed 7.0.14-16
proxmox-kernel-6.17 6.17.13-21
proxmox-kernel-6.17.13-21-pve-signed 6.17.13-21
proxmox-kernel-6.8.12-13-pve-signed 6.8.12-13
proxmox-kernel-6.8 6.8.12-13
proxmox-kernel-6.8.12-9-pve-signed 6.8.12-9
ifupdown2 3.3.0-1+pmx12
libjs-extjs 7.0.0-7
proxmox-backup-docs 4.2.5-1
proxmox-backup-client 4.2.5-1
proxmox-enterprise-support-keyring 1.1
proxmox-mail-forward 1.0.3
proxmox-mini-journalreader 1.7
proxmox-offline-mirror-helper 0.7.4
proxmox-widget-toolkit 5.2.8
pve-xtermjs 6.0.0-2
smartmontools 7.5-pve2
zfsutils-linux 2.4.4-pve1
 
Okay, in that case this would however point to the S3 API unexpectedly returning the namespace folder as independent object key, not the actually expected <datastore-name>/.cnt/ns/Backup/.namespace.
As a workaround you can do the s3-refresh yourself by downloading any contents under <datastore-name>/.cnt/ to a local tmp folder somewhere on your system, set the datastore into maintenance mode offline, clear any pre-existing content from the local store cache except the .chunks folder and move the contents from the tmp folder in place, making sure they are owned by backup:backup. After this, you can bring the datastore back online by clearing the maintenance mode. I would recommend to perform a verification afterwards and if that is successful run a garbage collection job to clear any referenced chunk from the S3 backend.
 
Thanks Chris, just did that. But once I've done that, the backups are gone. Problem is, the .chuncks directory is empty, not the namespaces (ns).
Can I do the same trick by downloading the chunks from S3 and put them in place?

Kind regards,
Albert
 
p-user said:
Problem is, the .chuncks directory is empty, not the namespaces (ns).
Click to expand...
You can recreate the folder structure for that easily as well, e.g. you could create a local tmp datastore, move over the .chunks folder from that to the s3 local cache and cleanup the tmp datastore again. The chunk marker files and locally cached chunks will then be filled again on-demand.

p-user said:
But once I've done that, the backups are gone
Click to expand...
Then you most likely either have an error in the folder layout or your s3 backend does not contain the snapshots anymore? The listing is independent from the chunks and the s3 backend since it uses the local files. Chunks on the other hand are fetched on demand and inserted into the local store cache when accessed. The marker files can be pre-generated by a garbage collection run, as this creates missing local markers so duplicate uploads can be avoided after that.
 
Last edited:
Thanks,
but I don't understand. Here's the contents of one of the vms on on the cache datastore:
root@pbs:~# ls -laR /mnt/datastore/cache/ns/Backup/vm/100
/mnt/datastore/cache/ns/Backup/vm/100:
total 16
drwxr-xr-x 3 backup backup 4096 Sep 13 01:45 .
drwxr-xr-x 22 backup backup 4096 Aug 2 01:20 ..
drwxr-xr-x 2 backup backup 4096 Sep 15 09:05 2026-09-12T22:30:02Z
-rw-r--r-- 1 backup backup 9 Jun 29 08:08 owner

'/mnt/datastore/cache/ns/Backup/vm/100/2026-09-12T22:30:02Z':
total 280
drwxr-xr-x 2 backup backup 4096 Sep 15 09:05 .
drwxr-xr-x 3 backup backup 4096 Sep 13 01:45 ..
-rw-r--r-- 1 backup backup 1238 Sep 13 01:15 client.log.blob
-rw-r--r-- 1 backup backup 266240 Sep 13 01:15 drive-scsi0.img.fidx
-rw-r--r-- 1 backup backup 492 Sep 13 01:15 index.json.blob
-rw-r--r-- 1 backup backup 380 Sep 13 01:15 qemu-server.conf.blob

I'ts only the .chunks directory that is empty (as I erronously deleted the content.

I was under the impression that indeed, the cache will automatically fill up again, e.g. when I try to restore one of the vms on the S3 datastore.
The log is attached of the restore.
 

Attachments

Oh I see, my understanding was that you cleared all contents. In that case you only need to re-create the folder structure for .chunks as suggested above, which is easiest done by copying/moving over the .chunks from a tmp datatore to your local cache. The cache does not create the digest prefix (e.g. 9d61) subfolders on insert. That is expected to be pre-existing.
 
Ok, progress.
Since the .chunks directory was empty (due to my own doing) I recreated the subdirectories in there:

cd /mnt/datastore/cache/.chunks
printf "%04x\n" {0..65535} | xargs mkdir -p
chown -R backup:backup *
chmod 750 *

This will create all the empty subdirectories 0000-ffff.

Now I am restoring a file from the S3 datastore. That worked, and backing up using the client to the S3 works as well.

Thanks for the pointer.

Are there handy commands (command line) to check or verify the backups on the S3 datastore?

And what does the option Refresh contents from S3 bucket actually do?

Very kind regards,
Albert
 
You still have the tmp files present in your local store cache. I assume these are the ones you downloaded from the s3 backend manually? If so it is safe to delete them.
 
You are correct, I removed the tmp directory with the copied ns files, and ran the garbage collection now succesfully.

However, the S3 refresh still fails (it failed before my screwup), and it looks like directory ownership issues:

026-09-15T19:58:16+02:00: Using datastore cache with capacity 55818 for store daDup
2026-09-15T19:58:17+02:00: Fetching object ns/Backup/
2026-09-15T19:58:17+02:00: TASK ERROR: failed to create target file "/mnt/datastore/cache/tmp_JD2QaS/ns/Backup/": Is a directory (os error 21)

root@pbs:~# ls -la /mnt
total 12
drwxr-xr-x 3 root root 4096 Sep 15 20:00 .
drwxr-xr-x 18 root root 4096 Nov 15 2025 ..
drwxr-xr-x 5 root root 4096 Oct 4 2025 datastore
root@pbs:~# ls -la /mnt/datastore/
total 25
drwxr-xr-x 5 root root 4096 Oct 4 2025 .
drwxr-xr-x 3 root root 4096 Sep 15 20:00 ..
drwxr-xr-x 6 backup backup 4096 Sep 15 19:58 cache
drwxr-xr-x 2 backup backup 4096 Oct 4 2025 LocalStorage
drwxr-xr-x 5 backup backup 7 Sep 15 16:57 PBSStorage
root@pbs:~# ls -la /mnt/datastore/cache
total 1076
drwxr-xr-x 6 backup backup 4096 Sep 15 19:58 .
drwxr-xr-x 5 root root 4096 Oct 4 2025 ..
drwxr-x--- 1 backup backup 1060864 Sep 15 16:17 .chunks
-rw-r--r-- 1 backup backup 349 Sep 15 19:29 .gc-status
-rw-r--r-- 1 backup backup 0 Sep 13 2025 .lock
drwx------ 2 backup backup 16384 Sep 13 2025 lost+found
drwxr-xr-x 7 backup backup 4096 Sep 14 11:47 ns
drwxr-xr-x 2 backup backup 4096 Sep 15 00:00 pbs-backup
 
You must log in or register to reply here.
Top Bottom
Back