So on my PBS server, I use a caching drive for syncing backups to an AWS S3 bucket. Last night, the caching drive went into a read only mode (I believe the disk is worn on endurance, waiting on usb SAS caddy to confirm). The SSD was a Dell enterprise write intensive disk. Could just be a disk things, not blaming pbs. I had another write intensive rated 800gb ssd on hand. Installed the disk, formatted the disk in ext4 with a single partition of the entire disk. Pointed my caching mount at the new disk and added the UUID entry in fstab. However my S3 datastore shows as unavailable in the web UI. Any ideas? Things to check?
Caching drive replacement
- Thread starter 2000gtacoma
- Start date
Hi, in order to replace the cache drive with a new one, you must:
- Put the datastore into maintenance mode offline, so no task can read/write the datastore
- Perform the replacement on the filesystem backing the local datastore cache disk, including all operation such as formatting filesystem, adapting the systemd mount unit, adapting the backing path if required,...
- Bring the datastore back into maintenance mode read-only
- Perform an s3 refresh, which will update the required local data in the cache by fetching it from the s3 backend.
Tried to do all this. I did not remove my AWS datastore. Guess I'm getting a permissions issue or something. I was able to do an s3 refresh that completed. When attempting to run a backup, they fail. I get the following message. I'm not sure how to fix. I'm thinking initially when I set this up and selecting the drive as caching during the datastore creation, PBS does some things behind the scenes with permission, directory creation, etc. I don't know how to recreate those. I also prefer not to remove the datastore and attempt to recreate.
ERROR: VM 506 qmp command 'backup' failed - backup register image failed: command error: inserting chunk on store 'MY-AWS' failed for 6d6c43a9a2af9b22ca99dbec41e7c8be82b6210f13bcbf540c9157fb4c06c507 - mkstemp "/mnt/pbs-cache/.chunks/6d6c/6d6c43a9a2af9b22ca99dbec41e7c8be82b6210f13bcbf540c9157fb4c06c507.tmp_XXXXXX" failed: ENOENT: No such file or directory
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 506 failed - VM 506 qmp command 'backup' failed - backup register image failed: command error: inserting chunk on store 'MY-AWS' failed for 6d6c43a9a2af9b22ca99dbec41e7c8be82b6210f13bcbf540c9157fb4c06c507 - mkstemp "/mnt/pbs-cache/.chunks/6d6c/6d6c43a9a2af9b22ca99dbec41e7c8be82b6210f13bcbf540c9157fb4c06c507.tmp_XXXXXX" failed: ENOENT: No such file or directory
INFO: Failed at 2026-04-17 09:43:48
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors
Last edited:
Yes, as mentioned you will have to recreate the same folder structure as on the original cache disk, this is currently not done automatically for you. Another option might be to create an other datastore on the cache folder first so it creates this for you as expected, then remove that temp datastore without the destroy datastore option and only after that point the pre-existing datastore to the new location.
Also just realized, I believe the original disk was in a zfs by itself. The caching was then pointed at the zfs pool I believe.
You can create a regular datastore instead, the folder layout is the same. Sorry, should have explicitly mentioned that
Edit: Another option is to remove the s3 datastore without destroying it, then recreate it as s3 store on the new cache folder location with the
reuse existing datastore and overwrite in-use marker flags set.
Last edited:
So create a new datastore and make the backing path point to my new disk? I don't have caching setup for my main datastore. Only reason it was setup is for AWS S3 because it was required.
This looks more like a cache path / mount issue than an S3 problem. When PBS marks an S3 datastore as “unavailable,” it’s usually because the local cache directory isn’t accessible or doesn’t match what PBS expects.
Here are a few things to check:
1. Verify the new disk is mounted correctly
Run:
lsblk
mount | grep <your mount point>
Make sure the new SSD is mounted at the exact same path as before.
2. Check /etc/fstab
Confirm the UUID is correct
Filesystem is set to ext4
No typos in mount point
Then test:
mount -a
3. Validate permissions
PBS needs access as the backup user:
ls -ld /path/to/cache
If needed:
chown -R backup:backup /path/to/cache
4. Confirm datastore path in PBS
In the UI, verify the S3 datastore is pointing to the exact same cache path you mounted.
5. Check logs
These usually show the exact failure:
journalctl -u proxmox-backup -n 100
journalctl -u proxmox-backup-proxy -n 100
6. Test write access
sudo -u backup touch /path/to/cache/testfile
7. Restart services after fixes
systemctl restart proxmox-backup
systemctl restart proxmox-backup-proxy
Here are a few things to check:
1. Verify the new disk is mounted correctly
Run:
lsblk
mount | grep <your mount point>
Make sure the new SSD is mounted at the exact same path as before.
2. Check /etc/fstab
Confirm the UUID is correct
Filesystem is set to ext4
No typos in mount point
Then test:
mount -a
3. Validate permissions
PBS needs access as the backup user:
ls -ld /path/to/cache
If needed:
chown -R backup:backup /path/to/cache
4. Confirm datastore path in PBS
In the UI, verify the S3 datastore is pointing to the exact same cache path you mounted.
5. Check logs
These usually show the exact failure:
journalctl -u proxmox-backup -n 100
journalctl -u proxmox-backup-proxy -n 100
6. Test write access
sudo -u backup touch /path/to/cache/testfile
7. Restart services after fixes
systemctl restart proxmox-backup
systemctl restart proxmox-backup-proxy
I really feel like this is my issue. I formatted the new drive with ext4. Previously the disk was mounted as /dev/sdg/pbs-cache/ (or something similar). If I look at the datastore config, should I have something related to my s3 datastore similar to below? I have nothing that references the disk and path as cache.
AWS-S3-Backup
cache "path"
Looks like I needed to recreate the .chunks hexadecimal folder structure. After doing that everything worked. Probably could have cloned the old disk since it was in still readable in read only mode and made it work as well.