PBS 4.0.20 - s3 cache becomes unavailable

[Str1atum]

Hi!

since PBS 4.0.20 (most probably it was this update) the local s3 cache becomes unavailable during sync or verification tasks. I did not find a pattern when this happens, but in shell the actual directory is mounted and I files are accessible. Reboot solves the problem for some time.

syslog error:
Nov 19 07:46:07 pbs-cloud proxmox-backup-proxy[676]: GET /api2/json/admin/datastore/pbs-cloud-s3/status: 400 Bad Request: [client [::ffff:192.168.1.4]:56556] mkstemp "/run/proxmox-backup/active-operations/pbs-cloud-s3.tmp_XXXXXX" failed: ENOSPC: No space left on device

Str1atum

 

[Chris]

the local s3 cache becomes unavailable during sync or verification tasks

what do you mean with this exactly? If the cache is mounted and accessible, then there should be no problem.

GET /api2/json/admin/datastore/pbs-cloud-s3/status: 400 Bad Request: [client [::ffff:192.168.1.4]:56556] mkstemp "/run/proxmox-backup/active-operations/pbs-cloud-s3.tmp_XXXXXX" failed: ENOSPC: No space left on device

Looks more like the tmpdir mounted on run is out of space. Please check and post the output of mount, df -h and df -hi next time this happens.

 

[Str1atum]

Got just 4 GB RAM and 2 GB swap on that box, but never experienced that error before. Root space is no issue. Will post when it re-occurs.

 

[fabian]

please post "df -i /run" in any case!

 

[Str1atum]

Filesystem     Inodes IUsed  IFree IUse% Mounted on
tmpfs          489124 11212 477912    3% /run

 

[fabian]

as a workaround, you can bump the number of inodes available for /run - it has slight security implications (if somebody can create arbitrary files under /run they might be able to DoS other services running on your system):

mount -o remount,nr_inodes=2M

for example would quadruple the limit, but you can also pick a higher number if PBS is the only thing running there.

 

[Str1atum]

root@pbs-cloud:~# df -i /run
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 489124 489124 0 100% /run

Strangely everything was working without issues for nearly 2 years. PBS is the only service running. Is 4 GB just too low for the latest releases?

 

[fabian]

well, the S3 feature is not that old yet the latest release brought some bug fixes for S3 to close some problematic edge cases/races, and those introduced a new locking mechanism that requires a lock file per touched chunk. those lockfiles live under /run, and if you have very little memory (or a lot of chunks are touched) you might run into this inode limit. a fix is already on the list:

https://lore.proxmox.com/pbs-devel/20251119143148.9383-1-c.ebner@proxmox.com/T/#u

 

[willseward]

I'm seeing the same behavior today.

Filesystem              Inodes  IUsed     IFree IUse% Mounted on
udev                    489697    559    489138    1% /dev
tmpfs                   500602 500601         1  100% /run

Let me know if there is anything else I can provide. I too am using the S3 feature attempting to backup a fairly sizable disk.

INFO:  42% (172.3 GiB of 408.0 GiB) in 11m 43s, read: 102.7 MiB/s, write: 102.7 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: mkstemp "/run/proxmox-backup/locks/XxXxXxXx/.chunks/XxXx/XxXxXx7fb6f6b47de4daafd458611b9e65956edb5d6feecdb6219d8.tmp_XXXXXX" failed: ENOSPC: No space left on device
INFO: aborting backup job
INFO: resuming VM again