S3 chunk lock files accumulate indefinitely in tmpfs causing memory exhaustion

[synexa]

Hi everyone,

We have been running PBS with an S3 backend for some time and recently started experiencing memory exhaustion issues that required manual server restarts to recover. After some investigation we would like to share what we have found so far and get some feedback from the community and developers.

Version: Proxmox Backup Server 4.2

Environment description:

• PBS with sync jobs (pull) running in a VM with 32 GB of RAM
• S3 backend with a 600 GB local cache on XFS, on enterprise NVMe disks (software RAID 1)
• Datastore with 355 VMs/CTs and a retention policy of 5 daily, 4 weekly and 2 monthly copies
• Several sync jobs configured to avoid or minimize overlap. Syncs are done per namespace from and to a single datastore

Problem:
PBS creates empty lock files under /run/proxmox-backup/locks/<datastore>/.chunks/ during any operation that accesses chunks (sync jobs, garbage collection...). These files are never deleted, accumulating indefinitely.
With the described data volume, millions of files accumulate in that directory after several days of operation. In our case we exceeded 23 million files at the time of performing an exhaustive analysis of the situation.

Although the files are empty and occupy negligible space on disk since they are stored in tmpfs, the kernel maintains a shmem_inode_cache object in the Slab for each of them. This memory is SUnreclaim (non-reclaimable without restarting), and with 23 million files it consumes approximately 21-22 GB of RAM in an unrecoverable way.

The result is that within a few days the system runs out of available memory, the OOM killer starts killing PBS processes, tasks get stuck in unknown state and the server requires a manual restart to recover normal operation.

Evidence:
Memory state with 23 million lock files accumulated:

MemTotal:       32861888 kB
MemFree:         2771876 kB
MemAvailable:    9348084 kB
Slab:           29376704 kB
SReclaimable:    6707936 kB
SUnreclaim:     22668768 kB

Top Slab objects (via slabtop):

23340620   shmem_inode_cache   24515328K (~23.4 GB)
23354709   dentry               4448824K (~4.2 GB)
23358524   lsm_inode_cache      2031176K (~1.9 GB)

After restarting the server with the lock files directory empty:

MemFree:         2771876 kB  →  26 GB
MemAvailable:    9348084 kB  →  30 GB
Slab:           29376704 kB  →  261 MB
SUnreclaim:     22668768 kB  →  103 MB

Confirmed behavior:

• Lock files accumulate continuously during any operation that accesses chunks
• Lock files are never deleted regardless of whether the operation completes successfully or not
• drop_caches does not free this memory as it is SUnreclaim
• The only confirmed remedy so far is a full server restart, as restarting the proxmox-backup-proxy service does not empty the directory. Over time, file accumulation is inevitable and the server will eventually run out of memory again

Is this the expected behavior for chunk lock files with an S3 backend? If so, is there any mechanism to clean them up periodically without having to restart the server?

Thanks in advance for any feedback or guidance on this issue.

 

[Chris]

Hi,

Is this the expected behavior for chunk lock files with an S3 backend? If so, is there any mechanism to clean them up periodically without having to restart the server?

the lockfiles not being cleaned up is intentional behavior to not run into races since the locking relies on flock() internally. But for these to consume so much space is definitely not intentional and we need to find a solution, so please open an issue for this at https://bugzilla.proxmox.com, referencing this thread.

As a workaround for the time being you could do one of the following:

• Set the datastore to maintenance mode offline and mount a filesystem backed by a physical disk on top of /run/proxmox-backup/locks/<datastore>/.chunks/, making sure of the correct backup:backup ownership for files and directories on that path.
• Periodically set the datastore into maintenance mode offline, once set, run rm -rf /run/proxmox-backup/locks/<datastore>/.chunks/* and then bring the datastore back online.

 

[synexa]

Hi,

Thank you for your help and quick response. We have submitted the bug to the tracker as requested. In the meantime, we have implemented the suggested workaround by mounting a dedicated filesystem on top of /run/proxmox-backup/locks/&lt;datastore&gt;/.chunks/. We will monitor how the jobs evolve with this solution.


Thanks again for your support.

 

[Tom8]

Hi all,

currently I'm running Proxmox Backup-Server 4.2.5 with production ready enterprise subscription on
Ugreen NAS DXP6800Pro
* 12 Core i5 CPU
* 8GB RAM (I know quiet less)
* 2x 1TB Samsung 990Pro NVMe SSD (50GB for ZFS root, 2nd unused NVMe only datastore)
* 4x 16TB Seagate Exos HDD
* there is a sync job to an OCI S3 Storage (with Archive)
* there is NO swap, which seems to be intentionally on ZFS root

The system died already 3x with OOM - which took a while to identify as systemd-journal was one of the first process to kill
claude-code created a logging script to watch io pressure, hdd temperature and slabcache and current proxmox-jobs (like verify, GC, sync, backup..) running.
After investigating this forum entry was found.
This steps where done

proxmox-backup-manager datastore update OCI-bucket-20260404-1414-pbs01 \
    --maintenance-mode type=offline

find /run/proxmox-backup/locks/OCI-bucket-20260404-1414-pbs01/.chunks/ \
     -mindepth 1 -delete

proxmox-backup-manager datastore update OCI-bucket-20260404-1414-pbs01 \
    --delete maintenance-mode

Some questions to it:

1. what is the current bugzilla report?
2. Should the cleanup done regulary, before the GC to an S3 storage is started?
3. it also recommended to add special vdevs to my hdd- ZFS pool - require first better understanding from my side.

br
Thomas

 

[Neobin]

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=7670

 

[budda]

PBS 4.2.5 S3 datastore: GC Phase 2 fails with​

Summary​

I have an S3 datastore on PBS 4.2.5. Initially GC was consuming almost all RAM because millions of lock files accumulated in /run/proxmox-backup/locks/<store>/.chunks (the issue discussed in the forum thread and bug #7670).

After applying the recommended workaround (maintenance mode + removing .chunks lock files), memory returned to normal and GC was started again. This time GC completed Phase 1 successfully, entered Phase 2, ran for about 7 hours, and failed with RequestTimeTooSkewed.

I believe this is a separate S3 backend issue.

​

Initial issue (lock files / slab memory)​

During GC, slab memory continuously increased until almost all RAM was consumed.

Before cleanup:

MemAvailable: 7037956 kB
Slab: 31311820 kB
SUnreclaim: 24184460 kB
Following the workaround suggested by Proxmox developers:

proxmox-backup-manager datastore update pbs-s3-sync-backup --maintenance-mode offline

find /run/proxmox-backup/locks/pbs-s3-sync-backup/.chunks \
-mindepth 2 -type f -delete

proxmox-backup-manager datastore update pbs-s3-sync-backup --delete maintenance-mode
Memory immediately returned to normal:

MemAvailable: 30286232 kB
Slab: 764384 kB
SUnreclaim: 462800 kB
So the lock cleanup workaround works as expected.

GC after lock cleanup​

I started GC again:

proxmox-backup-manager garbage-collection start pbs-s3-sync-backup
GC started normally:

Access time update check successful, proceeding with GC.
Start GC phase1 (mark used chunks)
...
marked 100% (3998 of 3998 index files)
Start GC phase2 (sweep unused chunks)
GC then stayed in Phase 2 for approximately 7 hours.

Failure​

GC finally terminated with:

<Error>
<Code>RequestTimeTooSkewed</Code>
<Message>The difference between the request time and the current time is too large.</Message>
<RequestTime>20260916T053816Z</RequestTime>
<ServerTime>2026-09-16T05:54:01Z</ServerTime>
<MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds>
</Error>

TASK ERROR: unexpected status code 403 Forbidden
The request timestamp differs from AWS server time by about 15 minutes 45 seconds.

Time synchronization​

The PBS server clock is synchronized correctly.

timedatectl status:

System clock synchronized: yes
NTP service: active
chronyc tracking immediately after the failure:

System time : 0.000062445 seconds fast of NTP time
Last offset : +0.000040192 seconds
Leap status : Normal
The clock differs from NTP by only a few microseconds.

Additional information​

• AWS CLI upload/delete works correctly using the same S3 credentials.
• IAM permissions include s3:* on the bucket.
• journalctl for proxmox-backup-proxy between 05:30 and 06:00 contains no timeout or network errors, only the RequestTimeTooSkewed response followed by TASK ERROR.

Expected behavior​

Garbage Collection should complete successfully on an S3 datastore even after running for several hours.

Actual behavior​

After applying the lock cleanup workaround, GC progresses much further but consistently fails during Phase 2 with 403 RequestTimeTooSkewed, despite the local system clock being synchronized.