S3 Storage via CEPH RGW & cache datastore?

[Funar]

I'm curious about the cache disk for S3 storage. We're experimenting with S3 storage in PBS utilizing CEPH's RGW. CEPH being local to the PBS sever.

Is there a way to calculate how much storage the datastore used for cache needs to be?

We don't have any concerns about API calls to the CEPH server or storage requirements as it's also on our network. However, the cache datastore is growing alongside the S3 storage bucket size. So, we're trying to plan for the future as this function matures.

 

[Chris]

Is there a way to calculate how much storage the datastore used for cache needs to be?

No, the cache is greedy and will use as much space as it will get. It needs to at least fit the various metadata files for the PBS to operate normally, so you might get away with only using a few GiB of storage.

 

[Funar]

So, as a real world example, I have 233 snapshots with 14 VMs which comes to 2.1tb (34% dedup) in the S3 bucket that PBS uses. My cache disk is occupying 680gb of 2tb. This is only a small fraction of what we're currently utilizing PBS for. My goal is to come up with a forecast for full production use. Having S3 capability is huge for us.

 

[dusatvoj]

I ran into the same issue with PBS 4.2.8 while syncing approximately 25 TB of backups from a local ZFS datastore to an S3 datastore.

Without max-cache-size configured, the local S3 cache grew to over 2.3 TiB. A previous sync also failed with EDQUOT after reaching the ZFS quota.

I resolved this by explicitly limiting the LRU cache and setting a separate ZFS quota.

1. Set max-cache-size=250G in /etc/proxmox-backup/datastore.cfg:

backend bucket=<BUCKET>,client=<S3_CLIENT>,type=s3,max-cache-size=250G

2. Set a 500 GiB ZFS quota to leave headroom for metadata, temporary files, and cache operations:

zfs set quota=500G pool/pbs-s3-cache
3. Reload the datastore by switching it offline and back online, after stopping any running jobs:

proxmox-backup-manager datastore update s3-backup --maintenance-mode type=offline
find /data/pbs-s3-cache/.chunks/ -type f -size +0c -exec truncate --size 0 {} +
proxmox-backup-manager datastore update s3-backup --delete maintenance-mode

This is important because the LRU cache capacity is initialized when the datastore is loaded. Configuration changes do not automatically update the existing cache instance.

I also cleared the previously accumulated local chunk cache while the datastore was offline, preserving the chunk marker files.

Results so far:

• 14.35 TiB uploaded to S3
• Approximately 6.6 million PUT requests
• Sustained throughput around 100–120 MB/s
• Local cache usage around 50–65 GB, instead of several TB
• No further EDQUOT errors so far

The attached screenshot shows the current statistics during synchronization.

Relevant upstream implementation:

LocalDatastoreLruCache – Proxmox source code patch

The implementation is in pbs-datastore/src/local_datastore_lru_cache.rs, with cache capacity initialization in pbs-datastore/src/datastore.rs.

The important distinction is that max-cache-size controls the local LRU chunk cache, while the ZFS quota limits overall filesystem usage. The LRU limit is based on the number of chunks rather than being a strict filesystem byte limit.

The sync is still running, so these are preliminary results, but the cache behavior is already significantly better.

Hopefully this helps others running larger PBS-to-S3 synchronization jobs.

 

[NickolasCrock951]

I'm curious about the cache disk for S3 storage. We're experimenting with S3 storage in PBS utilizing CEPH's RGW. CEPH being local to the PBS sever.

Is there a way to calculate how much storage the datastore used for cache needs to be?

We don't have any concerns about API calls to the CEPH server or storage requirements as it's also on our network. However, the cache datastore is growing alongside the S3 storage bucket size. So, we're trying to plan for the future as this function matures.

I’d be careful about extrapolating cache usage directly from the bucket size. The PBS documentation recommends 64–128 GiB for the local cache, since it stores metadata and data chunks, but the cache can grow with workload and access patterns. There isn’t a reliable fixed percentage of the S3 bucket you can use for capacity planning.

For production, I’d put the cache on a dedicated disk or a ZFS dataset with a quota, then monitor its growth through backup, garbage collection, and restore cycles. Your current 680 GB cache for 2.1 TB in S3 is quite a bit above the documented recommendation, so it would be worth checking whether the cache is simply accumulating chunks or whether your current PBS version is behaving as expected. I’d test with a cache quota before scaling up, and make sure there’s enough room for metadata and normal datastore operations.