PBS 4.0.11 API Not Exposing Datastores

[CryptoVibe]

Hello everyone who sees this. I've been trying to get this to work for the last few hours. I've not been able to get this to work. Any help would be GREATLY appreciated:

PBS 4.0.11 API Not Exposing Datastores - Critical Bug Report

Environment

- PBS Version: Proxmox Backup Server 4.0.11-2
- PVE Version: Proxmox VE (multiple nodes in cluster)
- PBS Setup: VM running on Proxmox VE host
- Storage Backend: NFS mount from TrueNAS SCALE via dedicated 40GbE network
- NFS Source: <truenas-ip>:/mnt/storage/pbs-backups
- Mount Point: /mnt/truenas-backups
- Capacity: 167TB available

Problem Summary

PBS datastore exists, is properly configured, and visible in the PBS web UI, but all API endpoints return empty datastore lists ({"data":[]}),
preventing Proxmox VE from discovering and using the PBS storage for backups.

Symptoms

1. PBS Web UI: Datastore truenas-backups is visible and shows 167TB available
2. PBS CLI: proxmox-backup-manager datastore list correctly shows the datastore
3. PBS API: All API endpoints return empty arrays:
- GET /api2/json/admin/datastore → {"data":[]}
- GET /api2/json/config/datastore → {"data":[],"digest":"..."}
4. Proxmox VE: Cannot add PBS storage - fails with:
- Via CLI: error fetching datastores - 400 Bad Request
- Via Web UI: Cannot find datastore 'truenas-backups', check permissions and existence! (500)

Configuration Details

Datastore Configuration (/etc/proxmox-backup/datastore.cfg)

datastore: truenas-backups
path /mnt/truenas-backups

User & Token Configuration

- User: backup@pbs (PBS realm)
- API Token: backup@pbs!proxmox-ve
- Alternative tested: root@pam with API token

ACL Permissions (/etc/proxmox-backup/acl.cfg)

acl:1:/:backup@pbs:Audit,DatastoreBackup
acl:1:/datastore/truenas-backups:backup@pbsatastoreAudit

Datastore Filesystem Status

# Mount is healthy
<nfs-server>:/mnt/storage/pbs-backups 167T 56M 167T 1% /mnt/truenas-backups

# Datastore structure exists
drwxr-xr-x 3 backup backup 4 Nov 19 19:24 .
drwxr-xr-x 3 root root 4096 Nov 20 04:27 ..
drwxr-x--- 49743 backup backup 49743 Nov 20 06:54 .chunks
-rw-r--r-- 1 backup backup 0 Nov 19 19:24 .lock

Troubleshooting Steps Performed

1. Authentication & Permissions Testing

- Created API tokens for both backup@pbs and root@pam
- Tested with various permission combinations:
- Admin on /
- Audit on /
- DatastoreBackup on /
- DatastoreAudit on /datastore/truenas-backups
- All combinations of the above
- Verified ACLs are correctly stored in /etc/proxmox-backup/acl.cfg
- Result: API still returns empty datastore list

2. API Endpoint Testing

Tested multiple API endpoints with curl:

Admin Endpoint:
curl -k "https://<pbs-ip>:8007/api2/json/admin/datastore" \
-H "Authorization: PBSAPIToken=backup@pbs!proxmox-ve:<TOKEN>"
# Returns: {"data":[]}

Config Endpoint:
curl -k "https://<pbs-ip>:8007/api2/json/config/datastore" \
-H "Authorization: PBSAPIToken=backup@pbs!proxmox-ve:<TOKEN>"
# Returns: {"data":[],"digest":"4628af642912df0b1ab9b3fbb0e6a106d021b59ec5c9cdb121b0a1242d0cf2e8"}

Direct Datastore Access:
curl -k "https://<pbs-ip>:8007/api2/json/config/datastore/truenas-backups" \
-H "Authorization: PBSAPIToken=backup@pbs!proxmox-ve:<TOKEN>"
# Returns: permission check failed

Note: The config endpoint returns a digest, proving it's reading the config file, but filtering out the datastore from the data array.

3. Datastore Recreation Attempts

- Deleted and recreated datastore via CLI (proxmox-backup-manager)
- Attempted creation via PBS web UI
- Manually edited /etc/proxmox-backup/datastore.cfg
- Restarted PBS services (proxmox-backup, proxmox-backup-proxy)
- Cleared lock files from /run/proxmox-backup/active-operations/
- Result: Datastore always appears in CLI/Web UI but never in API

4. Service Restarts

systemctl restart proxmox-backup proxmox-backup-proxy
# Verified services are running
# API still returns empty list

5. Different Authentication Methods Tested

- API tokens (backup@pbs!proxmox-ve, root@pam!proxmox-ve)
- Password authentication attempts (failed with "permission check failed")
- Both localhost and external IP API calls (same result)

Key Observations

1. Config File is Read: The /config/datastore endpoint returns a digest that matches the config file, proving the file is being parsed.
2. Permission Check Works: Accessing /config/datastore/truenas-backups directly returns "permission check failed" (not "not found"), proving the
datastore exists in the API but is being filtered by permissions.
3. CLI vs API Discrepancy: The CLI (proxmox-backup-manager) can list datastores perfectly, but the API cannot expose them externally.
4. Empty Datastore Theory: According to PBS documentation, viewing a datastore requires permissions on "any namespace it contains." This is a freshly
created datastore with no backups/namespaces yet. However, adding permissions to various namespace paths didn't resolve the issue.
5. Localhost Authentication Error: When calling API from localhost, occasionally get "authentication failed - invalid realm in user id" for root@pam
tokens.

Expected Behavior

The API endpoint /api2/json/admin/datastore or /config/datastore should return:
{
"data": [{
"name": "truenas-backups",
"path": "/mnt/truenas-backups",
...
}]
}

Actual Behavior

{"data":[]}

Impact

Critical - Prevents all Proxmox VE integration with PBS. Cannot configure backup storage, cannot create backup jobs, cannot use PBS for any backups.

Questions for Community

1. Is there a known issue with PBS 4.0.11 API datastore listing?
2. Are there additional permissions/configuration steps required in PBS 4.0.x that weren't needed in 3.x?
3. Does an empty datastore (no backups/namespaces) require special handling to appear in API?
4. Is there a different API endpoint that Proxmox VE should use for datastore discovery in 4.0.x?
5. Has anyone successfully configured PBS 4.0.11 with NFS-backed datastores and Proxmox VE integration?

Workaround Attempts

None successful. This appears to be a fundamental API issue that cannot be worked around via configuration changes.

Additional Information Available

- Full PBS and PVE logs
- Complete configuration files
- Network traces of API calls

Has anyone else encountered this issue or found a solution?

---
Redacted items:
- Specific IP addresses (replaced with placeholders)
- API token values (replaced with <TOKEN>)
- PBS fingerprint (not needed for forum post)
- Internal network details
- Specific VM IDs and hostnames
- Storage pool paths

 

[Chris]

HI,
please do not blow your issue out of proportion, this is most likely not a bug but rather a configuration issue.

ACL Permissions (/etc/proxmox-backup/acl.cfg)

acl:1:/:backup@pbs:Audit,DatastoreBackup
acl:1:/datastore/truenas-backups:backup@pbsatastoreAudit

You are trying to access your datastore via an API token, but there are no ACLs configured for the token. Please note that tokens do NOT inherit the permissions of the user they belong to. See https://pbs.proxmox.com/docs/user-management.html#api-tokens

 

[CryptoVibe]

Thank you for your quick reply, I edited the title of this post.

I've configured ACLs for both the user backup@pbs and the token backup@pbs!proxmox-ve with Admin permissions on path /.

However, when I try to authenticate via the API using the token, I get: authentication failed - invalid realm in user id

This happens with tokens from both @pam and @PBS realms. The ACL configuration appears correct. What could cause this 'invalid realm' error during token authentication?

 

[Chris]

Please share the exact (retrackted) command invocation and make sure you are escaping the header contents correctly so the ! (exclamation mark) is not interpreted by your shell. Also, use @pbs realm. Further, make sure to also propagate the permissions if you give them on root path for subpaths to be accessible as well (of course it is recommended to use the ACLs on the correct paths to limit exposure).

 

[CryptoVibe]

Thank you! That was exactly the issue - shell escaping of the exclamation mark.

I was using:
curl -H 'Authorization: PBSAPIToken=backup@pbs!proxmox-ve:TOKEN'

Which was causing bash to interpret the !.

Using a heredoc to prevent shell interpretation solved it:
ssh root@10.1.1.36 bash << 'ENDSSH'
curl -k -s 'https://localhost:8007/api2/json/admin/datastore' \
-H 'Authorization: PBSAPIToken=backup@pbs!proxmox-ve:TOKEN'
ENDSSH

The API now returns the datastore correctly. The ACLs for both the user backup@pbs and token backup@pbs!proxmox-ve with propagate enabled are working as expected.

THANK YOU!