S3 Backups stopped working after proxmox-backup-server was upgraded

[chudak]

Hi,

PBS 4.1.8, S3 datastore on DreamHost DreamObjects. Backups stopped working after proxmox-backup-server was upgraded from 4.1.6 to 4.1.7-2 (April 22) and then 4.1.8 (April 23). I did no changes to configuration — same bucket, same keys, same everything.

PVE backup jobs all failing with EACCES: Permission denied. PBS GUI Content tab shows Bad Request (400) EACCES: Permission denied. Thought it was S3 permissions but s3 check and list-buckets both work fine.

Spent a while digging and found the datastore was stuck in "S3 refresh maintenance mode" under Datastore → Options. This mode was introduced in the 4.1.7-2 release. Not sure if the upgrade triggered this automatically or a background job got stuck. Clearing it and restarting PBS fixed everything.

Posting details below for forum posterity in case others hit the same — the error message is completely misleading.


Summary from Claude chat:

PBS 4.1.8 S3 datastore inaccessible: "Bad Request (400) EACCES: Permission denied"

Environment: Proxmox Backup Server 4.1.8, S3 datastore backed by DreamHost DreamObjects, backups initiated from Proxmox VE.

Symptom: After upgrading proxmox-backup-server to 4.1.7-2 (April 22) and 4.1.8 (April 23), all PVE backup jobs to the S3-backed datastore failed with EACCES: Permission denied. The PBS GUI showed Bad Request (400) EACCES: Permission denied on the Content tab. S3 connectivity was fine — proxmox-backup-manager s3 check and s3 endpoint list-buckets both succeeded.

Likely trigger: The 4.1.7-2 changelog includes this S3-related change:

s3: bump proxmox-s3-client to 1.4.0, fixing the Content-Type for object
upload and copy requests to `application/octet-stream` instead of the
non-standard `binary/octet`

Finding: The datastore was in S3 refresh maintenance mode (visible under Datastore → Options). This mode was introduced in the 4.1.7-2 release. It is unclear whether the upgrade triggered this automatically or whether it was set by a failed background job.

Fix:

1. Datastore → Options → Edit → Maintenance Mode → set to None → Save
2. systemctl restart proxmox-backup

After restart the datastore was accessible and backups succeeded.

Note: The EACCES: Permission denied error message is misleading — it has nothing to do with S3 credentials or bucket permissions

 

[Chris]

Hi,

Hi,

PBS 4.1.8, S3 datastore on DreamHost DreamObjects. Backups stopped working after proxmox-backup-server was upgraded from 4.1.6 to 4.1.7-2 (April 22) and then 4.1.8 (April 23). I did no changes to configuration — same bucket, same keys, same everything.

PVE backup jobs all failing with EACCES: Permission denied. PBS GUI Content tab shows Bad Request (400) EACCES: Permission denied. Thought it was S3 permissions but s3 check and list-buckets both work fine.

Please share the full task log of the backup job.

Spent a while digging and found the datastore was stuck in "S3 refresh maintenance mode" under Datastore → Options. This mode was introduced in the 4.1.7-2 release. Not sure if the upgrade triggered this automatically or a background job got stuck. Clearing it and restarting PBS fixed everything.

This is not correct: the s3-refresh and corresponding maintenance mode are already present since the initial S3 support in version 4.0. Nothing changed here with recent versions. If the s3 refresh fails for some reason, the datastore is kept in it's maintenance mode in order for the user to double check what is going on.

Posting details below for forum posterity in case others hit the same — the error message is completely misleading.


Summary from Claude chat:

Please do not post AI generated output.

PBS 4.1.8 S3 datastore inaccessible: "Bad Request (400) EACCES: Permission denied"

Environment: Proxmox Backup Server 4.1.8, S3 datastore backed by DreamHost DreamObjects, backups initiated from Proxmox VE.

Symptom: After upgrading proxmox-backup-server to 4.1.7-2 (April 22) and 4.1.8 (April 23), all PVE backup jobs to the S3-backed datastore failed with EACCES: Permission denied. The PBS GUI showed Bad Request (400) EACCES: Permission denied on the Content tab. S3 connectivity was fine — proxmox-backup-manager s3 check and s3 endpoint list-buckets both succeeded.

Likely trigger: The 4.1.7-2 changelog includes this S3-related change:

s3: bump proxmox-s3-client to 1.4.0, fixing the Content-Type for object
upload and copy requests to `application/octet-stream` instead of the
non-standard `binary/octet`

So as you stated below, even with this code change your backups work just fine, so this has nothing to do with your error. Can you rather post the systemd journal of the PBS host at the time the EACCESS errors showed up. That might contain more usefull information for debugging.

Finding: The datastore was in S3 refresh maintenance mode (visible under Datastore → Options). This mode was introduced in the 4.1.7-2 release. It is unclear whether the upgrade triggered this automatically or whether it was set by a failed background job.

Fix:

1. Datastore → Options → Edit → Maintenance Mode → set to None → Save

As stated above, if an s3-refresh fails, the maintenance mode is kept on purpose. You should rather try to find out why it failed: please post the s3-refresh task log. Also, did you trigger the s3-refresh on purpose? This is definitely not something that happens during upgrade.

1. systemctl restart proxmox-backup

After restart the datastore was accessible and backups succeeded.

Note: The EACCES: Permission denied error message is misleading — it has nothing to do with S3 credentials or bucket permissions

I do think however that the eaccess error is a consequence of the underlying issue, but only once we dig deeper we'll find out. So please provide the requested information, thanks.

 

[chudak]

@Chris

Here is from my email, maybe helpful:

102: 2026-04-24 02:00:02 INFO: Starting Backup of VM 102 (qemu)
102: 2026-04-24 02:00:02 INFO: status = running
102: 2026-04-24 02:00:02 INFO: backup mode: stop
102: 2026-04-24 02:00:02 INFO: ionice priority: 7
102: 2026-04-24 02:00:02 INFO: VM Name: yuriw-nuc-vm
102: 2026-04-24 02:00:02 INFO: include disk 'virtio0' 'VMs:vm-102-disk-0' 120G
102: 2026-04-24 02:00:02 INFO: stopping virtual guest
102: 2026-04-24 02:00:11 INFO: creating Proxmox Backup Server archive 'vm/102/2026-04-24T09:00:02Z'
102: 2026-04-24 02:00:11 INFO: starting kvm to execute backup task
102: 2026-04-24 02:00:12 ERROR: VM 102 qmp command 'backup' failed - backup connect failed: command error: EACCES: Permission denied
102: 2026-04-24 02:00:12 INFO: aborting backup job
102: 2026-04-24 02:00:12 INFO: resuming VM again after 10 seconds
102: 2026-04-24 02:00:12 ERROR: Backup of VM 102 failed - VM 102 qmp command 'backup' failed - backup connect failed: command error: EACCES: Permission denied

The only s3-refresh log I have is from April 24 during my troubleshooting session, which I triggered manually and then killed. It shows this at the start:

2026-04-24T12:06:54-07:00: Using datastore cache with capacity 1093 for store pbs-backups12
2026-04-24T12:06:54-07:00: waiting for active operations to finish before continuing: read 0, write 1

What would cause an s3-refresh to fail automatically on DreamHost DreamObjects? That might help narrow it down.

 

[Chris]

The only s3-refresh log I have is from April 24 during my troubleshooting session, which I triggered manually and then killed

So that explains why you ended up in the maintenance mode. Note that the waiting for active operations to finish before continuing: read 0, write 1 means that the s3 refresh could not continue since there was still some ongoing task, which didn't allow to get exclusive access, which however is required for the sync. Since you stopped the refresh task, the maintenance mode was however not cleared.

102: 2026-04-24 02:00:12 ERROR: Backup of VM 102 failed - VM 102 qmp command 'backup' failed - backup connect failed: command error: EACCES: Permission denied

Anything in the systemd journal around this time?

 

[chudak]

Apr 23 10:03:29 pbs systemd[1]: proxmox-backup.service: Deactivated successfully.
Apr 23 10:03:29 pbs systemd[1]: Stopped proxmox-backup.service - Proxmox Backup API Server.
Apr 23 10:03:55 pbs systemd[1]: Starting proxmox-backup.service - Proxmox Backup API Server...
Apr 23 10:03:55 pbs proxmox-backup-api[837]: catching shutdown signal
Apr 23 10:03:55 pbs proxmox-backup-api[837]: catching reload signal
Apr 23 10:03:55 pbs systemd[1]: Started proxmox-backup.service - Proxmox Backup API Server.
Apr 24 00:00:06 pbs proxmox-backup-api[837]: notified via target `mail-to-root`
Apr 24 00:36:57 pbs proxmox-backup-api[837]: notified via target `mail-to-root`
Apr 24 07:03:29 pbs proxmox-backup-api[837]: pam_unix(proxmox-backup-auth:auth): authentication failure;
Apr 24 07:03:31 pbs proxmox-backup-api[837]: authentication failure; rhost=[::ffff:192.168.90.1]:43343 user=root@pam