PBS: proxmox-backup-proxy fails to start due to permission issues on /run/proxmox-backup

STRUBartacus · Thursday at 17:36

Hello,

after an update around two weeks ago, one of our Proxmox Backup Server instances stopped starting the service

proxmox-backup-proxy.service

Problem:

The service fails with permission errors related to the runtime directory:

mkstemp "/run/proxmox-backup/proxy.tmp_XXXXXX" failed: EACCES: Permission denied

As a result, proxmox-backup-proxy cannot create temporary files in

/run/proxmox-backup

and does not start correctly.

This PBS is running inside an LXC container.
Another PBS LXC with the same version and similar configuration is not affected.

Observed behavior:

- proxmox-backup-proxy runs as user "backup"
- There are permission issues on /run/proxmox-backup
- The proxy fails to start due to missing write access
- Removing and recreating the directory manually fixes the issue immediately

Manual workaround:

install -d -m 0755 -o backup -g backup /run/proxmox-backup
systemctl restart proxmox-backup-proxy

Permanent workaround:

To make the fix persistent, we added a systemd drop-in for the parent service
"proxmox-backup":

[Service]
ExecStartPre=/usr/bin/install -d -m 0755 -o backup -g backup /run/proxmox-backup

Since then, the proxy starts reliably, including after reboot.

Summary:

- After an update, proxmox-backup-proxy fails to start
- Cause is a permission problem on /run/proxmox-backup
- Manually creating the directory with correct permissions resolves the issue
- A systemd ExecStartPre workaround ensures reliable startup

Has anyone else observed similar behavior, or is there an official or recommended
way PBS should handle creation of /run/proxmox-backup?

oktay · Friday at 13:06

Thank you. I just started having this after upgrading pbs to the latest version. Your fix looks very clean. Will try it now.

fabian · Friday at 13:45

this absolutely should not be necessary.. please post the output of "journalctl -b" and "proxmox-backup-manager versions --verbose"

STRUBartacus · Friday at 14:55

Without the workaround:

Code:

root@pbs-vaduz:~# systemctl status proxmox-backup-proxy.service                                                                                                                                            
x proxmox-backup-proxy.service - Proxmox Backup API Proxy Server                                                                                                                                           
     Loaded: loaded (/usr/lib/systemd/system/proxmox-backup-proxy.service; enabled; preset: enabled)                                                                                                       
     Active: failed (Result: exit-code) since Fri 2025-12-26 15:02:35 CET; 2min 7s ago                                                                                                                     
 Invocation: 5b7e01547b3b48919b7ff93fe6e00d26                                                                                                                                                              
    Process: 279 ExecStart=/usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy (code=exited, status=1/FAILURE)                                                                                   
   Main PID: 279 (code=exited, status=1/FAILURE)                                                                                                                                                           
                                                                                                                                                                                                           
Dec 26 15:02:35 pbs-vaduz systemd[1]: proxmox-backup-proxy.service: Scheduled restart job, restart counter is at 5.                                                                                        
Dec 26 15:02:35 pbs-vaduz systemd[1]: proxmox-backup-proxy.service: Start request repeated too quickly.                                                                                                    
Dec 26 15:02:35 pbs-vaduz systemd[1]: proxmox-backup-proxy.service: Failed with result 'exit-code'.                                                                                                        
Dec 26 15:02:35 pbs-vaduz systemd[1]: Failed to start proxmox-backup-proxy.service - Proxmox Backup API Proxy Server.

Code:

root@pbs-vaduz:~# journalctl -b
Dec 26 14:50:28 pbs-vaduz systemd-journald[43]: Journal started
Dec 26 14:50:28 pbs-vaduz systemd-journald[43]: Runtime Journal (/run/log/journal/c5f50b3eb442425da060cb752624142e) is 8M, max 234.4M, 226.4M free.
Dec 26 14:50:28 pbs-vaduz mount[37]: mount: /dev/mqueue: must be superuser to use mount.
Dec 26 14:50:28 pbs-vaduz mount[37]:        dmesg(1) may have more information after failed mount system call.
Dec 26 14:50:28 pbs-vaduz mount[36]: mount: /dev/hugepages: must be superuser to use mount.
Dec 26 14:50:28 pbs-vaduz mount[36]:        dmesg(1) may have more information after failed mount system call.
Dec 26 14:50:28 pbs-vaduz mount[38]: mount: /run/lock: must be superuser to use mount.
Dec 26 14:50:28 pbs-vaduz mount[38]:        dmesg(1) may have more information after failed mount system call.
Dec 26 14:50:28 pbs-vaduz mount[39]: mount: /tmp: must be superuser to use mount.
Dec 26 14:50:28 pbs-vaduz mount[39]:        dmesg(1) may have more information after failed mount system call.
Dec 26 14:50:28 pbs-vaduz lvm[41]:   /dev/mapper/control: open failed: Operation not permitted
Dec 26 14:50:28 pbs-vaduz lvm[41]:   Failure to communicate with kernel device-mapper driver.
Dec 26 14:50:28 pbs-vaduz lvm[41]:   Check that device-mapper is available in the kernel.
Dec 26 14:50:28 pbs-vaduz lvm[41]:   Incompatible libdevmapper 1.02.205 (2025-02-27) and kernel driver (unknown version).
Dec 26 14:50:28 pbs-vaduz systemd[1]: Starting systemd-journal-flush.service - Flush Journal to Persistent Storage...
Dec 26 14:50:28 pbs-vaduz systemd[1]: Finished systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev.
Dec 26 14:50:28 pbs-vaduz systemd[1]: Reached target local-fs-pre.target - Preparation for Local File Systems.
Dec 26 14:50:28 pbs-vaduz systemd[1]: systemd-udevd.service - Rule-based Manager for Device Events and Files was skipped because of an unmet condition check (ConditionPathIsReadWrite=/sys).
Dec 26 14:50:28 pbs-vaduz systemd-journald[43]: Time spent on flushing to /var/log/journal/c5f50b3eb442425da060cb752624142e is 1.748ms for 18 entries.
Dec 26 14:50:28 pbs-vaduz systemd-journald[43]: System Journal (/var/log/journal/c5f50b3eb442425da060cb752624142e) is 818.2M, max 819.2M, 924.5K free.
Dec 26 14:50:28 pbs-vaduz systemd[1]: Starting systemd-networkd.service - Network Configuration...
Dec 26 14:50:28 pbs-vaduz systemd[1]: Finished systemd-journal-flush.service - Flush Journal to Persistent Storage.
Dec 26 14:50:28 pbs-vaduz systemd-networkd[79]: lo: Link UP
Dec 26 14:50:28 pbs-vaduz systemd-networkd[79]: lo: Gained carrier
Dec 26 14:50:28 pbs-vaduz systemd[1]: Started systemd-networkd.service - Network Configuration.
Dec 26 14:50:28 pbs-vaduz systemd[1]: Starting systemd-networkd-persistent-storage.service - Enable Persistent Storage in systemd-networkd...
Dec 26 14:50:28 pbs-vaduz systemd[1]: Starting systemd-networkd-wait-online.service - Wait for Network to be Configured...
Dec 26 14:50:28 pbs-vaduz systemd[1]: Finished systemd-networkd-persistent-storage.service - Enable Persistent Storage in systemd-networkd.
Dec 26 14:50:29 pbs-vaduz systemd[1]: Mounting run-proxmox\x2dbackup.mount - Mount tmpfs at /run/proxmox-backup...
Dec 26 14:50:29 pbs-vaduz mount[88]: mount: /run/proxmox-backup: must be superuser to use mount.
Dec 26 14:50:29 pbs-vaduz mount[88]:        dmesg(1) may have more information after failed mount system call.
Dec 26 14:50:29 pbs-vaduz systemd[1]: run-proxmox\x2dbackup.mount: Mount process exited, code=exited, status=32/n/a
Dec 26 14:50:29 pbs-vaduz systemd[1]: run-proxmox\x2dbackup.mount: Failed with result 'exit-code'.
Dec 26 14:50:29 pbs-vaduz systemd[1]: Failed to mount run-proxmox\x2dbackup.mount - Mount tmpfs at /run/proxmox-backup.
Dec 26 14:50:38 pbs-vaduz zfs[56]: /dev/zfs and /proc/self/mounts are required.
Dec 26 14:50:38 pbs-vaduz zfs[56]: Try running 'udevadm trigger' and 'mount -t proc proc /proc' as root.
Dec 26 14:50:38 pbs-vaduz systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Dec 26 14:50:38 pbs-vaduz systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
Dec 26 14:50:38 pbs-vaduz systemd[1]: Failed to start zfs-mount.service - Mount ZFS filesystems.
Dec 26 14:50:38 pbs-vaduz zvol_wait[61]: /dev/zfs and /proc/self/mounts are required.
Dec 26 14:50:38 pbs-vaduz zvol_wait[61]: Try running 'udevadm trigger' and 'mount -t proc proc /proc' as root.
Dec 26 14:50:38 pbs-vaduz systemd[1]: Reached target local-fs.target - Local File Systems.
Dec 26 14:50:38 pbs-vaduz zvol_wait[57]: No zvols found, nothing to do.
Dec 26 14:50:38 pbs-vaduz systemd[1]: Listening on systemd-sysext.socket - System Extension Image Management.
Dec 26 14:50:38 pbs-vaduz systemd[1]: Starting networking.service - Network initialization...
Dec 26 14:50:38 pbs-vaduz systemd[1]: Starting pbs-network-config-commit.service - Commit any pending Proxmox Backup Server network changes...
Dec 26 14:50:38 pbs-vaduz systemd[1]: Starting proxmox-backup-banner.service - Proxmox Backup Server Login Banner...
Dec 26 14:50:38 pbs-vaduz systemd[1]: systemd-binfmt.service - Set Up Additional Binary Formats was skipped because of an unmet condition check (ConditionPathIsMountPoint=/proc/sys/fs/binfmt_misc).
Dec 26 14:50:38 pbs-vaduz systemd[1]: systemd-confext.service - Merge System Configuration Images into /etc/ was skipped because no trigger condition checks were met.
Dec 26 14:50:38 pbs-vaduz systemd[1]: systemd-sysext.service - Merge System Extension Images into /usr/ and /opt/ was skipped because no trigger condition checks were met.
Dec 26 14:50:38 pbs-vaduz systemd[1]: Starting systemd-tmpfiles-setup.service - Create System Files and Directories...
Dec 26 14:50:38 pbs-vaduz systemd[1]: Finished zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev.
Dec 26 14:50:38 pbs-vaduz systemd[1]: Reached target zfs-volumes.target - ZFS volumes are ready.
Dec 26 14:50:38 pbs-vaduz systemd[1]: Finished pbs-network-config-commit.service - Commit any pending Proxmox Backup Server network changes.

Code:

root@pbs-vaduz:~# proxmox-backup-manager versions --verbose
proxmox-backup                unknown       running kernel: 6.17.4-1-pve
proxmox-backup-server         4.1.0-1       running version: 4.1.0    
ifupdown2                     3.3.0-1+pmx11                            
libjs-extjs                   7.0.0-5                                  
proxmox-backup-docs           4.1.0-1                                  
proxmox-backup-client         unknown                                  
proxmox-mail-forward          1.0.2                                    
proxmox-mini-journalreader    1.6                                      
proxmox-offline-mirror-helper 0.7.3                                    
proxmox-widget-toolkit        5.1.2                                    
pve-xtermjs                   5.5.0-3                                  
smartmontools                 7.4-pve1                                
zfsutils-linux                2.3.4-pve1

fabian · Friday at 18:17

is this running inside a container? which kind? if it's a PVE LXC container, could you post the config? the issue that it's not possible to mount the tmpfs:

Code:

Dec 26 14:50:29 pbs-vaduz systemd[1]: Mounting run-proxmox\x2dbackup.mount - Mount tmpfs at /run/proxmox-backup...<br>Dec 26 14:50:29 pbs-vaduz mount[88]: mount: /run/proxmox-backup: must be superuser to use mount.<br>Dec 26 14:50:29 pbs-vaduz mount[88]:        dmesg(1) may have more information after failed mount system call.<br>Dec 26 14:50:29 pbs-vaduz systemd[1]: run-proxmox\x2dbackup.mount: Mount process exited, code=exited, status=32/n/a<br>Dec 26 14:50:29 pbs-vaduz systemd[1]: run-proxmox\x2dbackup.mount: Failed with result 'exit-code'.<br>Dec 26 14:50:29 pbs-vaduz systemd[1]: Failed to mount run-proxmox\x2dbackup.mount - Mount tmpfs at /run/proxmox-backup.

oktay · Friday at 23:14

fabian said:
this absolutely should not be necessary.. please post the output of "journalctl -b" and "proxmox-backup-manager versions --verbose"

Meaning ownership of that directory by 'backup' should not be necessary or that it should already be set ? (There are some other files in there owned by 'backup')

STRUBartacus · Saturday at 23:00

fabian said:
is this running inside a container? which kind? if it's a PVE LXC container, could you post the config? the issue that it's not possible to mount the tmpfs:

Yes, it’s running inside a Debian LXC container (see the configuration below). I’m aware that the issue is related to missing permissions, which is why I’m using the workaround of creating /run/proxmox-backup before starting PBS.

Code:

arch: amd64
cores: 4
features: nesting=1
hostname: pbs-vaduz
memory: 2048
mp0: /mnt/pbs-vaduz/backups,mp=/mnt/backups
nameserver: 10.11.16.10 10.11.0.10 2a00:f8c0:9523:10::10 2a00:f8c0:9523::10
net0: name=eth0,bridge=vmbr0,gw=,gw6=fd44:0104:1983:11::1,hwaddr=BC:24:11:DC:F7:BB,ip=10.11.17.11/24,ip6=fd44:0104:1983:11::11/64,tag=117,type=veth
onboot: 1
ostype: debian
rootfs: local-zfs:subvol-311-disk-1,size=8G
searchdomain: ad.domain
swap: 2048
unprivileged: 0

sgw · 2025-12-29T15:48:53+0100

I currently search for an issue I see right now. The PBS here doesn't work anymore for a few days.
It was rebooted already .. right now I had to switch back to Kernel 6.17.2-1-pve, the latest one didn't boot (maybe other issue, something around zstd data corrupt).

`proxmox-backup-proxy` runs, the perms on the mentioned directory looks OK:

Code:

drwxr-xr-x  8 backup  backup  240 Dec 29 15:44 proxmox-backup

But I see these logs:

Bash:

# journalctl -f

Dec 29 15:44:51 pbs proxmox-backup-proxy[650]: GET /api2/json/nodes/localhost/tasks?limit=0&since=1764427205: 400 Bad Request: [client [::ffff:172.31.0.50]:58918] stream did not contain valid UTF-8

Dec 29 15:44:52 pbs proxmox-backup-proxy[650]: GET /api2/json/nodes/localhost/tasks?running=1&limit=100: 400 Bad Request: [client [::ffff:172.31.0.50]:58934] stream did not contain valid UTF-8

Dec 29 15:44:56 pbs proxmox-backup-proxy[650]: GET /api2/json/nodes/localhost/tasks?running=1&limit=100: 400 Bad Request: [client [::ffff:172.31.0.50]:58934] stream did not contain valid UTF-8

Dec 29 15:45:00 pbs proxmox-backup-proxy[650]: error writing task result to the tasklog

Dec 29 15:45:00 pbs proxmox-backup-proxy[650]: unable to start garbage collection job on datastore A - stream did not contain valid UTF-8

Dec 29 15:45:00 pbs proxmox-backup-proxy[650]: error writing task result to the tasklog

Dec 29 15:45:00 pbs proxmox-backup-proxy[650]: unable to start garbage collection job on datastore B - stream did not contain valid UTF-8

Dec 29 15:45:00 pbs proxmox-backup-proxy[650]: error writing task result to the tasklog

Dec 29 15:45:00 pbs proxmox-backup-proxy[650]: unable to start datastore prune job default-A-cfef1ecb-abe2-4 - stream did not contain valid UTF-8

Dec 29 15:45:00 pbs proxmox-backup-proxy[650]: error writing task result to the tasklog

Dec 29 15:45:00 pbs proxmox-backup-proxy[650]: unable to start task log rotation: stream did not contain valid UTF-8

172.31.0.50 is the IP of my laptop accessing the WebGUI.

The disks inside the PBS aren't full. Some corrupt file?
PBS runs as a VM on one of three PVEs.

The PVE I try to run the failing backups on is up-to-date in terms of patches.

sgw · 2025-12-29T16:17:16+0100

See the log of a failing backup:

Code:

INFO: starting new backup job: vzdump 142 --node srv1 --mode snapshot --storage pbsSSD --remove 0 --notes-template '{{guestname}}' --notification-mode notification-system

INFO: Starting Backup of VM 142 (qemu)

INFO: Backup started at 2025-12-29 16:16:00

INFO: status = running

INFO: VM Name: DEBIAN-webservices-sgw

INFO: include disk 'scsi0' 'PVE001:142/vm-142-disk-0.raw' 48G

INFO: backup mode: snapshot

INFO: ionice priority: 7

INFO: creating Proxmox Backup Server archive 'vm/142/2025-12-29T15:16:00Z'

INFO: issuing guest-agent 'fs-freeze' command

INFO: issuing guest-agent 'fs-thaw' command

ERROR: VM 142 qmp command 'backup' failed - backup connect failed: command error: stream did not contain valid UTF-8

INFO: aborting backup job

INFO: resuming VM again

ERROR: Backup of VM 142 failed - VM 142 qmp command 'backup' failed - backup connect failed: command error: stream did not contain valid UTF-8

INFO: Failed at 2025-12-29 16:16:01

INFO: Backup job finished with errors

INFO: notified via target `mail-to-root`

TASK ERROR: job errors

sgw · 2025-12-29T16:40:01+0100

Bash:

# proxmox-backup-manager versions --verbose
proxmox-backup                      4.0.0         running kernel: 6.17.2-1-pve
proxmox-backup-server               4.1.0-1       running version: 4.1.0     
proxmox-kernel-helper               9.0.4                                     
proxmox-kernel-6.17.4-1-pve-signed  6.17.4-1                                 
proxmox-kernel-6.17                 6.17.4-1                                 
proxmox-kernel-6.17.2-1-pve-signed  6.17.2-1                                 
proxmox-kernel-6.14.11-4-pve-signed 6.14.11-4                                 
proxmox-kernel-6.14                 6.14.11-4                                 
proxmox-kernel-6.14.11-1-pve        6.14.11-1                                 
proxmox-kernel-6.5.13-6-pve-signed  6.5.13-6                                 
proxmox-kernel-6.5                  6.5.13-6                                 
proxmox-kernel-6.5.11-6-pve-signed  6.5.11-6                                 
ifupdown2                           3.3.0-1+pmx11                             
libjs-extjs                         7.0.0-5                                   
proxmox-backup-docs                 4.1.0-1                                   
proxmox-backup-client               4.1.0-1                                   
proxmox-mail-forward                1.0.2                                     
proxmox-mini-journalreader          1.6                                       
proxmox-offline-mirror-helper       0.7.3                                     
proxmox-widget-toolkit              5.1.2                                     
pve-xtermjs                         5.5.0-3                                   
smartmontools                       7.4-pve1                                 
zfsutils-linux                      2.3.4-pve1

sgw · 2025-12-29T16:55:27+0100

Seems to be fixed by deleting

Code:

/var/log/proxmox-backup

and

Bash:

systemctl restart proxmox-backup.service proxmox-backup-proxy

Now I can start a backup ... I will see.

(idea from : https://forum.proxmox.com/threads/solved-eacces-permission-denied-when-backing-up-to-pbs.97907/ ... unsolved issue there also)

Search

Search

PBS: proxmox-backup-proxy fails to start due to permission issues on /run/proxmox-backup

STRUBartacus

Active Member

oktay

Active Member

fabian

Proxmox Staff Member

STRUBartacus

Active Member

fabian

Proxmox Staff Member

oktay

Active Member

STRUBartacus

Active Member

sgw

Member

sgw

Member

sgw

Member

sgw

Member

We value your privacy