PBS 4 - Datastore Offline in PVE

B

bco_

Member
Nov 7, 2022
14
3
8
Hello,

I've installed PBS 4 Beta and upgraded to final release, but since the 4.0.9 in beta, datastore becomes unavailable in PVE : Aug 07 22:06:04 pve01 pvestatd[1561]: pbs_2h: error fetching datastores - 500 Can't connect to 192.168.1.60:8007 (Connection timed out)
UI becomes unavailable, all process seems to be ok :
Code: 
root@pbs:~# systemctl status prox*
● proxmox-backup-banner.service - Proxmox Backup Server Login Banner
     Loaded: loaded (/usr/lib/systemd/system/proxmox-backup-banner.service; enabled; preset: enabled)
     Active: active (exited) since Wed 2025-08-06 20:41:15 CEST; 1 day 1h ago
 Invocation: bc231b71d3664a62a19dff7fdf0b37a8
    Process: 430 ExecStart=/usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-banner (code=exited, status=0/SUCCESS)
   Main PID: 430 (code=exited, status=0/SUCCESS)
   Mem peak: 1.8M
        CPU: 5ms

Aug 06 20:41:15 pbs systemd[1]: Starting proxmox-backup-banner.service - Proxmox Backup Server Login Banner...
Aug 06 20:41:15 pbs systemd[1]: Finished proxmox-backup-banner.service - Proxmox Backup Server Login Banner.

● proxmox-backup-daily-update.timer - Daily Proxmox Backup Server update and maintenance activities
     Loaded: loaded (/usr/lib/systemd/system/proxmox-backup-daily-update.timer; enabled; preset: enabled)
     Active: active (waiting) since Wed 2025-08-06 20:41:15 CEST; 1 day 1h ago
 Invocation: a45fac6eb27e4bb8a4d6676fef4bc6b6
    Trigger: Fri 2025-08-08 02:54:20 CEST; 4h 42min left
   Triggers: ● proxmox-backup-daily-update.service

Aug 06 20:41:15 pbs systemd[1]: Started proxmox-backup-daily-update.timer - Daily Proxmox Backup Server update and maintenance activities.

● proxmox-backup-proxy.service - Proxmox Backup API Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/proxmox-backup-proxy.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-08-06 20:41:20 CEST; 1 day 1h ago
 Invocation: 226f5095bfbf4867b75559702db34a4d
   Main PID: 644 (proxmox-backup-)
      Tasks: 10 (limit: 2275)
     Memory: 1.4G (peak: 1.5G)
        CPU: 27min 44.434s
     CGroup: /system.slice/proxmox-backup-proxy.service
             └─644 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy

Aug 07 04:14:24 pbs proxmox-backup-proxy[644]: Upload backup log to datastore 'pbs-sync', namespace 'VM_2h' ct/108/2025-08-07T02:02:37Z/client.log.blob

● proxmox-backup.service - Proxmox Backup API Server
     Loaded: loaded (/usr/lib/systemd/system/proxmox-backup.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-08-06 20:41:19 CEST; 1 day 1h ago
 Invocation: fe36105f54804160a13bd257fd040247
   Main PID: 567 (proxmox-backup-)
      Tasks: 6 (limit: 2275)
     Memory: 60.5M (peak: 317.3M)
        CPU: 8min 8.370s
     CGroup: /system.slice/proxmox-backup.service
             └─567 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-api

Aug 06 20:41:19 pbs proxmox-backup-api[567]: catching shutdown signal
Aug 06 20:41:19 pbs proxmox-backup-api[567]: catching reload signal
Aug 06 20:41:19 pbs systemd[1]: Started proxmox-backup.service - Proxmox Backup API Server.
Aug 06 22:45:52 pbs proxmox-backup-api[567]: pam_unix(proxmox-backup-auth:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=::ffff:192.168.1.10  user=root
Aug 06 22:45:54 pbs proxmox-backup-api[567]: authentication failure; rhost=[::ffff:192.168.1.10]:51566 user=root@pam msg=authentication error - AUTH_ERR (7)
Aug 06 22:45:54 pbs proxmox-backup-api[567]: pam_unix(proxmox-backup-auth:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=::ffff:192.168.1.10  user=root
Aug 06 22:45:56 pbs proxmox-backup-api[567]: authentication failure; rhost=[::ffff:192.168.1.10]:51566 user=root@pam msg=authentication error - AUTH_ERR (7)
Aug 06 22:46:03 pbs proxmox-backup-api[567]: pam_unix(proxmox-backup-auth:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=::ffff:192.168.1.10  user=root
Aug 06 22:46:06 pbs proxmox-backup-api[567]: authentication failure; rhost=[::ffff:192.168.1.10]:51557 user=root@pam msg=authentication error - AUTH_ERR (7)

journalctl for today :
https://pastebin.com/vJ1pi3ZY

If i restart all proxmox services, everything comes back for few hours.
 
  • Like
Reactions: fireon
Very similar here:

Code: 
error fetching datastores - 500 Status read failed: Broken pipe
 
I’m experiencing the same issue. When doing a backup to S3, the service gets stuck, and restarting the proxmox-backup-proxy brings it back online.


Here’s the status output:

Code: 
root@pbs:/var/log/proxmox-backup/tasks# service proxmox-backup-proxy status
● proxmox-backup-proxy.service - Proxmox Backup API Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/proxmox-backup-proxy.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-08-29 21:44:03 CEST; 7h ago
 Invocation: 5bd99bc500884e3ca0129adbc077305b
   Main PID: 673 (proxmox-backup-)
      Tasks: 7 (limit: 9361)
     Memory: 1.4G (peak: 1.4G)
        CPU: 15min 5.428s
     CGroup: /system.slice/proxmox-backup-proxy.service
             └─673 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy

Aug 29 21:44:30 pbs proxmox-backup-proxy[673]: error during snapshot file listing: 'unable to load blob '"/mnt/datastore/temp/vm/118/2025-08-29T19:...

And here’s the tail of the backup log:

Code: 
ERROR: backup close image failed: command error: stream closed because of a broken pipe
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 106 failed - backup close image failed: command error: stream closed because of a broken pipe
INFO: Failed at 2025-08-30 05:07:45
INFO: Backup job finished with errors
TASK ERROR: job errors
 
Last edited:
Hi,
monster400 said:
I’m experiencing the same issue. When doing a backup to S3, the service gets stuck, and restarting the proxmox-backup-proxy brings it back online.


Here’s the status output:

Code: 
root@pbs:/var/log/proxmox-backup/tasks# service proxmox-backup-proxy status
● proxmox-backup-proxy.service - Proxmox Backup API Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/proxmox-backup-proxy.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-08-29 21:44:03 CEST; 7h ago
 Invocation: 5bd99bc500884e3ca0129adbc077305b
   Main PID: 673 (proxmox-backup-)
      Tasks: 7 (limit: 9361)
     Memory: 1.4G (peak: 1.4G)
        CPU: 15min 5.428s
     CGroup: /system.slice/proxmox-backup-proxy.service
             └─673 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy

Aug 29 21:44:30 pbs proxmox-backup-proxy[673]: error during snapshot file listing: 'unable to load blob '"/mnt/datastore/temp/vm/118/2025-08-29T19:...

And here’s the tail of the backup log:

Code: 
ERROR: backup close image failed: command error: stream closed because of a broken pipe
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 106 failed - backup close image failed: command error: stream closed because of a broken pipe
INFO: Failed at 2025-08-30 05:07:45
INFO: Backup job finished with errors
TASK ERROR: job errors
Click to expand...
please add the full backup task log from the PVE and the corresponding one from the PBS, as well as the systemd journal from around the backup timespan.

There was an issue leading to blocking the proxy on client backup log upload, which has already been fixed (git only and not yet packaged at the time of writing [0]). This could be related to what you are observing, but according to the log excerpt you showed it might not be the same issue. Do you maybe have another backup job which does the log upload at this point?

[0] https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=dbaa4b6992e504007c0ce9b26c34c6f52f996dac
 
You must log in or register to reply here.
Top Bottom