[SOLVED] Container not backing up with error 255

[kameleon]

Yesterday I started getting a failed backup on one of my containers. It was backing up properly previous to this so I know it should work. The log of the container on the host is:

2025-04-17 10:00:14 INFO: Starting backup: ct/998/2025-04-17T15:00:14Z  
2025-04-17 10:00:14 INFO: Client name: camperpve  
2025-04-17 10:00:14 INFO: Starting backup protocol: Thu Apr 17 10:00:14 2025  
2025-04-17 10:00:14 INFO: Downloading previous manifest (Tue Apr 15 21:30:08 2025)  
2025-04-17 10:00:14 INFO: Upload config file '/var/tmp/vzdumptmp1704978_998/etc/vzdump/pct.conf' to 'root@pam@10.199.161.72:8007:PVE1_Backups' as pct.conf.blob  
2025-04-17 10:00:14 INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@10.199.161.72:8007:PVE1_Backups' as root.mpxar.didx  
2025-04-17 10:00:15 INFO: Using previous index as metadata reference for 'root.mpxar.didx'  
2025-04-17 10:18:38 INFO: HTTP/2.0 connection failed
2025-04-17 10:18:38 INFO: Error: connection reset
2025-04-17 10:18:38 ERROR: Backup of VM 998 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp1704978_998/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 998 --backup-time 1744902014 --change-detection-mode metadata --entries-max 1048576 --repository root@pam@10.199.161.72:PVE1_Backups' failed: exit code 255

The logs of the same backup task but from the PBS side:

2025-04-17T10:00:14-05:00: starting new backup on datastore 'PVE1_Backups': "ct/998/2025-04-17T15:00:14Z"
2025-04-17T10:00:14-05:00: protocol upgrade done
2025-04-17T10:00:14-05:00: GET /previous_backup_time
2025-04-17T10:00:14-05:00: GET /previous
2025-04-17T10:00:14-05:00: download 'index.json.blob' from previous backup.
2025-04-17T10:00:14-05:00: POST /blob
2025-04-17T10:00:14-05:00: add blob "/var/lib/vz/backups/ct/998/2025-04-17T15:00:14Z/pct.conf.blob" (472 bytes, comp: 472)
2025-04-17T10:00:14-05:00: GET /previous_backup_time
2025-04-17T10:00:15-05:00: GET /previous
2025-04-17T10:00:15-05:00: register chunks in 'root.ppxar.didx' from previous backup.
2025-04-17T10:00:15-05:00: download 'root.ppxar.didx' from previous backup.
2025-04-17T10:18:38-05:00: backup ended and finish failed: backup ended but finished flag is not set.
2025-04-17T10:18:38-05:00: removing unfinished backup
2025-04-17T10:18:38-05:00: TASK ERROR: backup ended but finished flag is not set.

It is only this one container, there is another container of similar size on this host that backs up no issue. Looking at the error code 255 and "backup ended but finished flag is not set" I come across some people with the same error saying to delete the oldest backups. I tried to delete all but the most recent 5 (I had 52) and the same error continues.

Host: pve-manager/8.4.1/2a5fa54a8503f96d (running kernel: 6.8.12-9-pve)
PBS: client version: 3.4.0

Both host and PBS are showing as fully up to date. What can I do to get this CT to properly back up again?

 

[kameleon]

Ok, so this is weird. As of 2 hours ago it started backing up properly again. This task runs every 30 minutes so it missed over a day of backups. Last good backup before the issue was 4-15-2025 at 21:30 until when it started working again on 4-17-2025 at 15:00. Where should I start looking to be sure this doesn't happen again? The other CT on this host has been backing up fine the whole time.

 

[Chris]

Hi,

2025-04-17 10:18:38 INFO: HTTP/2.0 connection failed

this error is typically encountered if there are some middle boxes not acting nice. So check your network, some things to consider:

• Do you have some proxy which might not handle the http2 upgrade as expected (e.g. not propagating the upgrade header [0])?
• Check your routes, e.g. via mtr. Maybe the traffic is not taking the path you expect it to?
• Do you use a VPN in-between the Proxmox VE and the Proxmox Backup Server host?
• Are you sure you are connecting to the right host? E.g. duplicate IPs might cause strange intermitten issues, check the output of ip neigh if your host are in the same subnet.

[0] https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Upgrade

 

[kameleon]

Hi,

this error is typically encountered if there are some middle boxes not acting nice. So check your network, some things to consider:

• Do you have some proxy which might not handle the http2 upgrade as expected (e.g. not propagating the upgrade header [0])?
• Check your routes, e.g. via mtr. Maybe the traffic is not taking the path you expect it to?
• Do you use a VPN in-between the Proxmox VE and the Proxmox Backup Server host?
• Are you sure you are connecting to the right host? E.g. duplicate IPs might cause strange intermitten issues, check the output of ip neigh if your host are in the same subnet.

[0] https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Upgrade

I do use zerotier as a medium for all my boxes to talk to each other so that may have been the issue. The thing that concerned me was it was only the 998 container that was experiencing this issue. The very next container 999 worked fine, every time. For now I'll just continue to monitor it and see if it does it again.