[SOLVED] Unprivileged Container (lxc) fails to backup

[gareththered]

I've just installed PBS on two physical servers to test. All VMs and containers back up fine to one. However, unprivileged containers fail to backup to the 2nd PBS (VMs and privileged containers backup fine).

Both PBS installs were on top of a base Debian OS.

The failing backups always show unexpected EOF as follows:

2021-02-25 20:44:29 INFO: Starting Backup of VM 122 (lxc)
2021-02-25 20:44:29 INFO: status = stopped
2021-02-25 20:44:29 INFO: backup mode: stop
2021-02-25 20:44:29 INFO: ionice priority: 7
2021-02-25 20:44:29 INFO: CT Name: openssl
2021-02-25 20:44:29 INFO: including mount point rootfs ('/') in backup
2021-02-25 20:44:29 INFO: creating Proxmox Backup Server archive 'ct/122/2021-02-25T20:44:29Z'
2021-02-25 20:44:29 INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp16888_122/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 122 --backup-time 1614285869 --repository pveBackup@pbs!token1@pbs.int.mydomain.test:pve
2021-02-25 20:44:29 INFO: Starting backup: ct/122/2021-02-25T20:44:29Z
2021-02-25 20:44:29 INFO: Client name: pve
2021-02-25 20:44:29 INFO: Starting backup protocol: Thu Feb 25 20:44:29 2021
2021-02-25 20:44:29 INFO: Error: error trying to connect: unexpected EOF
2021-02-25 20:44:29 ERROR: Backup of VM 122 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp16888_122/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 122 --backup-time 1614285869 --repository pveBackup@pbs!token1@pbs.int.mydomain.test:pve' failed: exit code 255

On the PBS which works, the failing line is replaced by: INFO: No previous manifest available.

Has anyone come across this previously?

 

[gareththered]

I've done some more digging and this seems related to the fact that I added a CA signed certificate to the failing PBS by replacing /etc/proxmox-backup/proxy.pem (and proxy.key if applicable). Replacing this with a self-signed one generated by proxmox-backup-manager cert update (after first moving or deleting the current ones) then restarting a few services and re-adding the PBS Storage on PVE fixes it.

 

[gareththered]

I finally got to the bottom of this...

The CA signed certificate I used was signed by an internal CA, whose Root CA certificate needs to be (and was) added to PVE's trust-anchor store (update-ca-certificates etc.). However, I failed to notice that the .crt file I added to /usr/local/share/ca-certificates wasn't world readable. A quick chown o+r <file.crt> fixed that and the backups now work.