Proxmox VE <-> Proxmox Backup Server trust issues

[NickyDoes]

I'm working with an existing PVE and a new PBS. I have self-signed certs that I've installed on both machines - to allow https. I installed them via the filesystem, following a process of replacing the certs with a cert chain, key, and CA cert, then running `pvecm updatecerts` and restarting pveproxy and pvedaemon services.

I did something quite similar on the pbs machine.

I've created a backup. The fingerprint matches, though I don't fully understand why this is necesssary if the machines have trusted certs from a trusted root CA.

When I create a backup job, I get the following errors.

I've been reading manuals, forum posts, and articles from afternoon into the late hours. What could be the problem?

INFO: starting new backup job: vzdump 107 --remove 0 --notification-mode auto --mode snapshot --notes-template '{{guestname}}' --storage fours --node app-server
INFO: Starting Backup of VM 107 (qemu)
INFO: Backup started at 2024-10-23 22:14:38
INFO: status = running
INFO: VM Name: NextCloud
INFO: include disk 'scsi0' 'vmct:vm-107-disk-2' 52G
INFO: include disk 'efidisk0' 'vmct:vm-107-disk-1' 1M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/107/2024-10-24T02:14:38Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 107 qmp command 'backup' failed - backup connect failed: command error: error trying to connect: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889:
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 107 failed - VM 107 qmp command 'backup' failed - backup connect failed: command error: error trying to connect: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889:
INFO: Failed at 2024-10-23 22:14:39
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

 

[fabian]

you don't need to have a fingerprint if the cert is trusted by the system.. could you try connecting with `curl -v` and `proxmox-backup-client login --repository ....` and print the output here?

 

[NickyDoes]

Thank you @fabian for pointing me in the right direction. In both PVE and PBS, I regenerated the certs, updated the storage/datastore's 'fingerprint' between PBS. Now the backup completes successfully. I'm following this guide more closely to implement self-signed certs.

root@app-server:~# curl -v https://backup-server.demarco.home.arpa:8007
*   Trying 10.71.73.29:8007...
* Connected to backup-server.demarco.home.arpa (10.71.73.29) port 8007 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self-signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@app-server:~# exit
logout
Connection to app-server closed.

1. I restored the certs on Proxmox VE.

root@app-server:/etc/pve/local# ls
lrm_status  openvz  pve-ssl-chain.pem  pve-ssl.pem     qemu-server
lxc         priv    pve-ssl.key        pve-ssl.pem.dh  ssh_known_hosts
root@app-server:/etc/pve/local# systemctl stop pveproxy
root@app-server:/etc/pve/local# systemctl stop pvedaemon
root@app-server:/etc/pve/local# rm /etc/pve/local/pve-ssl.pem
rm /etc/pve/local/pve-ssl.key
rm /etc/pve/pve-root-ca.pem
rm /etc/pve/priv/pve-root-ca.key
root@app-server:/etc/pve/local# pvecm updatecerts --force

2. I restored the certs on Proxmox Backup Server

root@backup-server:~# systemctl stop proxmox-backup-proxy.service
root@backup-server:~# cd /etc/proxmox-backup/
root@backup-server:/etc/proxmox-backup# ls
acl.cfg  authkey.key  authkey.pub  csrf.key  datastore.cfg  proxy.key  proxy.pem  shadow.json  tfa.json.lock  user.cfg
root@backup-server:/etc/proxmox-backup# rm proxy.pem
root@backup-server:/etc/proxmox-backup# rm proxy.key
root@backup-server:/etc/proxmox-backup# proxmox-backup-manager cert update  --force
root@backup-server:/etc/proxmox-backup# ls
acl.cfg  authkey.key  authkey.pub  csrf.key  datastore.cfg  proxy.key  proxy.pem  shadow.json  tfa.json.lock  user.cfg

Now the backup completes successfully.