[SOLVED] Proxmox Backup + tailscale = proxmox-backup-proxy.service won't boot

[latenightcat]

I've installed tailscale on my proxmox backup vm, along with a cronjob to update the certificate files daily... or so I thought.

/etc/cron.daily/tailscale_cert

#!/bin/sh
tailscale cert --cert-file /etc/proxmox-backup/proxy.pem --key-file /etc/proxmox-backup/proxy.key $(hostname -f) && systemctl reload proxmox-backup-proxy

However, I noticed that my proxmox server was complaining it couldn't reach the backup server. I gave the VM a reboot, no luck. Afterwards I logged onto it and found that proxmox-backup-proxy.service isn't running.

The error seems to be indicating that there's some SSL issue here, but I'm unsure what it is. Perhaps it doesn't like that tailscale generates EC keys?

░░ Subject: A start job for unit proxmox-backup-proxy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit proxmox-backup-proxy.service has begun execution.
░░
░░ The job identifier is 3889.
Aug 21 23:35:57 pbs proxmox-backup-proxy[22568]: apply old journal log rrd.journal-669a4d10
Aug 21 23:35:57 pbs proxmox-backup-proxy[22568]: Error: failed to set tls acceptor private key file
Aug 21 23:35:57 pbs proxmox-backup-proxy[22568]: Caused by:
Aug 21 23:35:57 pbs proxmox-backup-proxy[22568]:     error:8000000D:system library:file_ctrl:reason(2):../crypto/bio/bss_file.c:297:calling fopen(/etc/proxmox-backup/proxy.key, r), error:10080002:BIO routines:file_ctrl:system lib:../crypto/bio/bss_file.c:300:,  error:0A080002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib:../ssl/ssl_rsa.c:367:
Aug 21 23:35:57 pbs systemd[1]: proxmox-backup-proxy.service: Main process exited, code=exited, status=1/FAILURE

How can I get this to work so that proxmox backups work, with a tailscale generated certificate?

 

[ggoller]

Hi!
the error looks cut off, could you post the full one (using e.g. journalctl)? You can also check the /etc/proxmox-backup directory if the two files exist and have the correct permissions.

 

[latenightcat]

Hi!
the error looks cut off, could you post the full one (using e.g. journalctl)? You can also check the /etc/proxmox-backup directory if the two files exist and have the correct permissions.

Its not cut off, here's a few more lines for evidence:

░░ Subject: A start job for unit proxmox-backup-proxy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit proxmox-backup-proxy.service has begun execution.
░░
░░ The job identifier is 3889.
Aug 21 23:35:57 pbs proxmox-backup-proxy[22568]: apply old journal log rrd.journal-669a4d10
Aug 21 23:35:57 pbs proxmox-backup-proxy[22568]: Error: failed to set tls acceptor private key file
Aug 21 23:35:57 pbs proxmox-backup-proxy[22568]: Caused by:
Aug 21 23:35:57 pbs proxmox-backup-proxy[22568]:     error:8000000D:system library:file_ctrl:reason(2):../crypto/bio/bss_file.c:297:
calling fopen(/etc/proxmox-backup/proxy.key, r), error:10080002:BIO routines:file_ctrl:system lib:../crypto/bio/bss_file.c:300:, err
or:0A080002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib:../ssl/ssl_rsa.c:367:
Aug 21 23:35:57 pbs systemd[1]: proxmox-backup-proxy.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit proxmox-backup-proxy.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Aug 21 23:35:57 pbs systemd[1]: proxmox-backup-proxy.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit proxmox-backup-proxy.service has entered the 'failed' state with result 'exit-code'.
Aug 21 23:35:57 pbs systemd[1]: Failed to start proxmox-backup-proxy.service - Proxmox Backup API Proxy Server.
░░ Subject: A start job for unit proxmox-backup-proxy.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit proxmox-backup-proxy.service has finished with a failure.
░░
░░ The job identifier is 3889 and the job result is failed.
Aug 21 23:35:57 pbs systemd[1]: proxmox-backup-proxy.service: Scheduled restart job, restart counter is at 5.

As for correct permissions - these files were owned by root:root, I changed to backup:backup and that did work.

Good news for anyone else finding this thread, but I would propose this as an additional check for proxmox-backup to perform, and create a clearer error message for.

 

[latenightcat]

And for anyone in the future doing this setup, here's the new /etc/cron.daily/tailscale_cert script (remember to chmod +x it!)

#!/bin/sh
tailscale cert --cert-file /etc/proxmox-backup/proxy.pem --key-file /etc/proxmox-backup/proxy.key $(hostname -f) && \
chown backup:backup /etc/proxmox-backup/proxy.pem /etc/proxmox-backup/proxy.key && \
systemctl reload proxmox-backup-proxy

 

[ggoller]

Its not cut off, here's a few more lines for evidence:

Oops, my bad—turns out there's nothing after the colon.

Good news for anyone else finding this thread, but I would propose this as an additional check for proxmox-backup to perform, and create a clearer error message for.

openssl opens all the files, but we could still put a permission check in front.
Submitted this patch: https://lists.proxmox.com/pipermail/pbs-devel/2024-August/010657.html.

 

[ggoller]

Hi!
just fyi – this patch has been merged! https://git.proxmox.com/?p=proxmox.git;a=commit;h=d8fa495a50339a00de3fddc0839b8ea6b4bf96a0.