New domain. pve failures, need help fixing ssl

[slatko]

So, my proxmox install was originally location.domain.info, I recently moved to location.doman.pro
With a cloudflare tunnel pointed at the ip, it worked. I just rebooted my server for the first time since the domain change,
and upon boot pve doesn't start.

sudo systemctl status pve-manager

slatko@location:~$ sudo systemctl status pve-manager
● pve-guests.service - PVE guests
     Loaded: loaded (/lib/systemd/system/pve-guests.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-09-25 19:42:32 CDT; 9min ago
   Main PID: 2130 (code=exited, status=111)
        CPU: 1.361s

Sep 25 19:42:33 location.ratz.pro systemd[1]: Starting PVE guests...
Sep 25 19:42:32 location.ratz.pro pvesh[2130]: ipcc_send_rec[1] failed: Connection refused
Sep 25 19:42:32 location.ratz.pro pvesh[2130]: ipcc_send_rec[2] failed: Connection refused
Sep 25 19:42:32 location.ratz.pro pvesh[2130]: ipcc_send_rec[3] failed: Connection refused
Sep 25 19:42:32 location.ratz.pro pvesh[2130]: Unable to load access control list: Connection refused
Sep 25 19:42:32 location.ratz.pro systemd[1]: pve-guests.service: Main process exited, code=exited, status=111/n/a
Sep 25 19:42:32 location.ratz.pro systemd[1]: pve-guests.service: Failed with result 'exit-code'.
Sep 25 19:42:32 location.ratz.pro systemd[1]: Failed to start PVE guests.
Sep 25 19:42:32 location.ratz.pro systemd[1]: pve-guests.service: Consumed 1.361s CPU time.

Can't manually start it.

Then when I look at my syslog, it does this repeatedly, every few seconds
sudo tail -f /var/log/syslog

slatko@location:~$ sudo tail -f /var/log/syslog
Sep 25 19:53:10 location pveproxy[1923]: worker 2988 finished
Sep 25 19:53:10 location pveproxy[1923]: starting 1 worker(s)
Sep 25 19:53:10 location pveproxy[2989]: worker exit
Sep 25 19:53:10 location pveproxy[1923]: worker 2991 started
Sep 25 19:53:10 location pveproxy[2990]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1944.
Sep 25 19:53:10 location pveproxy[2991]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1944.
Sep 25 19:53:10 location pveproxy[1923]: worker 2989 finished
Sep 25 19:53:10 location pveproxy[1923]: starting 1 worker(s)
Sep 25 19:53:10 location pveproxy[1923]: worker 2992 started
Sep 25 19:53:10 location pveproxy[2992]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1944.
Sep 25 19:53:15 location pveproxy[2990]: worker exit
Sep 25 19:53:15 location pveproxy[2991]: worker exit
Sep 25 19:53:15 location pveproxy[1923]: worker 2990 finished
Sep 25 19:53:15 location pveproxy[1923]: starting 1 worker(s)
Sep 25 19:53:15 location pveproxy[1923]: worker 2995 started
Sep 25 19:53:15 location pveproxy[1923]: worker 2991 finished
Sep 25 19:53:15 location pveproxy[1923]: starting 1 worker(s)
Sep 25 19:53:15 location pveproxy[2992]: worker exit
Sep 25 19:53:15 location pveproxy[1923]: worker 2996 started
Sep 25 19:53:15 location pveproxy[2995]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1944.
Sep 25 19:53:15 location pveproxy[2996]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1944.
Sep 25 19:53:16 location pveproxy[1923]: worker 2992 finished
Sep 25 19:53:16 location pveproxy[1923]: starting 1 worker(s)
Sep 25 19:53:16 location pveproxy[1923]: worker 2997 started
Sep 25 19:53:16 location pveproxy[2997]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1944.

So the system can't find the new ssl, I would run letsencrypt via terminal, but apache2 and nginx aren't even installed on my proxmox server.
So it just loops that error, what can I do in cli to register the domain and ssl??

I'm seeing some things from google, but am terrified of losing my proxmox configurations and vm's.

I would try
pvenode acme account register account-name mail@example.com



But am scared of losing everything.

slatko@location:~$ sudo pvenode account list
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

sudo pvenode acme account register default D347h@pm.me
slatko@location:~$ sudo pvenode acme account register default D347h@pm.me ipcc_send_rec[1] failed: Connection refused ipcc_send_rec[2] failed: Connection refused ipcc_send_rec[3] failed: Connection refused Unable to load access control list: Connection refused

As far as I can tell, the ssl is the only issue, it's the only thing that's changed.
So if somebody could help me re-register to the new domain and new ssl, I need the help.

 

[Moayad]

Hi,

Can you tell us exactly how you changed the domain?
Can you also please check if the new domain is in the outputs of the following commands:
- cat /etc/hosts
- hostname -f
- cat nodes/<DOMAIN>/config


EDIT: You can also check from the following [0] for renaming the PVE node.


[0] https://pve.proxmox.com/wiki/Renaming_a_PVE_node

 

[slatko]

It's accessed through a cloudflare tunnel.
All for that, is you add your domain to your cloudflare account, then make a record pointing the domain at your local ip.

Dumb me, didn't change a single thing in the proxmox gui after the move.
After I setup the cloudflare tunnel initially it worked, but after reboot it's the ssl error in the syslog.

The only thing I changed on the server itself, is the /etc/hosts file.

Cat /etc/hosts

slatko@location:~$ cat /etc/hosts
127.0.0.1    localhost
144.82.12.x  redacted
144.82.12.x   redacted
144.82.12.x   redacted
144.82.12.x   redacted

hostname -f

slatko@location:~$ hostname -f
location.ratz.pro

slatko@location:/$ cat nodes/location/config
cat: nodes/location.ratz.pro/config: No such file or directory

Syslog still looks like this

slatko@location:/$ sudo tail -f /var/log/syslog
Sep 26 11:46:45 location pveproxy[2568]: starting 1 worker(s)
Sep 26 11:46:45 location pveproxy[2568]: worker 4386 finished
Sep 26 11:46:45 location pveproxy[2568]: worker 4388 started
Sep 26 11:46:45 location pveproxy[2568]: worker 4387 finished
Sep 26 11:46:45 location pveproxy[2568]: starting 2 worker(s)
Sep 26 11:46:45 location pveproxy[2568]: worker 4389 started
Sep 26 11:46:45 location pveproxy[2568]: worker 4390 started
Sep 26 11:46:45 location pveproxy[4388]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1944.
Sep 26 11:46:45 location pveproxy[4390]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1944.
Sep 26 11:46:45 location pveproxy[4389]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1944.
Sep 26 11:46:50 location pveproxy[4388]: worker exit
Sep 26 11:46:50 location pveproxy[4390]: worker exit
Sep 26 11:46:50 location pveproxy[4389]: worker exit

I know how normally to change the hostname, the issue is I never updated anything in the proxmox gui to the new domain and it now fails to run.

 

[Moayad]

slatko@location:/$ cat nodes/location.ratz.pro/config

This command should as the following:

cat nodes/location/config

 

[Moayad]

I would also mention that you can use the Self-Signed certs and then edit the new domain. To do that make backup of the current certificates, then run `pvecm updatecerts -f` command and then restart the pveproxy and pvedaemon services.

The above should let you access to your PVE GUI, and then you can configure your proper SSL.

 

[slatko]

I would also mention that you can use the Self-Signed certs and then edit the new domain. To do that make backup of the current certificates, then run `pvecm updatecerts -f` command and then restart the pveproxy and pvedaemon services.

The above should let you access to your PVE GUI, and then you can configure your proper SSL.

Damn, that might have done it.
3 days ago I gave up cus I'm impatient. I reinstalled proxmox.
Restored my vm's I had backed up from /var/lib/vz/dump

I'm up again