[SOLVED] Strange behaviour after reboot

[adam99]

My Proxmox Hypervisor 6.3-2 behaves really strange after reboot. It was used to test terraform deployment and after reboot it changed his hostname to one of the machines that were deployed by Terraform. Now I cannot access the 8006 port and it also has got out of the cluster. The logs says that the SSL key is invalid, but I think it's because of the fact, that the hostname changed. Also the pve services don't run anymore. Here is the log:

root@BackupStorage:~# tail /var/log/syslog
May 6 11:10:01 BackupStorage pveproxy[23197]: worker exit
May 6 11:10:01 BackupStorage pveproxy[23198]: worker exit
May 6 11:10:01 BackupStorage pveproxy[4110]: worker 23198 finished
May 6 11:10:01 BackupStorage pveproxy[4110]: worker 23197 finished
May 6 11:10:01 BackupStorage pveproxy[4110]: starting 2 worker(s)
May 6 11:10:01 BackupStorage pveproxy[4110]: worker 23254 started
May 6 11:10:01 BackupStorage pveproxy[4110]: worker 23255 started
May 6 11:10:01 BackupStorage pveproxy[23254]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.
May 6 11:10:01 BackupStorage pveproxy[23255]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.
May 6 11:10:01 BackupStorage cron[3784]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)

 

[oguz]

hi,

changing the hostname requires manual modification of the /etc/hosts and some other files [0]. beware if this is a cluster you will also have to edit other files like corosync configuration (see the link below). you can try changing it back to the original like this.

when you run hostname --ip-address you should be seeing the correct IP address that you will access PVE on (for example the public IP or the local IP in the network). if this works then you can restart the services and things should come back up.

[0]: https://pve.proxmox.com/wiki/Renaming_a_PVE_node

 

[adam99]

hi,

changing the hostname requires manual modification of the /etc/hosts and some other files [0]. beware if this is a cluster you will also have to edit other files like corosync configuration (see the link below). you can try changing it back to the original like this.

when you run hostname --ip-address you should be seeing the correct IP address that you will access PVE on (for example the public IP or the local IP in the network). if this works then you can restart the services and things should come back up.

[0]: https://pve.proxmox.com/wiki/Renaming_a_PVE_node

Ok, I have changed it correspondly, and hostname --ip-address return the IP of the last Virtual machine which was deployed by terrafrom even though i changed /etc/hosts file to proper IP address of Hypervisor.

 

[oguz]

can you post the output from cat /etc/hosts? it should look like here [0]

[0]: https://pve.proxmox.com/wiki/Instal...d_an_.2Fetc.2Fhosts_entry_for_your_IP_address

 

[adam99]

root@BackupStorage:~# cat /etc/hosts
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
# /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.0.1 localhost.localdomain localhost
192.168.8.18 pv18.mydomain.net pv18

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

root@BackupStorage:~# hostname --ip-address
172.29.11.102


Where mydomain is the real domain which exists.

 

[oguz]

can you also do hostnamectl set-hostname pv18.mydomain.net and try again with hostname?

 

[adam99]

can you also do hostnamectl set-hostname pv18.mydomain.net and try again with hostname?

Ok, now hostname --ip-address returns proper ip address, but web GUI is still not available and the syslog says:
root@BackupStorage:~# tail /var/log/syslog
May 6 13:54:13 BackupStorage pveproxy[4134]: starting 2 worker(s)
May 6 13:54:13 BackupStorage pveproxy[4134]: worker 15214 started
May 6 13:54:13 BackupStorage pveproxy[4134]: worker 15215 started
May 6 13:54:13 BackupStorage pveproxy[15214]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.
May 6 13:54:13 BackupStorage pveproxy[15215]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.
May 6 13:54:14 BackupStorage pveproxy[15166]: worker exit
May 6 13:54:14 BackupStorage pveproxy[4134]: worker 15166 finished
May 6 13:54:14 BackupStorage pveproxy[4134]: starting 1 worker(s)
May 6 13:54:14 BackupStorage pveproxy[4134]: worker 15216 started
May 6 13:54:14 BackupStorage pveproxy[15216]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.

 

[oguz]

Ok, now hostname --ip-address returns proper ip address

great!

but web GUI is still not available and the syslog says:

have you restarted the services? you could run systemctl restart pveproxy pvedaemon and then check with ss -antlp | grep 8006 if the port is open.

if that doesn't fix the issue please try rebooting

 

[adam99]

great!


have you restarted the services? you could run systemctl restart pveproxy pvedaemon and then check with ss -antlp | grep 8006 if the port is open.

if that doesn't fix the issue please try rebooting

Yeah, GUI worked, but all VMs were in 'unknown state', after reboot machine came back to being one of the virtual machines. I thought it was terraform fault, but tested it on different Proxmox Hypervisor and it does not happen.

 

[oguz]

after reboot machine came back to being one of the virtual machines.

maybe terraform left a cron job or some script to be run at reboot? i would look in this direction to find out what's being run.

i would also suggest you to disable and stop terraform and re-do the steps to fix the hostname and GUI again.

but all VMs were in 'unknown state',

this might be another issue so we can take a look at that once the gui starts working again

 

[adam99]

maybe terraform left a cron job or some script to be run at reboot? i would look in this direction to find out what's being run.

i would also suggest you to disable and stop terraform and re-do the steps to fix the hostname and GUI again.


this might be another issue so we can take a look at that once the gui starts working again

Checked for any scripts and cron jobs for all users - there is none. Turned off machine from which terraform was executed, Changed everything on Proxmox to make GUI work again, then restarted machine and the same problem occurs.

 

[oguz]

# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
# /etc/cloud/cloud.cfg or cloud-config from user-data
#

did you install cloud init?

that could explain why the /etc/hosts file gets reset at reboot...

 

[oguz]

you should uninstall cloud-init from the host, and fix the

* /etc/network/interfaces
* /etc/hostname
* /etc/hosts

files. and then after the reboot the changes should persist normally without intervention from cloud-init

 

[adam99]

you should uninstall cloud-init from the host, and fix the

* /etc/network/interfaces
* /etc/hostname
* /etc/hosts

files. and then after the reboot the changes should persist normally without intervention from cloud-init

It worked, thank you! But why cloud-init caused this strange behaviour?

 

[fiona]

Hi,

It worked, thank you! But why cloud-init caused this strange behaviour?

cloud-init is there to initialize system state and configuration, so it can mess up your hosts configuration. With PVE, you (normally) want to install cloud-init only within guests where needed.

 

[oguz]

It worked, thank you!

great, in that case i mark this thread solved