rewriting the hosts file zapped /etc/pve/*, now it's back

[Joris L.]

Dear,

Happy with proxmox i do find myself in a sudden challenge of sorts. As I found some prior posts on this I share the experience here. This is most recent proxmox running with ZFS.

I have no memory of changing anything when xterm.js console refused and only showed a message+spinner is it connecting, noVNC works fine throughout.

My /etc/hosts file was rewritten to include the hostname for the 127.0.0.1 host, this hostname was only known to the <non-routable-ip> in the form <hostname>.<sub>.<domain>.<tld>

After reboot this change resulted in a full-down for almost all PVE services. Despite reinstalling software the PVE setup remained broken. Main reason was all data in /etc/pve/* was gone and no reinstall fixed this. I even manually recreated the certificates.

After fooling myself for a few hours it became sensible to restore /etc/hosts to the prior state and reboot. Now the data is back. noVNC still works, xterm.js still fails.

Now, I noticed the .pem and .key files in /etc/pve are nnamed pve-ssl.key and pve-ssl.pem which is not entirely in-line with the repective name i saw mentioned in logs. It may be something is broken here ?

I've just reinstalled pve-manager which worked fine. I still hope to recover use for xterm.js and request some feedback.

I also'd like to know how data can be gone and not-gone with ZFS by adding a hostname to the localhost-IP which was not configured anywhere before.


Thanks

JL

 

[bbgeek17]

I also'd like to know how data can be gone and not-gone with ZFS by adding a hostname to the localhost-IP which was not configured anywhere before.

Never change the localhost string in /etc/hosts . The behavior with bad localhost entry is simply undefined and likely brakes the OS at many levels.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Joris L.]

This is somewhat informative https://pve.proxmox.com/wiki/Separate_Cluster_Network#Configure_hosts_file

Typically an /etc/hosts file can be modified with predictable outcomes. The big mistery to me is how this caused ZFS to omit /etc/pve/* entirely while the data base never altered and no change was instructed to ZFS at any time.

To illustrate, from the link shared above

127.0.0.1 localhost.localdomain localhost
192.168.15.151 one.proxmox.com one pvelocalhost

was changed into

127.0.0.1 localhost.localdomain localhost one
192.168.15.151 one.proxmox.com one pvelocalhost

as it was late i had not acknowledged 'one' was already assigned, the reason for doing so was a message suggesting to do something of that order and my desire to not have this host accessible anymore from the 192.168.15.151 IP

 

[Neobin]

The big mistery to me is how this caused ZFS to omit /etc/pve/* entirely

Maybe because of this:

Hi,

the /etc/hosts should look like this.
[...]
It is normal if this file is not correct the Proxmox VE services can't start.
Be sure that the hostname is the same as in /etc/hostname

https://forum.proxmox.com/threads/rebooted-proxmox-and-messed-up-etc-hosts-file.63268/#post-288036
which lead to not starting this:
https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)
?

 

[Joris L.]

Maybe because of this:

https://forum.proxmox.com/threads/rebooted-proxmox-and-messed-up-etc-hosts-file.63268/#post-288036
which lead to not starting this:
https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)
?

Thanks. To my knowledge I'm not using pmxcfs since it is a stand alone setup.

 

[fabian]

/etc/pve is not on ZFS, pmxcfs is always mounted there, no matter whether your system is a standalone host or part of a cluster. pmxcfs can only start up if the hostname resolves to a non-loopback IP (and it will log an error if it cannot start).

 

[Joris L.]

/etc/pve is not on ZFS, pmxcfs is always mounted there, no matter whether your system is a standalone host or part of a cluster. pmxcfs can only start up if the hostname resolves to a non-loopback IP (and it will log an error if it cannot start).

Thanks. That error escaped me. I was only confronted with a seemingly wiped folder.
is there a documented procuedure on how to change the non-loopback iP ?

 

[fabian]

there is no such procedure - pmxcfs will simply resolve the local hostname on startup, and error out if that fails or only returns a loopback address.

 

[Joris L.]

there is no such procedure - pmxcfs will simply resolve the local hostname on startup, and error out if that fails or only returns a loopback address.

so if i change the non-loopback-ip address to a different non-loopback address and reboot the system all will work well ?

 

[bbgeek17]

do you just want to change LAN IP of your Linux host that is running PVE? Then yes, as long as you update all the right places you wouldn't even need to reboot. Just restart some services.
It gets more complicated in cluster.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox