rewriting the hosts file zapped /etc/pve/*, now it's back

May 16, 2020
256
16
38
50
Antwerp, Belgium
commandline.be
Dear,

Happy with proxmox i do find myself in a sudden challenge of sorts. As I found some prior posts on this I share the experience here. This is most recent proxmox running with ZFS.

I have no memory of changing anything when xterm.js console refused and only showed a message+spinner is it connecting, noVNC works fine throughout.

My /etc/hosts file was rewritten to include the hostname for the 127.0.0.1 host, this hostname was only known to the <non-routable-ip> in the form <hostname>.<sub>.<domain>.<tld>

After reboot this change resulted in a full-down for almost all PVE services. Despite reinstalling software the PVE setup remained broken. Main reason was all data in /etc/pve/* was gone and no reinstall fixed this. I even manually recreated the certificates.

After fooling myself for a few hours it became sensible to restore /etc/hosts to the prior state and reboot. Now the data is back. noVNC still works, xterm.js still fails.

Now, I noticed the .pem and .key files in /etc/pve are nnamed pve-ssl.key and pve-ssl.pem which is not entirely in-line with the repective name i saw mentioned in logs. It may be something is broken here ?

I've just reinstalled pve-manager which worked fine. I still hope to recover use for xterm.js and request some feedback.

I also'd like to know how data can be gone and not-gone with ZFS by adding a hostname to the localhost-IP which was not configured anywhere before.


Thanks

JL
 
I also'd like to know how data can be gone and not-gone with ZFS by adding a hostname to the localhost-IP which was not configured anywhere before.
Never change the localhost string in /etc/hosts . The behavior with bad localhost entry is simply undefined and likely brakes the OS at many levels.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
This is somewhat informative https://pve.proxmox.com/wiki/Separate_Cluster_Network#Configure_hosts_file

Typically an /etc/hosts file can be modified with predictable outcomes. The big mistery to me is how this caused ZFS to omit /etc/pve/* entirely while the data base never altered and no change was instructed to ZFS at any time.

To illustrate, from the link shared above

127.0.0.1 localhost.localdomain localhost
192.168.15.151 one.proxmox.com one pvelocalhost

was changed into

127.0.0.1 localhost.localdomain localhost one
192.168.15.151 one.proxmox.com one pvelocalhost

as it was late i had not acknowledged 'one' was already assigned, the reason for doing so was a message suggesting to do something of that order and my desire to not have this host accessible anymore from the 192.168.15.151 IP
 
The big mistery to me is how this caused ZFS to omit /etc/pve/* entirely

Maybe because of this:
Hi,

the /etc/hosts should look like this.
[...]
It is normal if this file is not correct the Proxmox VE services can't start.
Be sure that the hostname is the same as in /etc/hostname
https://forum.proxmox.com/threads/rebooted-proxmox-and-messed-up-etc-hosts-file.63268/#post-288036
which lead to not starting this:
https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)
?
 
/etc/pve is not on ZFS, pmxcfs is always mounted there, no matter whether your system is a standalone host or part of a cluster. pmxcfs can only start up if the hostname resolves to a non-loopback IP (and it will log an error if it cannot start).
 
  • Like
Reactions: Neobin and Joris L.
/etc/pve is not on ZFS, pmxcfs is always mounted there, no matter whether your system is a standalone host or part of a cluster. pmxcfs can only start up if the hostname resolves to a non-loopback IP (and it will log an error if it cannot start).

Thanks. That error escaped me. I was only confronted with a seemingly wiped folder.
is there a documented procuedure on how to change the non-loopback iP ?
 
there is no such procedure - pmxcfs will simply resolve the local hostname on startup, and error out if that fails or only returns a loopback address.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!