/etc/pve wiped...

[cvalcke]

on a dell r620 file server (running Truenas over PM), I upgraded the network interface to a "2x 10G, 2x1G" card. The install and reboot went well. With vmbr0 operating, I was able to reconnect via web and ssh. Thinking the hard part was done, I added vmbr1 for second 10G port, no issues. When adding vmbr2 for the third port ("Apply configuration"), I lost web GUI
Searching through forums and some logs (SSH still works), I discovered that not only a key was missing from /etc/pve (cause of UI loss?) but the whole directory is wiped.
I checked /etc/network/interfaces and didn't see anything unusual.

Was I just unlucky, or is something else happening here?
(and yes a backup would come in handy but unfortunately, I'm still getting my PBS up and running)

 

[esi_y]

on a dell r620 file server (running Truenas over PM), I upgraded the network interface to a "2x 10G, 2x1G" card. The install and reboot went well. With vmbr0 operating, I was able to reconnect via web and ssh. Thinking the hard part was done, I added vmbr1 for second 10G port, no issues. When adding vmbr2 for the third port ("Apply configuration"), I lost web GUI
Searching through forums and some logs (SSH still works), I discovered that not only a key was missing from /etc/pve (cause of UI loss?) but the whole directory is wiped.
I checked /etc/network/interfaces and didn't see anything unusual.

Is there anything interesting in journalctl -u pve-cluster?

Was I just unlucky, or is something else happening here?

Can you resolve hostname --ip-address as expected after the network changes?

 

[cvalcke]

some entries from journalctl:

Sep 24 02:52:50 dellr620 pmxcfs[141862]: [main] crit: Unable to resolve node name 'dellr620' to a non-loopback IP addre>
Sep 24 02:52:50 dellr620 systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 24 02:52:50 dellr620 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 24 02:52:50 dellr620 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.
Sep 24 02:52:50 dellr620 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Sep 24 02:52:50 dellr620 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Sep 24 02:52:50 dellr620 systemd[1]: pve-cluster.service: Start request repeated too quickly.
Sep 24 02:52:50 dellr620 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 24 02:52:50 dellr620 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.

not sure on your second statement, below is results for "IP a" (only 1 port is attached to router)

root@dellr620:/etc/pve# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether d4:ae:52:7a:b3:22 brd ff:ff:ff:ff:ff:ff
altname enp1s0f0
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d4:ae:52:7a:b3:24 brd ff:ff:ff:ff:ff:ff
altname enp1s0f1
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d4:ae:52:7a:b3:26 brd ff:ff:ff:ff:ff:ff
altname enp1s0f2
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d4:ae:52:7a:b3:28 brd ff:ff:ff:ff:ff:ff
altname enp1s0f3
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d4:ae:52:7a:b3:22 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.57/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::d6ae:52ff:fe7a:b322/64 scope link
valid_lft forever preferred_lft forever

hostname returns "dellr620"
hostname -I returns correct IP address (192.168.1.57)
hostname -i returns "hostname: Name or service not known"

 

[esi_y]

some entries from journalctl:

Yes, so the reason your /etc/pve does not get mounted is:

pmxcfs[141862]: [main] crit: Unable to resolve node name 'dellr620' to a non-loopback IP addre>

not sure on your second statement, below is results for "IP a" (only 1 port is attached to router)

hostname returns "dellr620"
hostname -I returns correct IP address (192.168.1.57)

What's in your /etc/hosts?

 

[cvalcke]

looks like you found the problem.... /etc/hosts content below, no mention of r620...

root@dellr620:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
root@dellr620:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

 

[esi_y]

looks like you found the problem....

I have luck for spotting these posts these days

https://forum.proxmox.com/threads/dhcp-pve-install-on-top-of-debian-not-well-documented.154467

/etc/hosts content below, no mention of r620...

Add it then, and systemctl restart pve-cluster.

 

[esi_y]

NB I have referenced this thread from within bug 5670 [1], it seems to be a recurrent theme, quite understandably in my opinion (who puts together disappeared config directory on a non-cluster install with obscure changes in /etc/hosts, file that is not meant to be really used in this century).

Another thing, especially that the backend database likes to get corrupted and you have no backup on single node install, you might want to have a look at this (stub) tutorial [2] on backing it up.

PS Please set this thread as solved once you confirm issue resolved (you can do so by editing the title).

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=5670
[2] https://forum.proxmox.com/threads/backup-cluster-config-pmxcfs-etc-pve.154569/