Issues after renaming node

[instantautomation]

I followed the instructions here https://pve.proxmox.com/wiki/Renaming_a_PVE_node and the name changed but all of my container/VM configs were not there. After checking into /etc/pve/nodes the old node was there along with the new. I moved my configs into the new nodes folder, verified they were there and rebooted. Now I think I broke some stuff because I cannot connect to the gui and I cannot update certs. I get an error saying the directory doesn't exist. Also the /etc/pve/nodes doesn't exist either. I can connect via ssh.

 

[fiona]

Hi,
please share the output of the following:

cat /etc/hostname
cat /etc/hosts
systemctl pve-cluster.service
pveversion -v

 

[instantautomation]

Thanks for your reply,

Info as requested.

root@Creator:~# cat /etc/hostname
Creator

root@Creator:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.88.10 Creator.Proxmox.Server Creator

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

root@Creator:~# systemctl pve-cluster.service
Unknown command verb pve-cluster.service.

root@Creator:~# pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-4-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-7
pve-kernel-helper: 7.0-7
pve-kernel-5.4: 6.4-6
pve-kernel-5.11.22-4-pve: 5.11.22-8
pve-kernel-5.4.140-1-pve: 5.4.140-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-6
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-11
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.9-2
proxmox-backup-file-restore: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-3
pve-firmware: 3.3-1
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

 

[fiona]

Thanks for your reply,

Info as requested.

root@Creator:~# cat /etc/hostname
Creator

root@Creator:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.88.10 Creator.Proxmox.Server Creator

Is the IP actively configured on the host, i.e. shows up in the output of ip addr?

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

root@Creator:~# systemctl pve-cluster.service
Unknown command verb pve-cluster.service.

Sorry, I meant systemctl status pve-cluster.service. If there's an error journalctl -b0 -u pve-cluster.service might show a bit more.

root@Creator:~# pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-4-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-7
pve-kernel-helper: 7.0-7
pve-kernel-5.4: 6.4-6
pve-kernel-5.11.22-4-pve: 5.11.22-8
pve-kernel-5.4.140-1-pve: 5.4.140-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-6
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-11
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.9-2
proxmox-backup-file-restore: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-3
pve-firmware: 3.3-1
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

 

[instantautomation]

root@Creator:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
link/ether b4:2e:99:ec:5a:9c brd ff:ff:ff:ff:ff:ff
altname enp4s0
3: enp6s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000
link/ether a0:36:9f:8a:6b:e4 brd ff:ff:ff:ff:ff:ff
4: enp6s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr2 state DOWN group default qlen 1000
link/ether a0:36:9f:8a:6b:e5 brd ff:ff:ff:ff:ff:ff
5: enp6s0f2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether a0:36:9f:8a:6b:e6 brd ff:ff:ff:ff:ff:ff
6: enp6s0f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether a0:36:9f:8a:6b:e7 brd ff:ff:ff:ff:ff:ff
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 12:d2:a2:10:2b:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.88.10/24 brd 192.168.88.255 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::10d2:a2ff:fe10:2b45/64 scope link
valid_lft forever preferred_lft forever
8: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 86:f3:02:21:61:3e brd ff:ff:ff:ff:ff:ff
9: vmbr2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether a2:4f:8b:9d:ae:09 brd ff:ff:ff:ff:ff:ff


root@Creator:~# systemctl status pve-cluster.service
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2021-09-20 06:03:23 MST; 12h ago
Process: 5845 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
CPU: 6ms

Sep 20 06:03:23 Creator systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Sep 20 06:03:23 Creator systemd[1]: Stopped The Proxmox VE cluster filesystem.
Sep 20 06:03:23 Creator systemd[1]: pve-cluster.service: Start request repeated too quickly.
Sep 20 06:03:23 Creator systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 20 06:03:23 Creator systemd[1]: Failed to start The Proxmox VE cluster filesystem.

 

[fiona]

If there's an error journalctl -b0 -u pve-cluster.service might show a bit more.

 

[instantautomation]

root@Creator:~# journalctl -b0 -u pve-cluster.service
-- Journal begins at Sun 2021-09-19 13:45:27 MST, ends at Wed 2021-09-22 04:49:57 MST. --
Sep 20 04:30:26 Creator systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 20 04:30:26 Creator pmxcfs[1757]: [database] crit: found entry with duplicate name 'lxc' - Ainode = 0x00000000007CCDC7, parent = 0x00000000007C>
Sep 20 04:30:26 Creator pmxcfs[1757]: [database] crit: DB load failed
Sep 20 04:30:26 Creator pmxcfs[1757]: [database] crit: found entry with duplicate name 'lxc' - Ainode = 0x00000000007CCDC7, parent = 0x00000000007C>
Sep 20 04:30:26 Creator pmxcfs[1757]: [database] crit: DB load failed
Sep 20 04:30:26 Creator pmxcfs[1757]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 04:30:26 Creator pmxcfs[1757]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 04:30:26 Creator pmxcfs[1757]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 04:30:26 Creator pmxcfs[1757]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 04:30:26 Creator systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 20 04:30:26 Creator systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 20 04:30:26 Creator systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 20 04:30:27 Creator systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Sep 20 04:30:27 Creator systemd[1]: Stopped The Proxmox VE cluster filesystem.
Sep 20 04:30:27 Creator systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 20 04:30:27 Creator pmxcfs[1905]: [database] crit: found entry with duplicate name 'lxc' - Ainode = 0x00000000007CCDC7, parent = 0x00000000007C>
Sep 20 04:30:27 Creator pmxcfs[1905]: [database] crit: found entry with duplicate name 'lxc' - Ainode = 0x00000000007CCDC7, parent = 0x00000000007C>
Sep 20 04:30:27 Creator pmxcfs[1905]: [database] crit: DB load failed
Sep 20 04:30:27 Creator pmxcfs[1905]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 04:30:27 Creator pmxcfs[1905]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 04:30:27 Creator pmxcfs[1905]: [database] crit: DB load failed
Sep 20 04:30:27 Creator pmxcfs[1905]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 04:30:27 Creator pmxcfs[1905]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 04:30:27 Creator systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 20 04:30:27 Creator systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 20 04:30:27 Creator systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 20 04:30:27 Creator systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 2.
Sep 20 04:30:27 Creator systemd[1]: Stopped The Proxmox VE cluster filesystem.
Sep 20 04:30:27 Creator systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 20 04:30:27 Creator pmxcfs[1908]: [database] crit: found entry with duplicate name 'lxc' - Ainode = 0x00000000007CCDC7, parent = 0x00000000007C>

 

[fiona]

Please follow the steps described in this thread closely (the later half starting from the post the link points to). If there are questions, feel free to ask.

 

[instantautomation]

Thanks for the help,
I ran into this error after trying to start the pve-cluster


Sep 24 09:05:25 Creator pveproxy[1919]: worker 256127 finished
Sep 24 09:05:25 Creator pveproxy[1919]: starting 1 worker(s)
Sep 24 09:05:25 Creator pveproxy[1919]: worker 256130 started
Sep 24 09:05:25 Creator pveproxy[256130]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1891.

There are several line of the same error.

I think this is where my problem is [database] crit: found entry with duplicate name 'lxc'
Although I am unsure of where to look to remove the duplicate.

 

[instantautomation]

BTW I do not get an output of this command

sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE parent = (SELECT parent FROM tree WHERE inode = 12430028)'

And this is the output of the first

root@Creator:~# sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE name = "authorized_keys"'
8179152|9|1632074413|8|authorized_keys

 

[fiona]

I think this is where my problem is [database] crit: found entry with duplicate name 'lxc'

Although I am unsure of where to look to remove the duplicate.

Yes, to fix the database it's hopefully enough to remove that duplicate.

BTW I do not get an output of this command

sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE parent = (SELECT parent FROM tree WHERE inode = 12430028)'

In your case the inode is 0x00000000007CCDC7 (can be seen in your log) which can be converted to decimal with e.g.

echo "obase=10; ibase=16; 00000000007CCDC7" | bc

and that gives 8179143, so you need to use that instead of 12430028.

And this is the output of the first

root@Creator:~# sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE name = "authorized_keys"'
8179152|9|1632074413|8|authorized_keys

Here you need to use lxc instead of authorized_keys, because that's the duplicate entry causing problems.

 

[instantautomation]

Alright after making those changes here is the error I'm getting

-- Journal begins at Sun 2021-09-19 13:45:27 MST, ends at Wed 2021-09-29 10:24:35 MST. --
Sep 29 10:21:48 Creator systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 29 10:21:48 Creator pmxcfs[1418]: fuse: mountpoint is not empty
Sep 29 10:21:48 Creator pmxcfs[1418]: fuse: if you are sure this is safe, use the 'nonempty' mount option
Sep 29 10:21:48 Creator pmxcfs[1418]: [main] crit: fuse_mount error: File exists
Sep 29 10:21:48 Creator pmxcfs[1418]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 29 10:21:48 Creator pmxcfs[1418]: [main] crit: fuse_mount error: File exists
Sep 29 10:21:48 Creator pmxcfs[1418]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 29 10:21:48 Creator systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 29 10:21:48 Creator systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 29 10:21:48 Creator systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 29 10:21:48 Creator systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Sep 29 10:21:48 Creator systemd[1]: Stopped The Proxmox VE cluster filesystem.

 

[fiona]

Alright after making those changes here is the error I'm getting

-- Journal begins at Sun 2021-09-19 13:45:27 MST, ends at Wed 2021-09-29 10:24:35 MST. --
Sep 29 10:21:48 Creator systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 29 10:21:48 Creator pmxcfs[1418]: fuse: mountpoint is not empty

Sounds like something has been written to /etc/pve while the cluster filesystem was not mounted there. Try moving those files somewhere else (to inspect them) and try starting the service again.

Sep 29 10:21:48 Creator pmxcfs[1418]: fuse: if you are sure this is safe, use the 'nonempty' mount option
Sep 29 10:21:48 Creator pmxcfs[1418]: [main] crit: fuse_mount error: File exists
Sep 29 10:21:48 Creator pmxcfs[1418]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 29 10:21:48 Creator pmxcfs[1418]: [main] crit: fuse_mount error: File exists
Sep 29 10:21:48 Creator pmxcfs[1418]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 29 10:21:48 Creator systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 29 10:21:48 Creator systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 29 10:21:48 Creator systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 29 10:21:48 Creator systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Sep 29 10:21:48 Creator systemd[1]: Stopped The Proxmox VE cluster filesystem.

 

[instantautomation]

So I have tried moving all files aside from the SSL key stuff and I am still getting the same error. Do you have any other suggestions?

 

[fiona]

So I have tried moving all files aside from the SSL key stuff and I am still getting the same error. Do you have any other suggestions?

The mount point /etc/pve has to be completely empty. Just move the files back (if they are not already present) after the cluster file system has been mounted there.

 

[instantautomation]

Took me a while to get back to this. Moving everything allowed the file system to start. now I just need to restore my configs. Thank you for all your help!