/etc/pve empty after trying to change hostname, possible to salvage LXCs/VMs?

[dmpm]

I tried changing the hostname in /etc/hostname and /etc/hosts and then copying the contents of /var/lib/rrdcached/db/pve2-{node,storage}/old-hostname to /var/lib/rrdcached/db/pve2-{node,storage}/new-hostname, as described here https://pve.proxmox.com/wiki/Renaming_a_PVE_node

but after rebooting all my LXCs/VMs were listed under the old hostname/node and I couldn't access them. So I then tried copying the files in /etc/pve/old-hostname to new-hostname, but after rebooting /etc/pve is completely empty!

Is there anything I can do to salvage my LXCs/VMs so that I don't have to start from scratch? As shown in the attached output from lsblk, the pve-vm--xxx-disk-0 LV still exist and I can mount them to folders in /mnt/ so is there anyway I can backup those and restore them after reinstalling Proxmox?

 

[dmpm]

This is the output from journalctl -b -u pve-cluster

Jul 20 19:58:41 visuals systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Jul 20 19:58:41 visuals pmxcfs[1319]: [main] notice: resolved node name 'visuals' to '192.168.1.199' for default node IP address
Jul 20 19:58:41 visuals pmxcfs[1319]: [main] notice: resolved node name 'visuals' to '192.168.1.199' for default node IP address
Jul 20 19:58:41 visuals pmxcfs[1319]: [database] crit: found entry with duplicate name 'lxc' - A:(inode = 0x00000000000ACFCC, parent = 0x00000000000ACFCA, v./mtime = 0xACFCC/0x1721499303) vs. B:(inode = 0x00000000000AD044, parent = 0x00000000000ACFCA, v./mtime = 0xAD044/0x1721500567)
Jul 20 19:58:41 visuals pmxcfs[1319]: [database] crit: found entry with duplicate name 'lxc' - A:(inode = 0x00000000000ACFCC, parent = 0x00000000000ACFCA, v./mtime = 0xACFCC/0x1721499303) vs. B:(inode = 0x00000000000AD044, parent = 0x00000000000ACFCA, v./mtime = 0xAD044/0x1721500567)
Jul 20 19:58:41 visuals pmxcfs[1319]: [database] crit: DB load failed
Jul 20 19:58:41 visuals pmxcfs[1319]: [database] crit: DB load failed
Jul 20 19:58:41 visuals pmxcfs[1319]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jul 20 19:58:41 visuals pmxcfs[1319]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 20 19:58:41 visuals pmxcfs[1319]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jul 20 19:58:41 visuals pmxcfs[1319]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 20 19:58:41 visuals systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jul 20 19:58:41 visuals systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jul 20 19:58:41 visuals systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.
Jul 20 19:58:41 visuals systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Jul 20 19:58:41 visuals systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Jul 20 19:58:41 visuals systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...

 

[dmpm]

Also ran these commands. The last one didn't produce any output, so that doesn't seem to identify the problem.

root@visuals:/home# sqlite3 /var/lib/pve-cluster/config.db 'PRAGMA integrity_check'
ok

root@visuals:/home# sqlite3 /var/lib/pve-cluster/config.db .schema
CREATE TABLE tree (  inode INTEGER PRIMARY KEY NOT NULL,  parent INTEGER NOT NULL CHECK(typeof(parent)=='integer'),  version INTEGER NOT NULL CHECK(typeof(version)=='integer'),  writer INTEGER NOT NULL CHECK(typeof(writer)=='integer'),  mtime INTEGER NOT NULL CHECK(typeof(mtime)=='integer'),  type INTEGER NOT NULL CHECK(typeof(type)=='integer'),  name TEXT NOT NULL,  data BLOB);

root@visuals:/home# sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,mtime,name FROM tree WHERE parent = 0'
0|1721501160|__version__
6|1703243829|datacenter.cfg
8|1703243870|virtual-guest
9|1703243871|priv
11|1703243871|nodes
24|1703243872|pve-www.key
30|1703243874|pve-root-ca.pem
49|1703243874|firewall
50|1703243874|ha
51|1703243874|mapping
53|1703243874|sdn
124937|1712189923|user.cfg
706669|1721436853|replication.cfg
706673|1721436853|jobs.cfg
706678|1721436853|vzdump.cron
707352|1721483783|authkey.pub.old
707355|1721483783|authkey.pub
707697|1721486793|storage.cfg

root@visuals:/home# sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,mtime,name FROM tree WHERE parent = 347747 or inode = 347747'
root@visuals:/home#

 

[dmpm]

I found that the problem was there was multiple lxc and qemu-server entries:

sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE name = "lxc"'
13|12|1703243871|4|lxc
708556|708554|1721499303|4|lxc
708676|708554|1721500567|4|lxc
708735|708734|1721500957|4|lxc

sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE name = "qemu-server"'
14|12|1703243871|4|qemu-server
708555|708554|1721499303|4|qemu-server
708725|708554|1721500830|4|qemu-server

So I deleted the last qemu-server entry and the last two lxc entries and that got it working, with everything showing under the old hostname again.