[SOLVED] Changed cluster hostname, proxmox webserver not working after

[LECOQQ]

Hello
I'm here to ask for help: i've changed my hostname today, following this tutorial: https://pve.proxmox.com/wiki/Renaming_a_PVE_node
After changeing /etc/hosts & /etc/hostname from bonaparte to mortier, I did the clean up by moving what was inside of /etc/pve/nodes from the old node name to the new one. I then copied the /var/lib/rrdcached/db/pve2-{node,storage}/bonaparte to /var/lib/rrdcached/db/pve2-{node,storage}/mortier. I then rebooted the homelab.
After this, the web interface was not working. I could still ssh to the homelab, so I did.
Nothing was left in /etc/pve/nodes. A quick search showed me this : https://forum.proxmox.com/threads/hostname-changed-now-nodes-gone-from-etc-pve.111607/
I tried what was proposed - pmxcfs -l -d -f, and I got:
[main] notice: resolved node name 'mortier' to '192.168.1.46' for default node IP address (pmxcfs.c:859:main)
[database] debug: name __version__ (inode = 0000000000000000, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name datacenter.cfg (inode = 0000000000000002, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name user.cfg (inode = 0000000000000004, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name storage.cfg (inode = 0000000000000006, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name virtual-guest (inode = 0000000000000008, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name priv (inode = 0000000000000009, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name nodes (inode = 000000000000000B, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name lxc (inode = 000000000000000D, parent = 000000000019FB30) (database.c:370:bdb_backend_load_index)
[database] debug: name openvz (inode = 000000000000000F, parent = 000000000019FB30) (database.c:370:bdb_backend_load_index)
[database] debug: name priv (inode = 0000000000000010, parent = 000000000019FB30) (database.c:370:bdb_backend_load_index)
[database] debug: name lock (inode = 0000000000000011, parent = 0000000000000009) (database.c:370:bdb_backend_load_index)
[database] debug: name pve-www.key (inode = 0000000000000018, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name pve-ssl.key (inode = 000000000000001A, parent = 000000000019FB30) (database.c:370:bdb_backend_load_index)
[database] debug: name pve-root-ca.key (inode = 000000000000001C, parent = 0000000000000009) (database.c:370:bdb_backend_load_index)
[database] debug: name pve-root-ca.pem (inode = 000000000000001E, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name pve-root-ca.srl (inode = 0000000000000020, parent = 0000000000000009) (database.c:370:bdb_backend_load_index)
[database] debug: name pve-ssl.pem (inode = 0000000000000023, parent = 000000000019FB30) (database.c:370:bdb_backend_load_index)
[database] debug: name vzdump.cron (inode = 000000000000002D, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name firewall (inode = 0000000000000030, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name ha (inode = 0000000000000031, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name mapping (inode = 0000000000000032, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name acme (inode = 0000000000000033, parent = 0000000000000009) (database.c:370:bdb_backend_load_index)
[database] debug: name sdn (inode = 0000000000000034, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name 100.conf (inode = 0000000000000534, parent = 000000000019FBFB) (database.c:370:bdb_backend_load_index)
[database] debug: name ssh_known_hosts (inode = 00000000000CB85E, parent = 000000000019FB30) (database.c:370:bdb_backend_load_index)
[database] debug: name authkey.pub.old (inode = 000000000019E6E2, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name authkey.pub (inode = 000000000019E6E5, parent = 0000000000000000) (database.c:370:bdb_backend_load_index)
[database] debug: name authkey.key (inode = 000000000019E6E8, parent = 0000000000000009) (database.c:370:bdb_backend_load_index)
[database] debug: name mortier (inode = 000000000019FB30, parent = 000000000000000B) (database.c:370:bdb_backend_load_index)
[database] debug: name qemu-server (inode = 000000000019FB31, parent = 000000000019FB30) (database.c:370:bdb_backend_load_index)
[database] debug: name authorized_keys (inode = 000000000019FB3B, parent = 0000000000000009) (database.c:370:bdb_backend_load_index)
[database] debug: name qemu-server (inode = 000000000019FBFB, parent = 000000000019FB30) (database.c:370:bdb_backend_load_index)
[database] crit: found entry with duplicate name 'qemu-server' - Ainode = 0x000000000019FB31, parent = 0x000000000019FB30, v./mtime = 0x19FB31/0x1718973314) vs. Binode = 0x000000000019FBFB, parent = 0x000000000019FB30, v./mtime = 0x19FBFB/0x1718973575) (database.c:425:bdb_backend_load_index)
[database] crit: DB load failed (database.c:465:bdb_backend_load_index)
[main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db' (pmxcfs.c:898:main)
[main] notice: exit proxmox configuration filesystem (-1) (pmxcfs.c:1109:main)

From what I understand, the issue is due to duplicate qemu-server inside the DB. This is exactly the same case as in the link above, the answer brought by mira was to edit manually the db. Though I don't know at all how to do this.
I've went to this link after: https://forum.proxmox.com/threads/pve-cluster-fails-to-start.82861/
I found the DB at /var/lib/pve-cluster, and made a back-up thanks to t.lamprecht commands. Then I did the recommanded sanity checks:

sqlite3 /var/lib/pve-cluster/config.db 'PRAGMA integrity_check'
sqlite3 /var/lib/pve-cluster/config.db .schema
sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,mtime,name FROM tree WHERE parent = 0'
sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,mtime,name FROM tree WHERE parent = 347747 or inode = 347747'

Which resulted in :

sqlite3 /var/lib/pve-cluster/config.db 'PRAGMA integrity_check'

ok

sqlite3 /var/lib/pve-cluster/config.db .schema

CREATE TABLE tree ( inode INTEGER PRIMARY KEY NOT NULL, parent INTEGER NOT NULL CHECK(typeof(parent)=='integer'), version INTEGER NOT NULL CHECK(typeof(version)=='integer'), writer INTEGER NOT NULL CHECK(typeof(writer)=='integer'), mtime INTEGER NOT NULL CHECK(typeof(mtime)=='integer'), type INTEGER NOT NULL CHECK(typeof(type)=='integer'), name TEXT NOT NULL, data BLOB);

sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,mtime,name FROM tree WHERE parent = 0'

0|1718974040|__version__
2|1715006834|datacenter.cfg
4|1715006834|user.cfg
6|1715006834|storage.cfg
8|1715006872|virtual-guest
9|1715006873|priv
11|1715006873|nodes
24|1715006874|pve-www.key
30|1715006880|pve-root-ca.pem
45|1715006880|vzdump.cron
48|1715006880|firewall
49|1715006880|ha
50|1715006880|mapping
52|1715006880|sdn
1697506|1718965495|authkey.pub.old
1697509|1718965495|authkey.pub

sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,mtime,name FROM tree WHERE parent = 347747 or inode = 347747'

It seems to me that everything is good there.
After going into the DB, i tried to find what was in it. I did :

select * from tree;

And got my data, including rsa keys, etc. On the last lines, I indeed have 2 qemu-servers :

1702704|11|1702704|0|1718973314|4|mortier|
1702705|1702704|1702705|0|1718973314|4|qemu-server|
1702907|1702704|1702907|0|1718973575|4|qemu-server|
1703222|1702704|1703224|0|1718974040|8|lrm_status|{"state":"wait_for_agent_lock","mode":"restart","timestamp":1718974040,"results":{}}

But after this I don't know what to do. Which one is the good, and which one is the one giving me issues? How to delete the one ruining my moral?

Thanks a lot for any help anyone could provide !

Edit: Deleted private keys
Edit 2: Formating

 

[LECOQQ]

Hey, it's me - again.
I managed to solve my issue.

Here is a quick breakdown of what I did to solve it. During the process I learnt a lot about sqlite3.
Basically, it was all about getting to understand the /var/lib/pve-cluster/config.db file. By checking what's inside the DB, we can see there is only one table, w/ the command:

sqlite3 /var/lib/pve-cluster/config.db '.table'
sqlite3 /var/lib/pve-cluster/config.db '.schema'

CREATE TABLE tree (
inode INTEGER PRIMARY KEY NOT NULL,
parent INTEGER NOT NULL CHECK(typeof(parent)=='integer'),
version INTEGER NOT NULL CHECK(typeof(version)=='integer'),
writer INTEGER NOT NULL CHECK(typeof(writer)=='integer'),
mtime INTEGER NOT NULL CHECK(typeof(mtime)=='integer'),
type INTEGER NOT NULL CHECK(typeof(type)=='integer'),
name TEXT NOT NULL, data BLOB
);

Which means that every entry has this format in the table tree. When trying to check if there was two qemu-server entry, w/ the command :

sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE name = "qemu-server"'

I got:
1702705|1702704|1718973314|4|qemu-server
1702907|1702704|1718973575|4|qemu-server

I backuped the DB w/

tar czf pve-cluster-bak.tgz -C /var/lib/pve-cluster ./

Then deleted randomly one of the qemu node:

sqlite3 /var/lib/pve-cluster/config.db 'DELETE FROM tree WHERE name = "qemu-server" AND inode = 1702907'

I tried to run proxmox w/

pmxcfs -l -d -f

Obviously it wasn't the good one and got a critical error. I removed the DB and untar-ed my backup. I removed the qemu I hadn't touched and tried the last command again - it worked this time.
Now everything went back to normal, and the hostname of the cluster is good.
I now have to change the node hostname. I'll first back-up the nodes and pve-cluster - just in case.
Thanks