Need help to repair my pve-cluster/config.db

[rando]

I didint correctly change proxmox hostname, reboote the host and now i have broken pve-cluster. I will add log journalctl -b -u pve-cluster

Sep 20 03:21:02 host systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 20 09:10:27 host systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 20 09:10:27 host pmxcfs[1950]: [database] crit: found entry with duplicate name 'qemu-server' - A:(inode = 0x00000000018DA28B, parent = 0x00000000018DA28A, v./mtime = 0x18DA28B/0x1663617922) vs. B:(inode = 0x00000000018DA438, parent = 0x00000000018D
Sep 20 09:10:27 host pmxcfs[1950]: [database] crit: DB load failed
Sep 20 09:10:27 host pmxcfs[1950]: [database] crit: found entry with duplicate name 'qemu-server' - A:(inode = 0x00000000018DA28B, parent = 0x00000000018DA28A, v./mtime = 0x18DA28B/0x1663617922) vs. B:(inode = 0x00000000018DA438, parent = 0x00000000018D
Sep 20 09:10:27 host pmxcfs[1950]: [database] crit: DB load failed
Sep 20 09:10:27 host pmxcfs[1950]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 09:10:27 host pmxcfs[1950]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 09:10:27 host pmxcfs[1950]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 09:10:27 host pmxcfs[1950]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 20 09:10:27 host systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Sep 20 09:10:27 host systemd[1]: Stopped The Proxmox VE cluster filesystem.
Sep 20 09:10:27 host systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 20 09:10:27 host pmxcfs[1961]: [database] crit: found entry with duplicate name 'qemu-server' - A:(inode = 0x00000000018DA28B, parent = 0x00000000018DA28A, v./mtime = 0x18DA28B/0x1663617922) vs. B:(inode = 0x00000000018DA438, parent = 0x00000000018D
Sep 20 09:10:27 host pmxcfs[1961]: [database] crit: DB load failed
Sep 20 09:10:27 host pmxcfs[1961]: [database] crit: found entry with duplicate name 'qemu-server' - A:(inode = 0x00000000018DA28B, parent = 0x00000000018DA28A, v./mtime = 0x18DA28B/0x1663617922) vs. B:(inode = 0x00000000018DA438, parent = 0x00000000018D
Sep 20 09:10:27 host pmxcfs[1961]: [database] crit: DB load failed
Sep 20 09:10:27 host pmxcfs[1961]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 09:10:27 host pmxcfs[1961]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 09:10:27 host pmxcfs[1961]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 09:10:27 host pmxcfs[1961]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 20 09:10:27 host systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 2.
Sep 20 09:10:27 host systemd[1]: Stopped The Proxmox VE cluster filesystem.
Sep 20 09:10:27 host systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 20 09:10:27 host pmxcfs[1964]: [database] crit: found entry with duplicate name 'qemu-server' - A:(inode = 0x00000000018DA28B, parent = 0x00000000018DA28A, v./mtime = 0x18DA28B/0x1663617922) vs. B:(inode = 0x00000000018DA438, parent = 0x00000000018D
Sep 20 09:10:27 host pmxcfs[1964]: [database] crit: DB load failed
Sep 20 09:10:27 host pmxcfs[1964]: [database] crit: found entry with duplicate name 'qemu-server' - A:(inode = 0x00000000018DA28B, parent = 0x00000000018DA28A, v./mtime = 0x18DA28B/0x1663617922) vs. B:(inode = 0x00000000018DA438, parent = 0x00000000018D
Sep 20 09:10:27 host pmxcfs[1964]: [database] crit: DB load failed
Sep 20 09:10:27 host pmxcfs[1964]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 09:10:27 host pmxcfs[1964]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 09:10:27 host pmxcfs[1964]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 09:10:27 host pmxcfs[1964]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 20 09:10:27 host systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Sep 20 09:10:27 host systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 3.

How to delete duplicated name?

 

[fiona]

Hi,
your issue sounds very similar to the one described here. Please try following the steps suggested below that post.

 

[rando]

Thanks for reply, I already followed other threads, I made some steps like these, now i get new error

sqlite3 config.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> select * from tree where name='qemu-server';
14|12|14|0|1621943568|4|qemu-server|
26059403|26059402|26059403|0|1663617922|4|qemu-server|
26059832|26059402|26059832|0|1663618560|4|qemu-server|
sqlite> ^C^C
┌─[/var/lib/pve-cluster]-root㉾host-(09:53:24)
└─[11]$ sqlite3 /var/lib/pve-cluster/config.db 'DELETE FROM tree WHERE inode = 26059832'

Sep 20 09:10:28 host systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 20 09:54:22 host systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 20 09:54:22 host pmxcfs[18030]: [database] crit: missing directory inode (inode = 00000000018DA438)
Sep 20 09:54:22 host pmxcfs[18030]: [database] crit: missing directory inode (inode = 00000000018DA438)
Sep 20 09:54:22 host pmxcfs[18030]: [database] crit: DB load failed
Sep 20 09:54:22 host pmxcfs[18030]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 09:54:22 host pmxcfs[18030]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 09:54:22 host pmxcfs[18030]: [database] crit: DB load failed
Sep 20 09:54:22 host pmxcfs[18030]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 09:54:22 host pmxcfs[18030]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 09:54:22 host systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 20 09:54:22 host systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 20 09:54:22 host systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 20 09:54:23 host systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Sep 20 09:54:23 host systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Sep 20 09:54:23 host systemd[1]: Stopped The Proxmox VE cluster filesystem.
Sep 20 09:54:23 host systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 20 09:54:23 host pmxcfs[18037]: [database] crit: missing directory inode (inode = 00000000018DA438)
Sep 20 09:54:23 host pmxcfs[18037]: [database] crit: missing directory inode (inode = 00000000018DA438)
Sep 20 09:54:23 host pmxcfs[18037]: [database] crit: DB load failed
Sep 20 09:54:23 host pmxcfs[18037]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 09:54:23 host pmxcfs[18037]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 09:54:23 host pmxcfs[18037]: [database] crit: DB load failed
Sep 20 09:54:23 host pmxcfs[18037]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Sep 20 09:54:23 host pmxcfs[18037]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 09:54:23 host systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 20 09:54:23 host systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 20 09:54:23 host systemd[1]: Failed to start The Proxmox VE cluster filesystem.

I dont know what to do next

 

[fiona]

Thanks for reply, I already followed other threads, I made some steps like these, now i get new error

sqlite3 config.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> select * from tree where name='qemu-server';
14|12|14|0|1621943568|4|qemu-server|
26059403|26059402|26059403|0|1663617922|4|qemu-server|
26059832|26059402|26059832|0|1663618560|4|qemu-server|
sqlite> ^C^C
┌─[/var/lib/pve-cluster]-root㉾host-(09:53:24)
└─[11]$ sqlite3 /var/lib/pve-cluster/config.db 'DELETE FROM tree WHERE inode = 26059832'

Seems like you removed the entry with the newer modification time. The other thread mentions removing the entry with the older modification time. Can you try again restoring the backup and removing the other one instead?

 

[rando]

Ok, I delete the older entry, now I got new error, and i am stuck again

Sep 20 13:52:39 host systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 20 13:52:39 host pmxcfs[1958]: fuse: mountpoint is not empty
Sep 20 13:52:39 host pmxcfs[1958]: fuse: if you are sure this is safe, use the 'nonempty' mount option
Sep 20 13:52:39 host pmxcfs[1958]: [main] crit: fuse_mount error: File exists
Sep 20 13:52:39 host pmxcfs[1958]: [main] crit: fuse_mount error: File exists
Sep 20 13:52:39 host pmxcfs[1958]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 13:52:39 host pmxcfs[1958]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 20 13:52:39 host systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 20 13:52:39 host systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 20 13:52:39 host systemd[1]: Failed to start The Proxmox VE cluster filesystem.

I have 3 VM on that proxmox and dont want to lose data, one vm is sql database for company finance database.

/etc/pve is empty,

 

[t.lamprecht]

/etc/pve is empty

Are you sure that there are no hidden files? ls -la /etc/pve

 

[rando]

Well done, yes there was still some files, I removed them and now I see my VM-s in web gui again. Big thanks for Proxmox community, very happy to see such a good support!!!