Proxmox 4.4 failed - [database] crit: missing directory inode

[evgeniy2626]

Hi there, please help

One day I wanted to change hosname of PROXMOX node and to create cluster with two nodes.

I just changed hostname in /etc/hosts /etc/resolv.conf

After that in the webGUI I see two nodes with two names,

and I read that needs to cp -r /etc/pve/nodes/node1/* /etc/pve/nodes/node-new

ADN I DID IT.

Then I get error like as ;
pve-cluster[database] crit: found entry with duplicate name (inode = 0000000000000160, parent = 00000000000000F2, name = 'qemu-server')
[database] crit: DB load failed
[main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'

following the instruction http://blog.sjas.de/posts/proxmox-unable-to-open-database.html I was doing step by step as decribed. I deleted few records from db which associated with new node name (also I changed hostname on old hostname)

And this is does not helped me.
I still can't run pve-manages and others ....
And after that the directory /etc/pve/ is empty, how can I restore the node with all vm's?

# systemctl status pve-cluster.service
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled)
Active: failed (Result: exit-code) since Fri 2017-02-03 20:05:21 NL; 1min 31s ago
Process: 5336 ExecStart=/usr/bin/pmxcfs $DAEMON_OPTS (code=exited, status=255)

Feb 03 22:05:21 base pmxcfs[5336]: [database] crit: missing directory inode (inode = 00000000000C9C1F)
Feb 03 22:05:21 base pmxcfs[5336]: [database] crit: DB load failed
Feb 03 22:05:21 base pmxcfs[5336]: [database] crit: missing directory inode (inode = 00000000000C9C1F)
Feb 03 22:05:21 base pmxcfs[5336]: [database] crit: DB load failed
Feb 03 22:05:21 base pmxcfs[5336]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 03 22:05:21 base pmxcfs[5336]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 03 22:05:21 base systemd[1]: pve-cluster.service: control process exited, code=exited status=255
Feb 03 22:05:21 base systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 03 22:05:21 base systemd[1]: Unit pve-cluster.service entered failed state.



# pmxcfs -l
[database] crit: missing directory inode (inode = 00000000000C9C1F)
[database] crit: DB load failed
[main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
[main] notice: exit proxmox configuration filesystem (-1)



# journalctl -xn
-- Logs begin at Fri 2017-02-03 21:35:49 NL, end at Fri 2017-02-03 22:48:39 MSK. --
Feb 03 22:48:39 base pveproxy[2595]: worker 8863 finished
Feb 03 22:48:39 base pveproxy[2595]: starting 1 worker(s)
Feb 03 22:48:39 base pveproxy[2595]: worker 8867 started
Feb 03 22:48:39 base pveproxy[8867]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/HTTPServer.pm line 1646.
Feb 03 22:48:39 base pve-ha-lrm[2601]: ipcc_send_rec failed: Connection refused
Feb 03 22:48:39 base pve-ha-lrm[2601]: ipcc_send_rec failed: Connection refused
Feb 03 22:48:39 base pve-ha-lrm[2601]: ipcc_send_rec failed: Connection refused
Feb 03 22:48:39 base pve-ha-crm[2590]: ipcc_send_rec failed: Connection refused

# /etc/init.d/pve-manager status
● pve-manager.service - PVE VM Manager
Loaded: loaded (/lib/systemd/system/pve-manager.service; enabled)
Active: failed (Result: signal) since Fri 2017-02-03 23:34:42 MSK; 25s ago
Process: 11520 ExecStart=/usr/bin/pvesh --nooutput create /nodes/localhost/startall (code=killed, signal=KILL)
Main PID: 11520 (code=killed, signal=KILL)

Feb 03 23:21:58 base pve-manager[11520]: <root@pam> starting task UPID:base:00002D4:0009BA54:5894E666:startall::root@pam:
Feb 03 23:21:58 base pve-manager[11520]: writing cluster log failed: ipcc_send_rec failed: Connection refused
Feb 03 23:34:42 base systemd[1]: pve-manager.service: main process exited, code=killed, status=9/KILL
Feb 03 23:34:42 base systemd[1]: Failed to start PVE VM Manager.
Feb 03 23:34:42 base systemd[1]: Unit pve-manager.service entered failed state.
Feb 03 23:35:05 base systemd[1]: Stopped PVE VM Manager.

I noticed /dev/fuse is not mounted

~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 6.3G 25M 6.3G 1% /run
/dev/dm-0 950G 90.5G 81G 11% /
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup

Please help me!

 

[Mecanik]

Nice that nobody is answering this.... I have the similar problem.

 

[tom]

I do not know the linked third party howto, but hacking into the sqlite database is for sure not the way.

If you cannot fix it, better re-install or consult an Proxmox VE expert.

 

[evgeniy2626]

I do not know the linked third party howto, but hacking into the sqlite database is for sure not the way.

If you cannot fix it, better re-install or consult an Proxmox VE expert.

yes! its true

The only one way will help us)) as me too

 

[jmcgonegal]

I thought I'd post here because this was the only post that came up after googling what when wrong my sqllite database after changing my hostname. I was getting this error that my inode was missing.

Upon opening the SQLlite database the inode that was missing when converted from hex to decimal matched the parent of my `qemu-server` entry. Looking at the directory structure, this was supposed to be the new hostname. Because sqllite is supposed to be used to create a directory structure of /etc/pve/myhostname/qemu-server. With the myhostname inode missing, I needed to create it.


My missing inode was 00000000000188CB which hex->decimal converts to 100555.

I ran this command to find out it was my qemu-server.

select * from tree where parent = 100555

I ran this sql command on my database and uploaded it to the server (type = 4 is a folder).

insert into tree (inode, version, parent, type, writer, name, mtime) values (100555, 100555, 0, 4, 0, 'myhostname', 1619669401)

I started the cluster twice with

service pve-cluster start

I could then login to my cluster

To fix the errors with no stats I also had to run

systemctl start pvestatd

I hope this helps someone, killed a few hours of my time.

 

[Anatrop]

I thought I'd post here because this was the only post that came up after googling what when wrong my sqllite database after changing my hostname. I was getting this error that my inode was missing.

Upon opening the SQLlite database the inode that was missing when converted from hex to decimal matched the parent of my `qemu-server` entry. Looking at the directory structure, this was supposed to be the new hostname. Because sqllite is supposed to be used to create a directory structure of /etc/pve/myhostname/qemu-server. With the myhostname inode missing, I needed to create it.


My missing inode was 00000000000188CB which hex->decimal converts to 100555.

I ran this command to find out it was my qemu-server.

select * from tree where parent = 100555

I ran this sql command on my database and uploaded it to the server (type = 4 is a folder).

insert into tree (inode, version, parent, type, writer, name, mtime) values (100555, 100555, 0, 4, 0, 'myhostname', 1619669401)

I started the cluster twice with

service pve-cluster start

I could then login to my cluster

To fix the errors with no stats I also had to run

systemctl start pvestatd

I hope this helps someone, killed a few hours of my time.

Saved me, thank u so much!