pveproxy - /dev/fuse not mounted to /etc/pve

P

pszafer

New Member
Mar 13, 2017
5
0
1
38
Hello,
Since upgrade to Proxmox 4.3 (I think) we've started to have problems with rebooting it.
Process pveproxy not mounting /etc/pve saying only in logs:
Code: 
pve01 pveproxy[2463]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1618.

In fact /etc/pve is empty and it is nout mounted.
Only way to make it running is to run:
Code: 
systemctl restart pveproxy

then with command mount I see
Code: 
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

without that /etc/pve doesn't exist.
It is single node server without cluster configured.
 
what does
Code: 
systemctl status pve-cluster
say?
 
It says that it is running. I will paste all journals so you can see more.
Just history proxmox was restarted 30.05 at 21.52 and at 30.05 at 21.54/55 I restarted pveproxy manually.

root@pve01:~# journalctl -u pve-cluster.service -n 30
-- Logs begin at Tue 2017-05-30 21:52:47 CEST, end at Wed 2017-05-31 16:25:48 CEST. --
May 30 21:52:57 pve01 systemd[1]: Starting The Proxmox VE cluster filesystem...
May 30 21:53:31 pve01 pmxcfs[2136]: [main] crit: Unable to get local IP address
May 30 21:53:31 pve01 pmxcfs[2136]: [main] crit: Unable to get local IP address
May 30 21:53:31 pve01 systemd[1]: pve-cluster.service: control process exited, cod
May 30 21:53:31 pve01 systemd[1]: Failed to start The Proxmox VE cluster filesyste
May 30 21:53:31 pve01 systemd[1]: Unit pve-cluster.service entered failed state.
May 30 21:55:33 pve01 systemd[1]: Starting The Proxmox VE cluster filesystem...
May 30 21:55:34 pve01 systemd[1]: Started The Proxmox VE cluster filesystem.
Click to expand...

root@pve01:~# journalctl -u pve-manager.service
-- Logs begin at Tue 2017-05-30 21:52:47 CEST, end at Wed 2017-05-31 16:35:48 CEST. --
May 30 21:53:34 pve01 systemd[1]: Starting PVE VM Manager...
May 30 21:53:35 pve01 pve-manager[2255]: <root@pam> starting task UPID:p
May 30 21:53:35 pve01 pve-manager[2255]: writing cluster log failed: ipc
May 30 21:55:34 pve01 pve-manager[2255]: <root@pam> end task UPID:pve01:
May 30 21:55:34 pve01 systemd[1]: Started PVE VM Manager.
Click to expand...

root@pve01:~# journalctl -u pve-firewall.service
-- Logs begin at Tue 2017-05-30 21:52:47 CEST, end at Wed 2017-05-31 16:37:48 CEST. --
May 30 21:53:31 pve01.poznan.tbhydro.net systemd[1]: Starting Proxmox VE firewall...
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2218]: starting server
May 30 21:53:32 pve01.poznan.tbhydro.net systemd[1]: Started Proxmox VE firewall.
Click to expand...

May 30 21:53:32 pve01 systemd[1]: Starting PVE API Proxy Server...
May 30 21:53:33 pve01 pveproxy[2244]: starting server
May 30 21:53:33 pve01 pveproxy[2244]: starting 3 worker(s)
May 30 21:53:33 pve01 pveproxy[2244]: worker 2245 started
May 30 21:53:33 pve01 pveproxy[2244]: worker 2246 started
May 30 21:53:33 pve01 pveproxy[2244]: worker 2247 started
May 30 21:53:33 pve01 pveproxy[2245]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:33 pve01 pveproxy[2246]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:33 pve01 pveproxy[2247]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:33 pve01 systemd[1]: Started PVE API Proxy Server.
May 30 21:53:38 pve01 pveproxy[2245]: worker exit
May 30 21:53:38 pve01 pveproxy[2246]: worker exit
May 30 21:53:38 pve01 pveproxy[2247]: worker exit
May 30 21:53:38 pve01 pveproxy[2244]: worker 2245 finished
May 30 21:53:38 pve01 pveproxy[2244]: starting 1 worker(s)
May 30 21:53:38 pve01 pveproxy[2244]: worker 2264 started
May 30 21:53:38 pve01 pveproxy[2244]: worker 2247 finished
May 30 21:53:38 pve01 pveproxy[2244]: worker 2246 finished
May 30 21:53:38 pve01 pveproxy[2244]: starting 2 worker(s)
May 30 21:53:38 pve01 pveproxy[2244]: worker 2265 started
May 30 21:53:38 pve01 pveproxy[2244]: worker 2266 started
May 30 21:53:38 pve01 pveproxy[2264]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:38 pve01 pveproxy[2265]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:38 pve01 pveproxy[2266]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:43 pve01 pveproxy[2264]: worker exit
May 30 21:53:43 pve01 pveproxy[2265]: worker exit
Click to expand...

and this is going as this as long as I don't restart pveproxy service because /etc/pve is not mounted.
May 30 21:55:31 pve01 systemd[1]: Stopping PVE API Proxy Server...
May 30 21:55:32 pve01 pveproxy[2244]: received signal TERM
May 30 21:55:32 pve01 pveproxy[2244]: server closing
May 30 21:55:32 pve01 pveproxy[2244]: worker 2462 finished
May 30 21:55:32 pve01 pveproxy[2244]: worker 2464 finished
May 30 21:55:32 pve01 pveproxy[2244]: worker 2463 finished
May 30 21:55:32 pve01 pveproxy[2244]: server stopped
May 30 21:55:34 pve01 systemd[1]: Starting PVE API Proxy Server...
May 30 21:55:35 pve01 pveproxy[2519]: starting server
May 30 21:55:35 pve01 pveproxy[2519]: starting 3 worker(s)
May 30 21:55:35 pve01 pveproxy[2519]: worker 2520 started
May 30 21:55:35 pve01 pveproxy[2519]: worker 2521 started
May 30 21:55:35 pve01 pveproxy[2519]: worker 2522 started
May 30 21:55:35 pve01 systemd[1]: Started PVE API Proxy Server.
May 30 22:11:53 pve01 pveproxy[2522]: worker exit
May 30 22:11:53 pve01 pveproxy[2519]: worker 2522 finished
May 30 22:11:53 pve01 pveproxy[2519]: starting 1 worker(s)
May 30 22:11:53 pve01 pveproxy[2519]: worker 17805 started
May 31 03:41:42 pve01 pveproxy[2520]: worker exit
May 31 03:41:42 pve01 pveproxy[2519]: worker 2520 finished
May 31 03:41:42 pve01 pveproxy[2519]: starting 1 worker(s)
May 31 03:41:42 pve01 pveproxy[2519]: worker 23050 started
Click to expand...
 
so, I think it was correct, but changing it resolved the problem.

It looked like this:
Code: 
root@pve01:/etc# cat hosts
127.0.0.1 localhost.localdomain localhost
10.8.0.16 pve01 pve01.domain.name

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

and change it to this:
Code: 
root@pve01:/etc# cat hosts
127.0.0.1 localhost.localdomain localhost
10.8.0.16 pve01.domain.name pve01

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
of course my domain is different, but I don't want to write it here, so I changed it.

anyway, it started working. very weird for me.
 
Hi there,

we have the same problem on one of our systems

the mount for /etc/pve is missing

and if we try to mount it we get a permisssion denied error


Code: 
root@proxmox:~# systemctl status pve-cluster.service
● pve-cluster.service - The Proxmox VE cluster filesystem
   Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2021-02-08 10:03:05 UTC; 26s ago
  Process: 1869 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)

Feb 08 10:03:05 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 10:03:05 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Feb 08 10:03:05 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 10:03:05 proxmox systemd[1]: pve-cluster.service: Start request repeated too quickly.
Feb 08 10:03:05 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 10:03:05 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.

root@proxmox:~# mount | grep fuse
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

root@proxmox:~# jornalctl -xe
-bash: jornalctl: command not found
root@proxmox:~# journalctl -xe
Feb 08 10:04:09 proxmox pveproxy[1651]: worker 1924 started
Feb 08 10:04:09 proxmox pveproxy[1651]: worker 1925 started
Feb 08 10:04:09 proxmox pveproxy[1924]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:09 proxmox pveproxy[1925]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:14 proxmox pveproxy[1923]: worker exit
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1923 finished
Feb 08 10:04:14 proxmox pveproxy[1651]: starting 1 worker(s)
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1926 started
Feb 08 10:04:14 proxmox pveproxy[1926]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:14 proxmox pveproxy[1924]: worker exit
Feb 08 10:04:14 proxmox pveproxy[1925]: worker exit
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1925 finished
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1924 finished
Feb 08 10:04:14 proxmox pveproxy[1651]: starting 2 worker(s)
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1927 started
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1928 started
Feb 08 10:04:14 proxmox pveproxy[1927]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:14 proxmox pveproxy[1928]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:19 proxmox pveproxy[1926]: worker exit
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1926 finished
Feb 08 10:04:19 proxmox pveproxy[1651]: starting 1 worker(s)
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1930 started
Feb 08 10:04:19 proxmox pveproxy[1930]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:19 proxmox pveproxy[1927]: worker exit
Feb 08 10:04:19 proxmox pveproxy[1928]: worker exit
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1927 finished
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1928 finished
Feb 08 10:04:19 proxmox pveproxy[1651]: starting 2 worker(s)
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1931 started
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1932 started
Feb 08 10:04:19 proxmox pveproxy[1931]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:19 proxmox pveproxy[1932]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:24 proxmox pveproxy[1930]: worker exit
Feb 08 10:04:24 proxmox pveproxy[1651]: worker 1930 finished
Feb 08 10:04:24 proxmox pveproxy[1651]: starting 1 worker(s)
Feb 08 10:04:24 proxmox pveproxy[1651]: worker 1933 started
Feb 08 10:04:24 proxmox pveproxy[1933]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:24 proxmox pveproxy[1931]: worker exit
Feb 08 10:04:24 proxmox pveproxy[1932]: worker exit
Feb 08 10:04:24 proxmox pveproxy[1651]: worker 1932 finished
Feb 08 10:04:24 proxmox pveproxy[1651]: worker 1931 finished
Feb 08 10:04:24 proxmox pveproxy[1651]: starting 2 worker(s)

Any idea ?
Any help is greatly appreciated

Quiet urgent since this is a production system
 

Attachments

Last edited:
here is the output


Code: 
root@proxmox:~# journalctl -b -u pve-cluster.service
-- Logs begin at Mon 2021-02-08 11:03:01 UTC, end at Mon 2021-02-08 12:18:02 UTC. --
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Feb 08 11:03:06 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 2.
Feb 08 11:03:06 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
lines 1-43...skipping...
-- Logs begin at Mon 2021-02-08 11:03:01 UTC, end at Mon 2021-02-08 12:18:02 UTC. --
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Feb 08 11:03:06 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 2.
Feb 08 11:03:06 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 3.
Feb 08 11:03:07 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 11:03:07 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:07 proxmox pmxcfs[1655]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:07 proxmox pmxcfs[1655]: [database] crit: DB load failed
Feb 08 11:03:07 proxmox pmxcfs[1655]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:07 proxmox pmxcfs[1655]: [database] crit: DB load failed
Feb 08 11:03:07 proxmox pmxcfs[1655]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:07 proxmox pmxcfs[1655]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:07 proxmox pmxcfs[1655]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:07 proxmox pmxcfs[1655]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:07 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 4.
lines 1-60
 

Attachments

Your problem is completely different from the threads original author.

Code: 
[database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
[database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
[database] crit: DB load failed
[main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'

The database seems to have some incosistent data in there, was there a power outage or any other failure (HW, SW) ... ?

Please post the output of the following two commands:

Bash: 
sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE name = "authorized_keys"'
sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE parent = (SELECT parent FROM tree WHERE inode = 12430028)'

Is this a cluster or stand alone node?
 
  • Like
Reactions: Bent
Stand alone

This system is on a vessel (Bulker) most possible it was a power outage

first query

Code: 
12425152|8|1612728778|8|authorized_keys
12430028|8|1612736567|8|authorized_keys

second

Code: 
16|8|1571839943|4|lock
27|8|1571839946|8|pve-root-ca.key
31|8|1571839946|8|pve-root-ca.srl
6072567|8|1601042357|8|shadow.cfg
6072650|8|1601042464|8|tfa.cfg
12418364|8|1612715928|8|authkey.key
12425152|8|1612728778|8|authorized_keys
12430028|8|1612736567|8|authorized_keys
12430031|8|1612736567|8|known_hosts
 

Attachments

Last edited:
MLOrion said:
This system is on a vessel (Bulker) most possible it was a power outage
Click to expand...
Still rather weird, I'd inspect the syslog/journal for any pointers, and the HW especially disks for possible failure (or indications that failure may be imminent)

For recovery:

1. backup the whole /var/lib/pve-cluster/ folder just to be sure, as root you can do:
tar czf pve-cluster-bak.tgz -C /var/lib/pve-cluster ./
Copy the resulting archive somewhere save.

2. Drop the duplicate with the older modification time, that would be the entry with inode `12425152`:
Bash: 
sqlite3 /var/lib/pve-cluster/config.db 'DELETE FROM tree WHERE inode = 12425152'

3. Start pve-cluster again:
Bash: 
systemctl reset-failed pve-cluster
systemctl start pve-cluster

4.
Check if the service is working, the directory is mounted and start the remaining pve services (or reboot)
 
  • Like
Reactions: Bent
We have it running on one more vessel
Working very well ....

The one we just recovered is called "Meteor"

Meteor.jpg
 
  • Like
Reactions: heky, Bent, Geco-iT and 7 others
You must log in or register to reply here.
Top Bottom