pveproxy - /dev/fuse not mounted to /etc/pve

P

pszafer

New Member
Mar 13, 2017
5
0
1
37
Hello,
Since upgrade to Proxmox 4.3 (I think) we've started to have problems with rebooting it.
Process pveproxy not mounting /etc/pve saying only in logs:
Code: 
pve01 pveproxy[2463]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1618.

In fact /etc/pve is empty and it is nout mounted.
Only way to make it running is to run:
Code: 
systemctl restart pveproxy

then with command mount I see
Code: 
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

without that /etc/pve doesn't exist.
It is single node server without cluster configured.
 
what does
Code: 
systemctl status pve-cluster
say?
 
It says that it is running. I will paste all journals so you can see more.
Just history proxmox was restarted 30.05 at 21.52 and at 30.05 at 21.54/55 I restarted pveproxy manually.

root@pve01:~# journalctl -u pve-cluster.service -n 30
-- Logs begin at Tue 2017-05-30 21:52:47 CEST, end at Wed 2017-05-31 16:25:48 CEST. --
May 30 21:52:57 pve01 systemd[1]: Starting The Proxmox VE cluster filesystem...
May 30 21:53:31 pve01 pmxcfs[2136]: [main] crit: Unable to get local IP address
May 30 21:53:31 pve01 pmxcfs[2136]: [main] crit: Unable to get local IP address
May 30 21:53:31 pve01 systemd[1]: pve-cluster.service: control process exited, cod
May 30 21:53:31 pve01 systemd[1]: Failed to start The Proxmox VE cluster filesyste
May 30 21:53:31 pve01 systemd[1]: Unit pve-cluster.service entered failed state.
May 30 21:55:33 pve01 systemd[1]: Starting The Proxmox VE cluster filesystem...
May 30 21:55:34 pve01 systemd[1]: Started The Proxmox VE cluster filesystem.
Click to expand...

root@pve01:~# journalctl -u pve-manager.service
-- Logs begin at Tue 2017-05-30 21:52:47 CEST, end at Wed 2017-05-31 16:35:48 CEST. --
May 30 21:53:34 pve01 systemd[1]: Starting PVE VM Manager...
May 30 21:53:35 pve01 pve-manager[2255]: <root@pam> starting task UPID:p
May 30 21:53:35 pve01 pve-manager[2255]: writing cluster log failed: ipc
May 30 21:55:34 pve01 pve-manager[2255]: <root@pam> end task UPID:pve01:
May 30 21:55:34 pve01 systemd[1]: Started PVE VM Manager.
Click to expand...

root@pve01:~# journalctl -u pve-firewall.service
-- Logs begin at Tue 2017-05-30 21:52:47 CEST, end at Wed 2017-05-31 16:37:48 CEST. --
May 30 21:53:31 pve01.poznan.tbhydro.net systemd[1]: Starting Proxmox VE firewall...
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2214]: ipcc_send_rec failed: Connecti
May 30 21:53:32 pve01.poznan.tbhydro.net pve-firewall[2218]: starting server
May 30 21:53:32 pve01.poznan.tbhydro.net systemd[1]: Started Proxmox VE firewall.
Click to expand...

May 30 21:53:32 pve01 systemd[1]: Starting PVE API Proxy Server...
May 30 21:53:33 pve01 pveproxy[2244]: starting server
May 30 21:53:33 pve01 pveproxy[2244]: starting 3 worker(s)
May 30 21:53:33 pve01 pveproxy[2244]: worker 2245 started
May 30 21:53:33 pve01 pveproxy[2244]: worker 2246 started
May 30 21:53:33 pve01 pveproxy[2244]: worker 2247 started
May 30 21:53:33 pve01 pveproxy[2245]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:33 pve01 pveproxy[2246]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:33 pve01 pveproxy[2247]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:33 pve01 systemd[1]: Started PVE API Proxy Server.
May 30 21:53:38 pve01 pveproxy[2245]: worker exit
May 30 21:53:38 pve01 pveproxy[2246]: worker exit
May 30 21:53:38 pve01 pveproxy[2247]: worker exit
May 30 21:53:38 pve01 pveproxy[2244]: worker 2245 finished
May 30 21:53:38 pve01 pveproxy[2244]: starting 1 worker(s)
May 30 21:53:38 pve01 pveproxy[2244]: worker 2264 started
May 30 21:53:38 pve01 pveproxy[2244]: worker 2247 finished
May 30 21:53:38 pve01 pveproxy[2244]: worker 2246 finished
May 30 21:53:38 pve01 pveproxy[2244]: starting 2 worker(s)
May 30 21:53:38 pve01 pveproxy[2244]: worker 2265 started
May 30 21:53:38 pve01 pveproxy[2244]: worker 2266 started
May 30 21:53:38 pve01 pveproxy[2264]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:38 pve01 pveproxy[2265]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:38 pve01 pveproxy[2266]: /etc/pve/local/pve-ssl.key: failed to load lo
May 30 21:53:43 pve01 pveproxy[2264]: worker exit
May 30 21:53:43 pve01 pveproxy[2265]: worker exit
Click to expand...

and this is going as this as long as I don't restart pveproxy service because /etc/pve is not mounted.
May 30 21:55:31 pve01 systemd[1]: Stopping PVE API Proxy Server...
May 30 21:55:32 pve01 pveproxy[2244]: received signal TERM
May 30 21:55:32 pve01 pveproxy[2244]: server closing
May 30 21:55:32 pve01 pveproxy[2244]: worker 2462 finished
May 30 21:55:32 pve01 pveproxy[2244]: worker 2464 finished
May 30 21:55:32 pve01 pveproxy[2244]: worker 2463 finished
May 30 21:55:32 pve01 pveproxy[2244]: server stopped
May 30 21:55:34 pve01 systemd[1]: Starting PVE API Proxy Server...
May 30 21:55:35 pve01 pveproxy[2519]: starting server
May 30 21:55:35 pve01 pveproxy[2519]: starting 3 worker(s)
May 30 21:55:35 pve01 pveproxy[2519]: worker 2520 started
May 30 21:55:35 pve01 pveproxy[2519]: worker 2521 started
May 30 21:55:35 pve01 pveproxy[2519]: worker 2522 started
May 30 21:55:35 pve01 systemd[1]: Started PVE API Proxy Server.
May 30 22:11:53 pve01 pveproxy[2522]: worker exit
May 30 22:11:53 pve01 pveproxy[2519]: worker 2522 finished
May 30 22:11:53 pve01 pveproxy[2519]: starting 1 worker(s)
May 30 22:11:53 pve01 pveproxy[2519]: worker 17805 started
May 31 03:41:42 pve01 pveproxy[2520]: worker exit
May 31 03:41:42 pve01 pveproxy[2519]: worker 2520 finished
May 31 03:41:42 pve01 pveproxy[2519]: starting 1 worker(s)
May 31 03:41:42 pve01 pveproxy[2519]: worker 23050 started
Click to expand...
 
so, I think it was correct, but changing it resolved the problem.

It looked like this:
Code: 
root@pve01:/etc# cat hosts
127.0.0.1 localhost.localdomain localhost
10.8.0.16 pve01 pve01.domain.name

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

and change it to this:
Code: 
root@pve01:/etc# cat hosts
127.0.0.1 localhost.localdomain localhost
10.8.0.16 pve01.domain.name pve01

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
of course my domain is different, but I don't want to write it here, so I changed it.

anyway, it started working. very weird for me.
 
Hi there,

we have the same problem on one of our systems

the mount for /etc/pve is missing

and if we try to mount it we get a permisssion denied error


Code: 
root@proxmox:~# systemctl status pve-cluster.service
● pve-cluster.service - The Proxmox VE cluster filesystem
   Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2021-02-08 10:03:05 UTC; 26s ago
  Process: 1869 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)

Feb 08 10:03:05 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 10:03:05 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Feb 08 10:03:05 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 10:03:05 proxmox systemd[1]: pve-cluster.service: Start request repeated too quickly.
Feb 08 10:03:05 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 10:03:05 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.

root@proxmox:~# mount | grep fuse
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

root@proxmox:~# jornalctl -xe
-bash: jornalctl: command not found
root@proxmox:~# journalctl -xe
Feb 08 10:04:09 proxmox pveproxy[1651]: worker 1924 started
Feb 08 10:04:09 proxmox pveproxy[1651]: worker 1925 started
Feb 08 10:04:09 proxmox pveproxy[1924]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:09 proxmox pveproxy[1925]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:14 proxmox pveproxy[1923]: worker exit
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1923 finished
Feb 08 10:04:14 proxmox pveproxy[1651]: starting 1 worker(s)
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1926 started
Feb 08 10:04:14 proxmox pveproxy[1926]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:14 proxmox pveproxy[1924]: worker exit
Feb 08 10:04:14 proxmox pveproxy[1925]: worker exit
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1925 finished
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1924 finished
Feb 08 10:04:14 proxmox pveproxy[1651]: starting 2 worker(s)
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1927 started
Feb 08 10:04:14 proxmox pveproxy[1651]: worker 1928 started
Feb 08 10:04:14 proxmox pveproxy[1927]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:14 proxmox pveproxy[1928]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:19 proxmox pveproxy[1926]: worker exit
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1926 finished
Feb 08 10:04:19 proxmox pveproxy[1651]: starting 1 worker(s)
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1930 started
Feb 08 10:04:19 proxmox pveproxy[1930]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:19 proxmox pveproxy[1927]: worker exit
Feb 08 10:04:19 proxmox pveproxy[1928]: worker exit
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1927 finished
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1928 finished
Feb 08 10:04:19 proxmox pveproxy[1651]: starting 2 worker(s)
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1931 started
Feb 08 10:04:19 proxmox pveproxy[1651]: worker 1932 started
Feb 08 10:04:19 proxmox pveproxy[1931]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:19 proxmox pveproxy[1932]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:24 proxmox pveproxy[1930]: worker exit
Feb 08 10:04:24 proxmox pveproxy[1651]: worker 1930 finished
Feb 08 10:04:24 proxmox pveproxy[1651]: starting 1 worker(s)
Feb 08 10:04:24 proxmox pveproxy[1651]: worker 1933 started
Feb 08 10:04:24 proxmox pveproxy[1933]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Feb 08 10:04:24 proxmox pveproxy[1931]: worker exit
Feb 08 10:04:24 proxmox pveproxy[1932]: worker exit
Feb 08 10:04:24 proxmox pveproxy[1651]: worker 1932 finished
Feb 08 10:04:24 proxmox pveproxy[1651]: worker 1931 finished
Feb 08 10:04:24 proxmox pveproxy[1651]: starting 2 worker(s)

Any idea ?
Any help is greatly appreciated

Quiet urgent since this is a production system
 

Attachments

  • config.zip
    12.7 KB · Views: 3
Last edited:
here is the output


Code: 
root@proxmox:~# journalctl -b -u pve-cluster.service
-- Logs begin at Mon 2021-02-08 11:03:01 UTC, end at Mon 2021-02-08 12:18:02 UTC. --
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Feb 08 11:03:06 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 2.
Feb 08 11:03:06 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
lines 1-43...skipping...
-- Logs begin at Mon 2021-02-08 11:03:01 UTC, end at Mon 2021-02-08 12:18:02 UTC. --
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1505]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1505]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Feb 08 11:03:06 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1621]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1621]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 2.
Feb 08 11:03:06 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 11:03:06 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox pmxcfs[1654]: [database] crit: DB load failed
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:06 proxmox pmxcfs[1654]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:06 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:06 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 3.
Feb 08 11:03:07 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 11:03:07 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 11:03:07 proxmox pmxcfs[1655]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:07 proxmox pmxcfs[1655]: [database] crit: DB load failed
Feb 08 11:03:07 proxmox pmxcfs[1655]: [database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
Feb 08 11:03:07 proxmox pmxcfs[1655]: [database] crit: DB load failed
Feb 08 11:03:07 proxmox pmxcfs[1655]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:07 proxmox pmxcfs[1655]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:07 proxmox pmxcfs[1655]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 08 11:03:07 proxmox pmxcfs[1655]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 11:03:07 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 11:03:07 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 4.
lines 1-60
 

Attachments

  • pve_logs.txt
    8.1 KB · Views: 1
Your problem is completely different from the threads original author.

Code: 
[database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
[database] crit: found entry with duplicate name (inode = 0000000000BDAACC, parent = 0000000000000008, name = 'authorized_keys')
[database] crit: DB load failed
[main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'

The database seems to have some incosistent data in there, was there a power outage or any other failure (HW, SW) ... ?

Please post the output of the following two commands:

Bash: 
sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE name = "authorized_keys"'
sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE parent = (SELECT parent FROM tree WHERE inode = 12430028)'

Is this a cluster or stand alone node?
 
  • Like
Reactions: Bent
Stand alone

This system is on a vessel (Bulker) most possible it was a power outage

first query

Code: 
12425152|8|1612728778|8|authorized_keys
12430028|8|1612736567|8|authorized_keys

second

Code: 
16|8|1571839943|4|lock
27|8|1571839946|8|pve-root-ca.key
31|8|1571839946|8|pve-root-ca.srl
6072567|8|1601042357|8|shadow.cfg
6072650|8|1601042464|8|tfa.cfg
12418364|8|1612715928|8|authkey.key
12425152|8|1612728778|8|authorized_keys
12430028|8|1612736567|8|authorized_keys
12430031|8|1612736567|8|known_hosts
 

Attachments

  • sqllite1.txt
    80 bytes · Views: 4
  • sqllite2.txt
    308 bytes · Views: 1
Last edited:
MLOrion said:
This system is on a vessel (Bulker) most possible it was a power outage
Click to expand...
Still rather weird, I'd inspect the syslog/journal for any pointers, and the HW especially disks for possible failure (or indications that failure may be imminent)

For recovery:

1. backup the whole /var/lib/pve-cluster/ folder just to be sure, as root you can do:
tar czf pve-cluster-bak.tgz -C /var/lib/pve-cluster ./
Copy the resulting archive somewhere save.

2. Drop the duplicate with the older modification time, that would be the entry with inode `12425152`:
Bash: 
sqlite3 /var/lib/pve-cluster/config.db 'DELETE FROM tree WHERE inode = 12425152'

3. Start pve-cluster again:
Bash: 
systemctl reset-failed pve-cluster
systemctl start pve-cluster

4.
Check if the service is working, the directory is mounted and start the remaining pve services (or reboot)
 
  • Like
Reactions: Bent
We have it running on one more vessel
Working very well ....

The one we just recovered is called "Meteor"

Meteor.jpg
 
  • Like
Reactions: heky, Bent, Geco-iT and 7 others
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back