Unable to open database after power outage

[pathfinder]

I had an unexpected power outage today and after booting my proxmox machine I wasn't able to load the web GUI.
After some digging around I managed to track the problem to a corrupted config.db file thanks to another thread on this forum.
When I ran journalctl -b -u pve-cluster I got the following:

-- Journal begins at Sun 2022-03-20 21:55:25 IST, ends at Tue 2023-01-17 21:17:55 IST. --
Jan 17 20:09:43 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jan 17 20:09:43 pve pmxcfs[1026]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:43 pve pmxcfs[1026]: [database] crit: DB load failed
Jan 17 20:09:43 pve pmxcfs[1026]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:43 pve pmxcfs[1026]: [database] crit: DB load failed
Jan 17 20:09:43 pve pmxcfs[1026]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:43 pve pmxcfs[1026]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:43 pve pmxcfs[1026]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:43 pve pmxcfs[1026]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:43 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jan 17 20:09:43 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jan 17 20:09:43 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jan 17 20:09:43 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Jan 17 20:09:43 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jan 17 20:09:43 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jan 17 20:09:43 pve pmxcfs[1142]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:43 pve pmxcfs[1142]: [database] crit: DB load failed
Jan 17 20:09:43 pve pmxcfs[1142]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:43 pve pmxcfs[1142]: [database] crit: DB load failed
Jan 17 20:09:43 pve pmxcfs[1142]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:43 pve pmxcfs[1142]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:43 pve pmxcfs[1142]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:43 pve pmxcfs[1142]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:43 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jan 17 20:09:43 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jan 17 20:09:43 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jan 17 20:09:44 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 2.
Jan 17 20:09:44 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
lines 1-29...skipping...
-- Journal begins at Sun 2022-03-20 21:55:25 IST, ends at Tue 2023-01-17 21:17:55 IST. --
Jan 17 20:09:43 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jan 17 20:09:43 pve pmxcfs[1026]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:43 pve pmxcfs[1026]: [database] crit: DB load failed
Jan 17 20:09:43 pve pmxcfs[1026]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:43 pve pmxcfs[1026]: [database] crit: DB load failed
Jan 17 20:09:43 pve pmxcfs[1026]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:43 pve pmxcfs[1026]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:43 pve pmxcfs[1026]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:43 pve pmxcfs[1026]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:43 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jan 17 20:09:43 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jan 17 20:09:43 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jan 17 20:09:43 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Jan 17 20:09:43 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jan 17 20:09:43 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jan 17 20:09:43 pve pmxcfs[1142]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:43 pve pmxcfs[1142]: [database] crit: DB load failed
Jan 17 20:09:43 pve pmxcfs[1142]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:43 pve pmxcfs[1142]: [database] crit: DB load failed
Jan 17 20:09:43 pve pmxcfs[1142]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:43 pve pmxcfs[1142]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:43 pve pmxcfs[1142]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:43 pve pmxcfs[1142]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:43 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jan 17 20:09:43 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jan 17 20:09:43 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jan 17 20:09:44 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 2.
Jan 17 20:09:44 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jan 17 20:09:44 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jan 17 20:09:44 pve pmxcfs[1146]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:44 pve pmxcfs[1146]: [database] crit: DB load failed
Jan 17 20:09:44 pve pmxcfs[1146]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:44 pve pmxcfs[1146]: [database] crit: DB load failed
Jan 17 20:09:44 pve pmxcfs[1146]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:44 pve pmxcfs[1146]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:44 pve pmxcfs[1146]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:44 pve pmxcfs[1146]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:44 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jan 17 20:09:44 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jan 17 20:09:44 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jan 17 20:09:44 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 3.
Jan 17 20:09:44 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jan 17 20:09:44 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jan 17 20:09:44 pve pmxcfs[1151]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:44 pve pmxcfs[1151]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:44 pve pmxcfs[1151]: [database] crit: DB load failed
Jan 17 20:09:44 pve pmxcfs[1151]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:44 pve pmxcfs[1151]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:44 pve pmxcfs[1151]: [database] crit: DB load failed
Jan 17 20:09:44 pve pmxcfs[1151]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:44 pve pmxcfs[1151]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:44 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jan 17 20:09:44 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jan 17 20:09:44 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jan 17 20:09:44 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 4.
Jan 17 20:09:44 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jan 17 20:09:44 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jan 17 20:09:44 pve pmxcfs[1155]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:44 pve pmxcfs[1155]: [database] crit: inode has no name (inode = 000000002C027209)
Jan 17 20:09:44 pve pmxcfs[1155]: [database] crit: DB load failed
Jan 17 20:09:44 pve pmxcfs[1155]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:44 pve pmxcfs[1155]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:44 pve pmxcfs[1155]: [database] crit: DB load failed
Jan 17 20:09:44 pve pmxcfs[1155]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jan 17 20:09:44 pve pmxcfs[1155]: [main] notice: exit proxmox configuration filesystem (-1)
Jan 17 20:09:44 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jan 17 20:09:44 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jan 17 20:09:44 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jan 17 20:09:45 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Jan 17 20:09:45 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jan 17 20:09:45 pve systemd[1]: pve-cluster.service: Start request repeated too quickly.
Jan 17 20:09:45 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jan 17 20:09:45 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.

So after that I tried running sqlite3 /var/lib/pve-cluster/config.db 'PRAGMA integrity_check' for a sanity check, but I got:

NULL value in tree.parent
NULL value in tree.version
NULL value in tree.writer
NULL value in tree.mtime
NULL value in tree.type
NULL value in tree.name
CHECK constraint failed in tree

Does this mean my config.db file is badly corrupted, is there anyway to restore ?
If anyone can help me I would greatly appreciate it.

 

[Chris]

Hi,
it seems like your filesystem got corrupted by the power outage, leading to a corrupt config.db which is the backing database for the proxmox cluster file system.

Have you tried to run a filesystem check on the unmounted disks from recovery? Do you have a backup of the host from which you might be able to restore the database?

 

[pathfinder]

Hi,
it seems like your filesystem got corrupted by the power outage, leading to a corrupt config.db which is the backing database for the proxmox cluster file system.

Have you tried to run a filesystem check on the unmounted disks from recovery? Do you have a backup of the host from which you might be able to restore the database?

Sorry, I am still a bit new to Proxmox / Linux. What commands should I run ?
When I ran lsblk it seems like my main drive is mounted fine (I think)

root@pve:~# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0  14.6T  0 disk
nvme0n1                      259:0    0 476.9G  0 disk
├─nvme0n1p1                  259:1    0  1007K  0 part
├─nvme0n1p2                  259:2    0   512M  0 part /boot/efi
└─nvme0n1p3                  259:3    0 476.4G  0 part
  ├─pve-swap                 253:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                 253:1    0    96G  0 lvm  /
  ├─pve-data_tmeta           253:2    0   3.6G  0 lvm
  │ └─pve-data-tpool         253:4    0 349.3G  0 lvm
  │   ├─pve-data             253:5    0 349.3G  1 lvm
  │   ├─pve-vm--100--disk--0 253:6    0    64G  0 lvm
  │   ├─pve-vm--101--disk--0 253:7    0    64G  0 lvm
  │   ├─pve-vm--102--disk--0 253:8    0     8G  0 lvm
  │   └─pve-vm--103--disk--0 253:9    0     8G  0 lvm
  └─pve-data_tdata           253:3    0 349.3G  0 lvm
    └─pve-data-tpool         253:4    0 349.3G  0 lvm
      ├─pve-data             253:5    0 349.3G  1 lvm
      ├─pve-vm--100--disk--0 253:6    0    64G  0 lvm
      ├─pve-vm--101--disk--0 253:7    0    64G  0 lvm
      ├─pve-vm--102--disk--0 253:8    0     8G  0 lvm
      └─pve-vm--103--disk--0 253:9    0     8G  0 lvm

*sda drive is just for mass storage (passthrough to a VM)
I am guessing you want me enter recovery mode and unmount the drive/s before running the FS check right ?

 

[Chris]

I am guessing you want me enter recovery mode and unmount the drive/s before running the FS check right ?

Yes, thats what I meant. Anyway, as a first step make a backup of the /var/lib/pve-cluster/config.db file and post the output of the following command:

sqlite3 /var/lib/pve-cluster/config.db 'SELECT parent,mtime,type,name FROM tree WHERE parent = 738357769 OR inode = 738357769;'

738357769 is the integer rappresentation of the inode hex value 000000002C027209

 

[pathfinder]

Yes, thats what I meant. Anyway, as a first step make a backup of the /var/lib/pve-cluster/config.db file and post the output of the following command:

sqlite3 /var/lib/pve-cluster/config.db 'SELECT parent,mtime,type,name FROM tree WHERE parent = 738357769 OR inode = 738357769;'

738357769 is the integer rappresentation of the inode hex value 000000002C027209

When I ran the query all I got was 3 vertical lines:

|||

 

[Chris]

When I ran the query all I got was 3 vertical lines:

|||

What about

sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE version = NULL OR name = NULL OR parent = NULL OR writer = NULL;'

 

[pathfinder]

What about

sqlite3 /var/lib/pve-cluster/config.db 'SELECT inode,parent,mtime,type,name FROM tree WHERE version = NULL OR name = NULL OR parent = NULL OR writer = NULL;'

Now it returns nothing

*just an FYI I did edit the grub file to run an fsck + repair on boot an rebooted. but it seems it didn't do anything so I reverted it back to just quiet and rebooted again - don't know if it did anything to the config.db file.

 

[Chris]

Now it returns nothing

*just an FYI I did edit the grub file to run an fsck + repair on boot an rebooted. but it seems it didn't do anything so I reverted it back to just quiet and rebooted again - don't know if it did anything to the config.db file.

Ok, in that case can you please check the sqlite3 /var/lib/pve-cluster/config.db 'PRAGMA integrity_check' again?

 

[pathfinder]

Ok, in that case can you please check the sqlite3 /var/lib/pve-cluster/config.db 'PRAGMA integrity_check' again?

I still get the same result:

NULL value in tree.parent
NULL value in tree.version
NULL value in tree.writer
NULL value in tree.mtime
NULL value in tree.type
NULL value in tree.name
CHECK constraint failed in tree

 

[t.lamprecht]

I'd recommend to first copy the current config.db to a backup location:

tar czf "/var/lib/pve-cluster/backup-corrupted-config-$(date -uIs)" -C /var/lib/pve-cluster/ config.db  config.db-shm  config.db-wal

Then check if the .recover command can help to get most, or even, all data out.

sqlite3 /var/lib/pve-cluster/config.db .recover >pmxcfs-recovery.sql
# import it into a new db
sqlite3 recovered.db <pmxcfs-recovery.sql

# then post the output of the following command so that we can evaluate what
# possible important files are missing:
sqlite3 recovered.db 'SELECT inode,mtime,name FROM tree'

 

[t.lamprecht]

Alternatively you might get away by just deleting the gone bad inode, but in any case make a backup!

The deletion command would be:

sqlite3 /var/lib/pve-cluster/config.db 'DELETE FROM tree WHERE parent = 738357769 OR inode = 738357769;'

 

[pathfinder]

I'd recommend to first copy the current config.db to a backup location:

tar czf "/var/lib/pve-cluster/backup-corrupted-config-$(date -uIs)" -C /var/lib/pve-cluster/ config.db  config.db-shm  config.db-wal

Then check if the .recover command can help to get most, or even, all data out.

sqlite3 /var/lib/pve-cluster/config.db .recover >pmxcfs-recovery.sql
# import it into a new db
sqlite3 recovered.db <pmxcfs-recovery.sql

# then post the output of the following command so that we can evaluate what
# possible important files are missing:
sqlite3 recovered.db 'SELECT inode,mtime,name FROM tree'

I already made a backup earlier and copied it to another machine so should be safe to play around.

Here is the output I received after running the commands:

root@pve:~# sqlite3 /var/lib/pve-cluster/config.db .recover >pmxcfs-recovery.sql
root@pve:~# sqlite3 recovered.db <pmxcfs-recovery.sql
Error: near line 41: NOT NULL constraint failed: tree.parent
root@pve:~# sqlite3 recovered.db 'SELECT inode,mtime,name FROM tree'
0|1673811875|__version__
2|1664816304|storage.cfg
4|1664816304|user.cfg
6|1664816304|datacenter.cfg
8|1664816354|virtual-guest
9|1664816355|priv
11|1664816355|nodes
12|1664816355|pve
13|1664816355|lxc
14|1664816355|qemu-server
15|1664816355|openvz
16|1664816355|priv
17|1664816355|lock
24|1664816355|pve-www.key
26|1664816355|pve-ssl.key
28|1664816355|pve-root-ca.key
30|1664816355|pve-root-ca.pem
32|1664816355|pve-root-ca.srl
35|1664816355|pve-ssl.pem
46|1664816355|vzdump.cron
49|1664816355|ha
50|1664816355|acme
51|1664816355|sdn
15640|1664880191|101.conf
2415541|1668628836|jobs.cfg
3686999|1670616415|firewall
3724594|1670673081|lrm_status.tmp.3930
5358448|1673178243|authorized_keys
5358451|1673178243|known_hosts
5740789|1673751747|100.conf
5740801|1673751757|102.conf
5740810|1673751764|103.conf
5780875|1673811871|lrm_status
5780879|1673811875|authkey.pub.old
5780882|1673811875|authkey.pub
5780885|1673811875|authkey.key

 

[t.lamprecht]

Can you post the inserts with NULLS (the data column NULL is ok):

grep '^INSERT.*NULL' pmxcfs-recovery.sql

Also, did you try my proposed alternative?

 

[pathfinder]

I ran the insert command but it still didn't work, so I tried the alternative and it did manage to get the cluster running again but now it has a problem getting data/status from the lxc/vm it's running (they do run fine now).
here is the complete output:

root@pve:~# grep '^INSERT.*NULL' pmxcfs-recovery.sql
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 0, 0, 5780888, 0, 1673811875, 8, '__version__', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 8, 0, 8, 0, 1664816354, 4, 'virtual-guest', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 9, 0, 9, 0, 1664816355, 4, 'priv', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 11, 0, 11, 0, 1664816355, 4, 'nodes', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 12, 11, 12, 0, 1664816355, 4, 'pve', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 13, 12, 13, 0, 1664816355, 4, 'lxc', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 14, 12, 14, 0, 1664816355, 4, 'qemu-server', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 15, 12, 15, 0, 1664816355, 4, 'openvz', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 16, 12, 16, 0, 1664816355, 4, 'priv', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 17, 9, 17, 0, 1664816355, 4, 'lock', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 49, 0, 49, 0, 1664816355, 4, 'ha', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 50, 9, 50, 0, 1664816355, 4, 'acme', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 51, 0, 51, 0, 1664816355, 4, 'sdn', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 3686999, 0, 3686999, 0, 1670616415, 4, 'firewall', NULL );
INSERT INTO "tree"("inode", "parent", "version", "writer", "mtime", "type", "name", "data") VALUES( 3724594, 12, 3724594, 0, 1670673081, 8, 'lrm_status.tmp.3930', NULL );
root@pve:~# sqlite3 recovered.db 'SELECT inode,mtime,name FROM tree'
0|1673811875|__version__
2|1664816304|storage.cfg
4|1664816304|user.cfg
6|1664816304|datacenter.cfg
8|1664816354|virtual-guest
9|1664816355|priv
11|1664816355|nodes
12|1664816355|pve
13|1664816355|lxc
14|1664816355|qemu-server
15|1664816355|openvz
16|1664816355|priv
17|1664816355|lock
24|1664816355|pve-www.key
26|1664816355|pve-ssl.key
28|1664816355|pve-root-ca.key
30|1664816355|pve-root-ca.pem
32|1664816355|pve-root-ca.srl
35|1664816355|pve-ssl.pem
46|1664816355|vzdump.cron
49|1664816355|ha
50|1664816355|acme
51|1664816355|sdn
15640|1664880191|101.conf
2415541|1668628836|jobs.cfg
3686999|1670616415|firewall
3724594|1670673081|lrm_status.tmp.3930
5358448|1673178243|authorized_keys
5358451|1673178243|known_hosts
5740789|1673751747|100.conf
5740801|1673751757|102.conf
5740810|1673751764|103.conf
5780875|1673811871|lrm_status
5780879|1673811875|authkey.pub.old
5780882|1673811875|authkey.pub
5780885|1673811875|authkey.key
root@pve:~# systemctl restart pveproxy
root@pve:~# systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-01-18 12:04:38 IST; 5s ago
    Process: 4464 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=111)
    Process: 4466 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 4474 (pveproxy)
      Tasks: 3 (limit: 57628)
     Memory: 129.2M
        CPU: 1.047s
     CGroup: /system.slice/pveproxy.service
             ├─4474 pveproxy
             ├─4479 pveproxy worker
             └─4480 pveproxy worker

Jan 18 12:04:43 pve pveproxy[4477]: worker exit
Jan 18 12:04:43 pve pveproxy[4474]: worker 4475 finished
Jan 18 12:04:43 pve pveproxy[4474]: starting 1 worker(s)
Jan 18 12:04:43 pve pveproxy[4474]: worker 4479 started
Jan 18 12:04:43 pve pveproxy[4474]: worker 4476 finished
Jan 18 12:04:43 pve pveproxy[4474]: starting 1 worker(s)
Jan 18 12:04:43 pve pveproxy[4474]: worker 4477 finished
Jan 18 12:04:43 pve pveproxy[4474]: worker 4480 started
Jan 18 12:04:43 pve pveproxy[4479]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1899.
Jan 18 12:04:43 pve pveproxy[4480]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1899.
root@pve:~# sqlite3 /var/lib/pve-cluster/config.db 'DELETE FROM tree WHERE parent = 738357769 OR inode = 738357769;'
root@pve:~# sqlite3 /var/lib/pve-cluster/config.db 'PRAGMA integrity_check'
ok
root@pve:~# systemctl restart pveproxy
root@pve:~# systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-01-18 12:05:24 IST; 2s ago
    Process: 4519 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 4521 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 4522 (pveproxy)
      Tasks: 4 (limit: 57628)
     Memory: 134.2M
        CPU: 957ms
     CGroup: /system.slice/pveproxy.service
             ├─4522 pveproxy
             ├─4523 pveproxy worker
             ├─4524 pveproxy worker
             └─4525 pveproxy worker

Jan 18 12:05:23 pve systemd[1]: Starting PVE API Proxy Server...
Jan 18 12:05:23 pve pveproxy[4522]: starting server
Jan 18 12:05:23 pve pveproxy[4522]: starting 3 worker(s)
Jan 18 12:05:23 pve pveproxy[4522]: worker 4523 started
Jan 18 12:05:23 pve pveproxy[4522]: worker 4524 started
Jan 18 12:05:23 pve pveproxy[4522]: worker 4525 started
Jan 18 12:05:24 pve systemd[1]: Started PVE API Proxy Server.
root@pve:~# systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-01-18 12:05:24 IST; 2min 19s ago
    Process: 4519 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 4521 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 4522 (pveproxy)
      Tasks: 4 (limit: 57628)
     Memory: 239.6M
        CPU: 1.967s
     CGroup: /system.slice/pveproxy.service
             ├─4522 pveproxy
             ├─4523 pveproxy worker
             ├─4524 pveproxy worker
             └─4525 pveproxy worker

Jan 18 12:05:23 pve systemd[1]: Starting PVE API Proxy Server...
Jan 18 12:05:23 pve pveproxy[4522]: starting server
Jan 18 12:05:23 pve pveproxy[4522]: starting 3 worker(s)
Jan 18 12:05:23 pve pveproxy[4522]: worker 4523 started
Jan 18 12:05:23 pve pveproxy[4522]: worker 4524 started
Jan 18 12:05:23 pve pveproxy[4522]: worker 4525 started
Jan 18 12:05:24 pve systemd[1]: Started PVE API Proxy Server.

And what I see in the dashboard:

 

[t.lamprecht]

has a problem getting data/status from the lxc/vm it's running (they do run fine now).

That's probably because some services that depend on pve-cluster failed, for the status that would be pvestatd:

systemctl status pvestatd
systemctl restart pvestatd

# maybe there are some others:
systemctl list-units --failed

 

[pathfinder]

That's probably because some services that depend on pve-cluster failed, for the status that would be pvestatd:

systemctl status pvestatd
systemctl restart pvestatd

# maybe there are some others:
systemctl list-units --failed

Awesome, all I needed was to restart pvestatd, now everything seems to work fine.

Thanks a lot man, I really appreciate the help.
Both you and Chris are LIFE SAVERS!!!!

 

[Leone]

Oie estou com um problema parecido alguem pode ajudar.

Segue ou erro



root@Lynx:/var/lib/pve-cluster# systemctl status pvestatd
● pvestatd.service - PVE Status Daemon
Carregado: carregado (/lib/systemd/system/pvestatd.service; ativado; predefinição do fornecedor: ativado)
Ativo: falhou (Resultado: código de saída) desde qua 2023-01-18 14:02:55 -03; 38 minutos atrás
Processo: 1193 ExecStart=/usr/bin/pvestatd start (código=sair, status=111)
Processador: 1.409s

18 de janeiro 14:02:54 Lynx pvestatd[1193]: ipcc_send_rec[2] falhou: Conexão recusada
18 de janeiro 14:02:54 Lynx pvestatd[1193]: ipcc_send_rec[1] falhou: Conexão recusada
18 de janeiro 14:02:55 Lynx pvestatd[1193]: ipcc_send_rec[3] falhou: Conexão recusada
18 de janeiro 14:02:55 Lynx pvestatd[1193]: Não foi possível carregar a lista de controle de acesso: Conexão recusada
18 de janeiro 14:02:54 Lynx pvestatd[1193]: ipcc_send_rec[2] falhou: Conexão recusada
18 de janeiro 14:02:54 Lynx pvestatd[1193]: ipcc_send_rec[3] falhou: Conexão recusada
18 de janeiro 14:02:55 Lynx systemd[1]: pvestatd.service: Processo de controle encerrado, código=sair, status=111/n/a
18 de janeiro 14:02:55 Lynx systemd[1]: pvestatd.service: Falha com o resultado 'exit-code'.
18 de janeiro 14:02:55 Lynx systemd[1]: Falha ao iniciar o PVE Status Daemon.
18 de janeiro 14:02:55 Lynx systemd[1]: pvestatd.service: 1.409s de tempo de CPU consumido.
root@Lynx:/var/lib/pve-cluster# systemctl restart pvestatd
A tarefa para pvestatd.service falhou porque o processo de controle foi encerrado com código de erro.
Consulte "systemctl status pvestatd.service" e "journalctl -xe" para obter detalhes.
root@Lynx:/var/lib/pve-cluster# systemctl list-units --failed
SUB DESCRIÇÃO DE CARGA DA UNIDADE ATIVA
● ceph-mgr@pve.service carregado falhou falhou Ceph cluster gerenciador daemon
● pve-cluster.service carregado falhou falhou O sistema de arquivos de cluster Proxmox VE
● pve-firewall.service carregado com falha com falha no firewall Proxmox VE
● pve-guests.service carregado falhou convidados PVE falhados
● pve-ha-crm.service carregado falhou falhou PVE Cluster HA Resource Manager Daemon
● pve-ha-lrm.service carregado falhou falhou PVE Local HA Resource Manager Daemon
● pvescheduler.service carregado falhou agendador Proxmox VE com falha
● pvestatd.service carregado falhou falhou PVE Status Daemon

LOAD = Reflete se a definição da unidade foi carregada corretamente.
ACTIVE = O estado de ativação da unidade de alto nível, ou seja, generalização de SUB.
SUB = O estado de ativação da unidade de baixo nível, os valores dependem do tipo de unidade.
8 unidades carregadas listadas.

 

[Chris]

Hi,
@Leone this is an English speaking forum, so in order for people to help you please post in Engilsh.

Also, since your problem is not directly related to this issue, please open a new thread for that. There you can post the output of journalctl -r -b.

Thanks

 

[Leone]

Olá Cris Bom Dia,

root@Lynx:/# journalctl -r -b
-- O diário começa na sexta-feira 18/11/2022 13:36:19 -03, termina na quinta 19/01/2023 07:12:19 -03. --
19 de janeiro 07:12:19 Lynx pveproxy[38769]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:12:19 Lynx pveproxy[1771]: trabalhador 38769 iniciado
19 de janeiro 07:12:19 Lynx pveproxy[1771]: trabalhador 38766 concluído
19 de janeiro 07:12:19 Lynx pveproxy[1771]: iniciando 1 trabalhador(es)
19 de janeiro 07:12:19 Lynx pveproxy[1771]: trabalhador 38767 concluído
19 de janeiro 07:12:19 Lynx pveproxy[38767]: saída do trabalhador
19 de janeiro 07:12:19 Lynx pveproxy[38766]: saída do trabalhador
19 de janeiro 07:12:19 Lynx pveproxy[38768]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:12:19 Lynx pveproxy[1771]: trabalhador 38768 iniciado
19 de janeiro 07:12:19 Lynx pveproxy[1771]: iniciando 1 trabalhador(es)
19 de janeiro 07:12:19 Lynx pveproxy[1771]: trabalhador 38765 concluído
19 de janeiro 07:12:19 Lynx pveproxy[38765]: saída do trabalhador
19 de janeiro 07:12:14 Lynx pveproxy[38767]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:12:14 Lynx pveproxy[38766]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:12:14 Lynx pveproxy[1771]: trabalhador 38767 iniciado
19 de janeiro 07:12:14 Lynx pveproxy[1771]: trabalhador 38766 iniciado
19 de janeiro 07:12:14 Lynx pveproxy[1771]: iniciando 2 trabalhador(es)
19 de janeiro 07:12:14 Lynx pveproxy[1771]: trabalhador 38764 concluído
19 de janeiro 07:12:14 Lynx pveproxy[38764]: saída do trabalhador
19 de janeiro 07:12:14 Lynx pveproxy[38765]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:12:14 Lynx pveproxy[1771]: trabalhador 38765 iniciado
19 de janeiro 07:12:14 Lynx pveproxy[1771]: trabalhador 38762 concluído
19 de janeiro 07:12:14 Lynx pveproxy[1771]: iniciando 1 trabalhador(es)
19 de janeiro 07:12:14 Lynx pveproxy[1771]: trabalhador 38763 concluído
19 de janeiro 07:12:14 Lynx pveproxy[38763]: saída do trabalhador
19 de janeiro 07:12:14 Lynx pveproxy[38762]: saída do trabalhador
19 de janeiro 07:12:09 Lynx pveproxy[38764]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:12:09 Lynx pveproxy[1771]: trabalhador 38764 iniciado
19 de janeiro 07:12:09 Lynx pveproxy[1771]: iniciando 1 trabalhador(es)
19 de janeiro 07:12:09 Lynx pveproxy[1771]: trabalhador 38761 concluído
19 de janeiro 07:12:09 Lynx pveproxy[38761]: saída do trabalhador
19 de janeiro 07:12:09 Lynx pveproxy[38763]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:12:09 Lynx pveproxy[38762]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:12:09 Lynx pveproxy[1771]: trabalhador 38763 iniciado
19 de janeiro 07:12:09 Lynx pveproxy[1771]: trabalhador 38762 iniciado
19 de janeiro 07:12:09 Lynx pveproxy[1771]: iniciando 2 trabalhador(es)
19 de janeiro 07:12:09 Lynx pveproxy[1771]: trabalhador 38760 concluído
19 de janeiro 07:12:09 Lynx pveproxy[38760]: saída do trabalhador
19 de janeiro 07:12:04 Lynx pveproxy[38761]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:12:04 Lynx pveproxy[1771]: trabalhador 38761 iniciado
19 de janeiro 07:12:04 Lynx pveproxy[1771]: iniciando 1 trabalhador(es)
19 de janeiro 07:12:04 Lynx pveproxy[1771]: trabalhador 38758 concluído
19 de janeiro 07:12:04 Lynx pveproxy[1771]: trabalhador 38759 concluído
19 de janeiro 07:12:04 Lynx pveproxy[38759]: saída do trabalhador
19 de janeiro 07:12:04 Lynx pveproxy[38758]: saída do trabalhador
19 de janeiro 07:12:04 Lynx pveproxy[38760]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:12:04 Lynx pveproxy[1771]: trabalhador 38760 iniciado
19 de janeiro 07:12:04 Lynx pveproxy[1771]: iniciando 1 trabalhador(es)
19 de janeiro 07:12:04 Lynx pveproxy[1771]: trabalhador 38757 concluído
19 de janeiro 07:12:04 Lynx pveproxy[38757]: saída do trabalhador
Jan 19 07:12:01 Lynx cron[1185]: (*system*vzdump) NÃO É POSSÍVEL ABRIR O SYMLINK (/etc/cron.d/vzdump)
19 de janeiro 07:11:59 Lynx pveproxy[38759]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.
19 de janeiro 07:11:59 Lynx pveproxy[38758]: /etc/pve/local/pve-ssl.key: falha ao carregar chave privada local (key_file ou chave) em /usr/share/perl5/PVE/APIServer/AnyEvent .pm linha 1899.