Lots of things seems broken after full-upgrade...

[marvindosh]

I need help, and I'm French, difficult to express my ideas or infos here.

I updated and did a full-upgrade last hours.
I shown that finished bad... but late
Now I couldn't finish pve-manager installation, marked as not completed
Nothing in pve folder (totally empty)
Some errors here and there ; like duplicate entry in config.db ... nothing duplicated here after verifications ...

some commands here

/# apt full-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up pve-manager (7.2-14) ...
Job for pvestatd.service failed because the control process exited with error code.
See "systemctl status pvestatd.service" and "journalctl -xe" for details.
dpkg: error processing package pve-manager (--configure):
 installed pve-manager package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 pve-manager
E: Sub-process /usr/bin/dpkg returned an error code (1)

Nov 20 22:54:38 ns316917 pveproxy[35563]: worker exit
Nov 20 22:54:38 ns316917 pveproxy[1464]: starting 1 worker(s)
Nov 20 22:54:38 ns316917 pveproxy[1464]: worker 35563 finished
Nov 20 22:54:38 ns316917 pveproxy[1464]: worker 35570 started
Nov 20 22:54:38 ns316917 pveproxy[35564]: worker exit
Nov 20 22:54:38 ns316917 pveproxy[35570]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1899.
Nov 20 22:54:38 ns316917 pveproxy[1464]: worker 35564 finished
Nov 20 22:54:38 ns316917 pveproxy[1464]: starting 2 worker(s)
Nov 20 22:54:38 ns316917 pveproxy[1464]: worker 35571 started
Nov 20 22:54:38 ns316917 pveproxy[1464]: worker 35572 started
Nov 20 22:54:38 ns316917 pveproxy[35571]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1899.
Nov 20 22:54:38 ns316917 pveproxy[35572]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1899.
Nov 20 22:54:43 ns316917 pveproxy[35570]: worker exit
Nov 20 22:54:43 ns316917 pveproxy[1464]: worker 35570 finished
Nov 20 22:54:43 ns316917 pveproxy[1464]: starting 1 worker(s)
Nov 20 22:54:43 ns316917 pveproxy[1464]: worker 35573 started
Nov 20 22:54:43 ns316917 pveproxy[35571]: worker exit
Nov 20 22:54:43 ns316917 pveproxy[35572]: worker exit
Nov 20 22:54:43 ns316917 pveproxy[35573]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1899.
Nov 20 22:54:43 ns316917 pveproxy[1464]: worker 35571 finished
Nov 20 22:54:43 ns316917 pveproxy[1464]: worker 35572 finished
Nov 20 22:54:43 ns316917 pveproxy[1464]: starting 1 worker(s)
Nov 20 22:54:43 ns316917 pveproxy[1464]: worker 35574 started
Nov 20 22:54:43 ns316917 pveproxy[35574]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1899.
Nov 20 22:54:48 ns316917 pveproxy[35573]: worker exit
Nov 20 22:54:48 ns316917 pveproxy[1464]: worker 35573 finished
Nov 20 22:54:48 ns316917 pveproxy[1464]: starting 2 worker(s)
Nov 20 22:54:48 ns316917 pveproxy[1464]: worker 35575 started
Nov 20 22:54:48 ns316917 pveproxy[1464]: worker 35576 started
Nov 20 22:54:48 ns316917 pveproxy[35574]: worker exit
Nov 20 22:54:48 ns316917 pveproxy[35575]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1899.
Nov 20 22:54:48 ns316917 pveproxy[35576]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1899.
Nov 20 22:54:48 ns316917 pveproxy[1464]: worker 35574 finished
Nov 20 22:54:48 ns316917 pveproxy[1464]: starting 1 worker(s)
Nov 20 22:54:48 ns316917 pveproxy[1464]: worker 35577 started
Nov 20 22:54:48 ns316917 pveproxy[35577]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1899.

pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2022-11-20 22:54:08 CET; 2min 44s ago
    Process: 35534 ExecStart=/usr/bin/pvestatd start (code=exited, status=111)
        CPU: 429ms

Nov 20 22:54:08 ns316917 pvestatd[35534]: ipcc_send_rec[1] failed: Connection refused
Nov 20 22:54:08 ns316917 pvestatd[35534]: ipcc_send_rec[2] failed: Connection refused
Nov 20 22:54:08 ns316917 pvestatd[35534]: ipcc_send_rec[1] failed: Connection refused
Nov 20 22:54:08 ns316917 pvestatd[35534]: ipcc_send_rec[3] failed: Connection refused
Nov 20 22:54:08 ns316917 pvestatd[35534]: ipcc_send_rec[2] failed: Connection refused
Nov 20 22:54:08 ns316917 pvestatd[35534]: ipcc_send_rec[3] failed: Connection refused
Nov 20 22:54:08 ns316917 pvestatd[35534]: Unable to load access control list: Connection refused
Nov 20 22:54:08 ns316917 systemd[1]: pvestatd.service: Control process exited, code=exited, status=111/n/a
Nov 20 22:54:08 ns316917 systemd[1]: pvestatd.service: Failed with result 'exit-code'.
Nov 20 22:54:08 ns316917 systemd[1]: Failed to start PVE Status Daemon.

.
.
.

 pve-guests.service - PVE guests
     Loaded: loaded (/lib/systemd/system/pve-guests.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2022-11-20 22:27:02 CET; 54min ago
   Main PID: 1473 (code=exited, status=111)
        CPU: 771ms

Nov 20 22:27:02 ns316917 systemd[1]: Starting PVE guests...
Nov 20 22:27:02 ns316917 pvesh[1473]: ipcc_send_rec[1] failed: Connection refused
Nov 20 22:27:02 ns316917 pvesh[1473]: ipcc_send_rec[2] failed: Connection refused
Nov 20 22:27:02 ns316917 pvesh[1473]: ipcc_send_rec[3] failed: Connection refused
Nov 20 22:27:02 ns316917 pvesh[1473]: Unable to load access control list: Connection refused
Nov 20 22:27:02 ns316917 systemd[1]: pve-guests.service: Main process exited, code=exited, status=111/n/a
Nov 20 22:27:02 ns316917 systemd[1]: pve-guests.service: Failed with result 'exit-code'.
Nov 20 22:27:02 ns316917 systemd[1]: Failed to start PVE guests.

My hosts and hostname via PM

 

[marvindosh]

/# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.74-1-pve)
pve-manager: not correctly installed (running version: 7.2-14/65898fbc)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.39-3-pve: 5.15.39-3
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-7
libpve-guest-common-perl: 4.2-2
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.2
pve-cluster: 7.2-3
pve-container: 4.3-5
pve-docs: 7.2-3
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.1.0-3
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-10
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

 

[marvindosh]

/# pvecm updatecerts
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

 

[marvindosh]

:/# systemctl list-units --state=failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● pve-cluster.service loaded failed failed The Proxmox VE cluster filesystem
● pve-firewall.service loaded failed failed Proxmox VE firewall
● pve-guests.service loaded failed failed PVE guests
● pve-ha-crm.service loaded failed failed PVE Cluster HA Resource Manager Daemon
● pve-ha-lrm.service loaded failed failed PVE Local HA Resource Manager Daemon
● pvescheduler.service loaded failed failed Proxmox VE scheduler
● pvestatd.service loaded failed failed PVE Status Daemon

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
7 loaded units listed.

 

[fabian]

what does journalctl -u pve-cluster -u corosync say?

 

[marvindosh]

Hi, resolved that by renaming Config.db to try some tricks on it, GUI became accessible again, repairing of some dependances and apps, like pve-manager, switch my vm to other folder, simulate the creation of a new one for config file recreation, deleting raw disk of the new and replace it with the real VM disk. Repairing lots of broken things in background... etc. So confused on what was impacted, And why all these broke !
But I'm now looking if that couldn't came from an attack. Submitted some files to my server renter for inspection.

Thanks, because the system wasn't destroyed ^^ and recovering was not easy, but possible.

Sorry for the spamming too ^^