Can't start VMs

[OPz13QzUUoZ87aQLvIsu]

Hi everyone,

Got a multiple issues with one of my Proxmox installation.
It started with the issue when question marks displayed on all machines and storage time to time. VMs were working at that time. I used to apply a fix as described here:
[SOLVED] Promox question marks on all machines and storage

But now I lost connection with those VMs and after rebooting couldn't start them. And all VMs becoming grey soon after rebooting or after restarting services:

service pve-cluster restart
service pveproxy restart
service pvedaemon restart
service pvestatd restart

I noticed the pvestatd service fails quite often and displays different errors when I check its status.

service pvestatd status
● pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-03-27 09:37:15 CET; 35min ago
    Process: 4902 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
   Main PID: 4909 (pvestatd)
      Tasks: 1 (limit: 154174)
     Memory: 83.8M
        CPU: 349ms
     CGroup: /system.slice/pvestatd.service
             └─4909 pvestatd

Mar 27 09:37:15 proxmox4 systemd[1]: Starting pvestatd.service - PVE Status Daemon...
Mar 27 09:37:15 proxmox4 pvestatd[4909]: starting server
Mar 27 09:37:15 proxmox4 systemd[1]: Started pvestatd.service - PVE Status Daemon.
Mar 27 09:37:35 proxmox4 pvestatd[4909]: qemu status update error: format error
                                         current: type check ('integer') failed - got SCALAR
Mar 27 09:37:35 proxmox4 pvestatd[4909]: lxc console cleanup error: Unknown open() mode '111356189901584' at /usr/lib/x86_64-linux-gnu/perl-base/IO/Fi>

Sometimes even pveversion -v command returns errors:

Not a subroutine reference at /usr/lib/x86_64-linux-gnu/perl/5.36/bits/syscall.ph line 831.
Compilation failed in require at /usr/lib/x86_64-linux-gnu/perl/5.36/sys/syscall.ph line 8.
Compilation failed in require at /usr/lib/x86_64-linux-gnu/perl/5.36/syscall.ph line 5.
Compilation failed in require at /usr/share/perl5/PVE/Syscall.pm line 10.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Syscall.pm line 82.
Compilation failed in require at /usr/share/perl5/PVE/Tools.pm line 31.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Tools.pm line 31.
Compilation failed in require at /usr/share/perl5/PVE/API2/APT.pm line 16.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/APT.pm line 16.
Compilation failed in require at /usr/bin/pveversion line 7.
BEGIN failed--compilation aborted at /usr/bin/pveversion line 7.

After restarting pve-cluster, pveproxy, pvedaemon, pvestatd I managed to rung the command:

pveversion -v
proxmox-ve: not correctly installed (running kernel: 6.5.13-3-pve)
pve-manager: not correctly installed (running version: 8.1.5/60e01c6ac2325b3f)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
proxmox-kernel-6.5: 6.5.13-3
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx8
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.2
libpve-apiclient-perl: 3.3.1
libpve-cluster-api-perl: 8.0.5
libpve-cluster-perl: 8.0.5
libpve-common-perl: 8.1.1
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.1.0
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.4
pve-cluster: 8.0.5
pve-container: 5.0.9
pve-docs: 8.1.4
pve-edk2-firmware: not correctly installed
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.1.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0

I also tried to reinstall some packages and got errors like this

apt-get install --reinstall pve-manager
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
E: Internal Error, No file name for pve-manager:amd64

Other things to share:
All VMs are turned off now, so the server's resource consumption is low.

journalctl -xe
Mar 27 10:22:38 proxmox4 kernel: Hardware name: ASUSTeK COMPUTER INC. System Product Name/W680/MB DC, BIOS 2004 12/16/2022
Mar 27 10:22:38 proxmox4 kernel: Call Trace:
Mar 27 10:22:38 proxmox4 kernel:  <TASK>
Mar 27 10:22:38 proxmox4 kernel:  dump_stack_lvl+0x48/0x70
Mar 27 10:22:38 proxmox4 kernel:  dump_stack+0x10/0x20
Mar 27 10:22:38 proxmox4 kernel:  print_bad_pte+0x1ca/0x2a0
Mar 27 10:22:38 proxmox4 kernel:  vm_normal_page+0xf6/0x110
Mar 27 10:22:38 proxmox4 kernel:  unmap_page_range+0x519/0x11c0
Mar 27 10:22:38 proxmox4 kernel:  unmap_single_vma+0x89/0xf0
Mar 27 10:22:38 proxmox4 kernel:  unmap_vmas+0xff/0x1a0
Mar 27 10:22:38 proxmox4 kernel:  unmap_region+0xe2/0x160
Mar 27 10:22:38 proxmox4 kernel:  do_vmi_align_munmap+0x37f/0x550
Mar 27 10:22:38 proxmox4 kernel:  do_vmi_munmap+0xdf/0x190
Mar 27 10:22:38 proxmox4 kernel:  __vm_munmap+0xae/0x180
Mar 27 10:22:38 proxmox4 kernel:  __x64_sys_munmap+0x27/0x40
Mar 27 10:22:38 proxmox4 kernel:  do_syscall_64+0x58/0x90
Mar 27 10:22:38 proxmox4 kernel:  ? syscall_exit_to_user_mode+0x37/0x60
Mar 27 10:22:38 proxmox4 kernel:  ? do_syscall_64+0x67/0x90
Mar 27 10:22:38 proxmox4 kernel:  ? do_syscall_64+0x67/0x90
Mar 27 10:22:38 proxmox4 kernel:  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Mar 27 10:22:38 proxmox4 kernel: RIP: 0033:0x5ccde1208f7
Mar 27 10:22:38 proxmox4 kernel: Code: 00 00 00 48 8b 15 09 05 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 b8 0b >
Mar 27 10:22:38 proxmox4 kernel: RSP: 002b:00007ffe8e074b28 EFLAGS: 00000246 ORIG_RAX: 000000000000000b
Mar 27 10:22:38 proxmox4 kernel: RAX: ffffffffffffffda RBX: 00006180f7b25530 RCX: 000005ccde1208f7
Mar 27 10:22:38 proxmox4 kernel: RDX: 0000000000000004 RSI: 000000000278a223 RDI: 000005ccdb000000
Mar 27 10:22:38 proxmox4 kernel: RBP: 00007ffe8e075070 R08: 0000000000000007 R09: 00006180f7b63af0
Mar 27 10:22:38 proxmox4 kernel: R10: e264a770a1c8b25c R11: 0000000000000246 R12: 0000000000000000
Mar 27 10:22:38 proxmox4 kernel: R13: 0000000000000000 R14: 00007ffe8e074e70 R15: 00007ffe8e074f50
Mar 27 10:22:38 proxmox4 kernel:  </TASK>
Mar 27 10:22:38 proxmox4 kernel: BUG: Bad rss-counter state mm:000000003bc9fa01 type:MM_FILEPAGES val:1
Mar 27 10:22:38 proxmox4 kernel: BUG: Bad rss-counter state mm:000000003bc9fa01 type:MM_ANONPAGES val:1
Mar 27 10:25:43 proxmox4 sshd[27381]: fatal: Timeout before authentication for 218.92.0.47 port 37760
Mar 27 10:27:26 proxmox4 sshd[29100]: Invalid user admin from 212.70.149.150 port 9879
Mar 27 10:27:26 proxmox4 sshd[29100]: pam_unix(sshd:auth): check pass; user unknown
Mar 27 10:27:26 proxmox4 sshd[29100]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=212.70.149.150
Mar 27 10:27:28 proxmox4 sshd[29100]: Failed password for invalid user admin from 212.70.149.150 port 9879 ssh2
Mar 27 10:27:28 proxmox4 sshd[29100]: Received disconnect from 212.70.149.150 port 9879:11: Bye [preauth]
Mar 27 10:27:28 proxmox4 sshd[29100]: Disconnected from invalid user admin 212.70.149.150 port 9879 [preauth]
Mar 27 10:28:34 proxmox4 pveproxy[14949]: proxy detected vanished client connection