So I went to update my PVE 8.3.2 last night and ran into this issue and I've now hit a wall and would love some assistance. pve-cluster, pve-firewall and pve-fstab stopped responding after rebooting the server after updating to PVE 8.3.3. After trying a few things (apt install --fix-broken, checking /etc/hosts and checking hostname -f. Everything came back well) and I haven't gotten anywhere. I have verified that I have no files in my /etc/pve directory. I have SSH access to proxmox as well as physical access to the server
Code:
pveversion -v
proxmox-ve: 8.3.0 (running kernel: 6.8.12-7-pve)
pve-manager: 8.3.3 (running version: 8.3.3/f157a38b211595d6)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-15
proxmox-kernel-6.8: 6.8.12-7
proxmox-kernel-6.8.12-7-pve-signed: 6.8.12-7
proxmox-kernel-6.8.12-6-pve-signed: 6.8.12-6
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
proxmox-kernel-6.8.12-3-pve-signed: 6.8.12-3
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
pve-kernel-5.15.158-2-pve: 5.15.158-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.15+ds-0+deb12u1
corosync: 3.1.7-pve3
criu: 3.17.1-2+deb12u1
dnsmasq: 2.90-4~deb12u1
frr-pythontools: 8.5.2-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.3.4
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-3
pve-ha-manager: 4.0.6
pve-i18n: 3.3.3
pve-qemu-kvm: 9.0.2-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve1
Code:
root@proxmox:~# systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
Active: active (running) since Sun 2025-01-26 13:46:22 PST; 29min ago
Main PID: 1241 (pveproxy)
Tasks: 4 (limit: 38400)
Memory: 153.9M
CPU: 1min 13.633s
CGroup: /system.slice/pveproxy.service
├─1241 pveproxy
├─3700 "pveproxy worker"
├─3701 "pveproxy worker"
└─3702 "pveproxy worker"
Jan 26 14:16:06 proxmox pveproxy[1241]: worker 3698 finished
Jan 26 14:16:06 proxmox pveproxy[1241]: worker 3700 started
Jan 26 14:16:06 proxmox pveproxy[3700]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
Jan 26 14:16:06 proxmox pveproxy[3699]: worker exit
Jan 26 14:16:06 proxmox pveproxy[1241]: worker 3699 finished
Jan 26 14:16:06 proxmox pveproxy[1241]: starting 2 worker(s)
Jan 26 14:16:06 proxmox pveproxy[1241]: worker 3701 started
Jan 26 14:16:06 proxmox pveproxy[1241]: worker 3702 started
Jan 26 14:16:06 proxmox pveproxy[3701]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
Jan 26 14:16:06 proxmox pveproxy[3702]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
Code:
root@proxmox:~# pmxcfs -l
[main] notice: resolved node name 'proxmox' to 'xxx.xxx.xxx.xxx' for default node IP address
[database] crit: missing directory inode (inode = 0000000000453600)
[database] crit: DB load failed
[main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
[main] notice: exit proxmox configuration filesystem (-1)
Code:
root@proxmox:~# tail -f /var/log/syslog
2025-01-26T13:58:57.349825-08:00 proxmox pveproxy[1868]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
2025-01-26T13:58:57.421391-08:00 proxmox pveproxy[1862]: worker exit
2025-01-26T13:58:57.422088-08:00 proxmox pveproxy[1863]: worker exit
2025-01-26T13:58:57.455626-08:00 proxmox pveproxy[1241]: worker 1862 finished
2025-01-26T13:58:57.455765-08:00 proxmox pveproxy[1241]: worker 1863 finished
2025-01-26T13:58:57.455827-08:00 proxmox pveproxy[1241]: starting 2 worker(s)
2025-01-26T13:58:57.458199-08:00 proxmox pveproxy[1241]: worker 1869 started
2025-01-26T13:58:57.460642-08:00 proxmox pveproxy[1241]: worker 1870 started
2025-01-26T13:58:57.492060-08:00 proxmox pveproxy[1869]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
2025-01-26T13:58:57.492613-08:00 proxmox pveproxy[1870]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
2025-01-26T13:59:01.900418-08:00 proxmox cron[1222]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
2025-01-26T13:59:02.001220-08:00 proxmox proxmox-firewall[1223]: proxmox_firewall: error updating firewall rules: failed to read guest map from /etc/pve/.vmlist
Last edited:
