Updating to 8.3.3 broke some things

N

nsan

New Member
Oct 28, 2024
4
0
1
So I went to update my PVE 8.3.2 last night and ran into this issue and I've now hit a wall and would love some assistance. pve-cluster, pve-firewall and pve-fstab stopped responding after rebooting the server after updating to PVE 8.3.3. After trying a few things (apt install --fix-broken, checking /etc/hosts and checking hostname -f. Everything came back well) and I haven't gotten anywhere. I have verified that I have no files in my /etc/pve directory. I have SSH access to proxmox as well as physical access to the server


Code: 
pveversion -v
proxmox-ve: 8.3.0 (running kernel: 6.8.12-7-pve)
pve-manager: 8.3.3 (running version: 8.3.3/f157a38b211595d6)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-15
proxmox-kernel-6.8: 6.8.12-7
proxmox-kernel-6.8.12-7-pve-signed: 6.8.12-7
proxmox-kernel-6.8.12-6-pve-signed: 6.8.12-6
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
proxmox-kernel-6.8.12-3-pve-signed: 6.8.12-3
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
pve-kernel-5.15.158-2-pve: 5.15.158-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.15+ds-0+deb12u1
corosync: 3.1.7-pve3
criu: 3.17.1-2+deb12u1
dnsmasq: 2.90-4~deb12u1
frr-pythontools: 8.5.2-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.3.4
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-3
pve-ha-manager: 4.0.6
pve-i18n: 3.3.3
pve-qemu-kvm: 9.0.2-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve1

Code: 
root@proxmox:~# systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-01-26 13:46:22 PST; 29min ago
   Main PID: 1241 (pveproxy)
      Tasks: 4 (limit: 38400)
     Memory: 153.9M
        CPU: 1min 13.633s
     CGroup: /system.slice/pveproxy.service
             ├─1241 pveproxy
             ├─3700 "pveproxy worker"
             ├─3701 "pveproxy worker"
             └─3702 "pveproxy worker"

Jan 26 14:16:06 proxmox pveproxy[1241]: worker 3698 finished
Jan 26 14:16:06 proxmox pveproxy[1241]: worker 3700 started
Jan 26 14:16:06 proxmox pveproxy[3700]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
Jan 26 14:16:06 proxmox pveproxy[3699]: worker exit
Jan 26 14:16:06 proxmox pveproxy[1241]: worker 3699 finished
Jan 26 14:16:06 proxmox pveproxy[1241]: starting 2 worker(s)
Jan 26 14:16:06 proxmox pveproxy[1241]: worker 3701 started
Jan 26 14:16:06 proxmox pveproxy[1241]: worker 3702 started
Jan 26 14:16:06 proxmox pveproxy[3701]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
Jan 26 14:16:06 proxmox pveproxy[3702]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.

Code: 
root@proxmox:~# pmxcfs -l
[main] notice: resolved node name 'proxmox' to 'xxx.xxx.xxx.xxx' for default node IP address
[database] crit: missing directory inode (inode = 0000000000453600)
[database] crit: DB load failed
[main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
[main] notice: exit proxmox configuration filesystem (-1)

Code: 
root@proxmox:~# tail -f /var/log/syslog
2025-01-26T13:58:57.349825-08:00 proxmox pveproxy[1868]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
2025-01-26T13:58:57.421391-08:00 proxmox pveproxy[1862]: worker exit
2025-01-26T13:58:57.422088-08:00 proxmox pveproxy[1863]: worker exit
2025-01-26T13:58:57.455626-08:00 proxmox pveproxy[1241]: worker 1862 finished
2025-01-26T13:58:57.455765-08:00 proxmox pveproxy[1241]: worker 1863 finished
2025-01-26T13:58:57.455827-08:00 proxmox pveproxy[1241]: starting 2 worker(s)
2025-01-26T13:58:57.458199-08:00 proxmox pveproxy[1241]: worker 1869 started
2025-01-26T13:58:57.460642-08:00 proxmox pveproxy[1241]: worker 1870 started
2025-01-26T13:58:57.492060-08:00 proxmox pveproxy[1869]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
2025-01-26T13:58:57.492613-08:00 proxmox pveproxy[1870]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2037.
2025-01-26T13:59:01.900418-08:00 proxmox cron[1222]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
2025-01-26T13:59:02.001220-08:00 proxmox proxmox-firewall[1223]: proxmox_firewall: error updating firewall rules: failed to read guest map from /etc/pve/.vmlist
 
Last edited:
are you running any custom scripts interfering with the database backing /etc/pve / pmxcfs? e.g., something keeping /var/lib/pve-cluster on a ramdisk, or similar things? could you post the system log before the reboot?
 
I am not running any custom script's that I am aware of that should be interfering with Proxmox
And unfortunately my logs only go back to 1/26/25 at 12:43pm, where as this event occurred around 1/25/25 at 4pm-ish
Would you recommend a rollback of the updates? Or would that introduce more problem's then it solve's?
 
Just a thought, based on an issue I had last week (not related to an upgrade)...

by any chance do you have 2FA enabled for the web ui. It was failing to present 2fa authentication. In the end had to remove 2fa requirement through ssh and was able to access the web ui.

Also, had to open it in an incognito window on the Brower as the browser cache was messing with it.

Not sure, if this helps to solve the issue, but thinking, it may help in looking at the issue from another angle.
 
msreedata said:
Just a thought, based on an issue I had last week (not related to an upgrade)...

by any chance do you have 2FA enabled for the web ui. It was failing to present 2fa authentication. In the end had to remove 2fa requirement through ssh and was able to access the web ui.

Also, had to open it in an incognito window on the Brower as the browser cache was messing with it.

Not sure, if this helps to solve the issue, but thinking, it may help in looking at the issue from another angle.
Click to expand...
Fortunately since this is a testing server I didn’t set up 2FA but that is a great thing to know in the future
I will try the incognito window when I get a chance tomorrow as this has taken a back seat for now since I’m hoping to have more info before I do anything else to the server
 
Unfortunately I can't help with resolving this issue but I was able to reinstall proxmox VE since all my vm backup's and VM disk's are on a seperate zpool than the install disk. So that was my solution. Hopefully someone else can get this solution
 
You must log in or register to reply here.
Top Bottom