Proxmox not working after updates

[mvank]

Hi, I am using proxmox 7.3 for some time as home user, but never ran into serious issues.
today i installed the latest updated, and after reboot proxmox won't start anymore...
Hope someoen is able to point me into the right direction before i do a full reinstall and restore everything from yesterday
My knowledge about linux os is minimal, only thing what i found was:

●pve-ha-crm.service - PVE Cluster HA Resource Manager Daemon
Loaded: loaded (/lib/systemd/system/pve-ha-crm.service; enabled; vendor preset: enabled)
Active: inactive (dead)

●pve-ha-lrm.service - PVE Local HA Resource Manager Daemon
Loaded: loaded (/lib/systemd/system/pve-ha-lrm.service; enabled; vendor preset: enabled)
Active: inactive (dead)

● pvebanner.service - Proxmox VE Login Banner
Loaded: loaded (/lib/systemd/system/pvebanner.service; enabled; vendor preset: enabled)
Active: active (exited) since Sat 2023-02-11 13:21:04 CET; 23min ago
Process: 719 ExecStart=/usr/bin/pvebanner (code=exited, status=0/SUCCESS)
Main PID: 719 (code=exited, status=0/SUCCESS)
CPU: 161ms

Feb 11 13:21:03 pve systemd[1]: Starting Proxmox VE Login Banner...
Feb 11 13:21:04 pve systemd[1]: Finished Proxmox VE Login Banner.

● pve-storage.target - PVE Storage Target
Loaded: loaded (/lib/systemd/system/pve-storage.target; static)
Active: active since Sat 2023-02-11 13:21:04 CET; 23min ago

Feb 11 13:21:04 pve systemd[1]: Reached target PVE Storage Target.

● pvedaemon.service - PVE API Daemon
Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
Active: inactive (dead)

● pve-lxc-syscalld.service - Proxmox VE LXC Syscall Daemon
Loaded: loaded (/lib/systemd/system/pve-lxc-syscalld.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2023-02-11 13:21:03 CET; 23min ago
Main PID: 754 (pve-lxc-syscall)
Tasks: 5 (limit: 38141)
Memory: 1.3M
CPU: 2ms
CGroup: /system.slice/pve-lxc-syscalld.service
└─754 /usr/lib/x86_64-linux-gnu/pve-lxc-syscalld/pve-lxc-syscalld --system /run/pve/lxc-syscalld.sock

Feb 11 13:21:03 pve systemd[1]: Starting Proxmox VE LXC Syscall Daemon...
Feb 11 13:21:03 pve systemd[1]: Started Proxmox VE LXC Syscall Daemon.

● pve-guests.service - PVE guests
Loaded: loaded (/lib/systemd/system/pve-guests.service; enabled; vendor preset: enabled)
Active: inactive (dead)

● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
Active: inactive (dead)

● pve-daily-update.timer - Daily PVE download activities
Loaded: loaded (/lib/systemd/system/pve-daily-update.timer; enabled; vendor preset: enabled)
Active: inactive (dead)
Trigger: n/a
Triggers: ● pve-daily-update.service

● pvenetcommit.service - Commit Proxmox VE network changes
Loaded: loaded (/lib/systemd/system/pvenetcommit.service; enabled; vendor preset: enabled)
Active: active (exited) since Sat 2023-02-11 13:21:03 CET; 23min ago
Process: 722 ExecStartPre=/bin/rm -f /etc/openvswitch/conf.db (code=exited, status=0/SUCCESS)
Process: 726 ExecStart=sh -c if [ -f ${FN}.new ]; then mv ${FN}.new ${FN}; fi (code=exited, status=0/SUCCESS)
Main PID: 726 (code=exited, status=0/SUCCESS)
CPU: 1ms

Feb 11 13:21:03 pve systemd[1]: Starting Commit Proxmox VE network changes...
Feb 11 13:21:03 pve systemd[1]: Finished Commit Proxmox VE network changes.

● pvefw-logger.service - Proxmox VE firewall logger
Loaded: loaded (/lib/systemd/system/pvefw-logger.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2023-02-11 13:21:03 CET; 23min ago
Process: 720 ExecStart=/usr/sbin/pvefw-logger (code=exited, status=0/SUCCESS)
Main PID: 737 (pvefw-logger)
Tasks: 2 (limit: 38141)
Memory: 1.9M
CPU: 140ms
CGroup: /system.slice/pvefw-logger.service
└─737 /usr/sbin/pvefw-logger

Feb 11 13:21:03 pve systemd[1]: Starting Proxmox VE firewall logger...
Feb 11 13:21:03 pve pvefw-logger[737]: starting pvefw logger
Feb 11 13:21:03 pve systemd[1]: Started Proxmox VE firewall logger.

● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
Active: inactive (dead)

● pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; vendor preset: enabled)
Active: inactive (dead)

● pvescheduler.service - Proxmox VE scheduler
Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; vendor preset: enabled)
Active: inactive (dead)

 

[Dunuin]

What is the console/syslog/journal showing after boot?

 

[mvank]

could find a direct issues in syslog, but need to confess don;t either know where to look for

 

[mvank]

root@pve:/# cat var/log/syslog | grep error
Feb 11 10:08:29 pve kernel: [ 4.608096] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 10:08:29 pve kernel: [ 4.744045] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Feb 11 10:20:59 pve kernel: [ 4.627889] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 10:20:59 pve kernel: [ 4.778821] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Feb 11 10:38:36 pve kernel: [ 4.597674] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 10:38:36 pve kernel: [ 4.724714] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Feb 11 10:43:04 pve kernel: [ 4.572543] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 10:43:04 pve kernel: [ 4.709236] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Feb 11 10:44:58 pve kernel: [ 4.468172] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 10:44:58 pve kernel: [ 4.611486] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Feb 11 10:47:36 pve kernel: [ 4.611940] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 10:47:36 pve kernel: [ 4.794011] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Feb 11 10:48:17 pve kernel: [ 4.572080] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 10:48:17 pve kernel: [ 4.700045] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Feb 11 10:50:47 pve kernel: [ 5.095091] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 10:50:47 pve kernel: [ 5.236523] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Feb 11 11:00:00 pve kernel: [ 4.596037] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 11:00:00 pve kernel: [ 4.730503] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Feb 11 11:20:45 pve kernel: [ 4.793093] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 11:20:45 pve kernel: [ 4.930690] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Feb 11 13:21:03 pve kernel: [ 4.579066] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro. Quota mode: none.
Feb 11 13:21:03 pve kernel: [ 4.690422] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2

 

[Dunuin]

Check if your root filesystem is running out of space (lsblk or lvs). Because looks like your ext4 is read-only and a full root filesysten often causes this.

 

[mvank]

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p1 259:1 0 1007K 0 part
├─nvme0n1p2 259:2 0 512M 0 part /boot/efi
└─nvme0n1p3 259:3 0 931G 0 part
├─pve-swap 253:0 0 8G 0 lvm [SWAP]
├─pve-root 253:1 0 96G 0 lvm /
├─pve-data_tmeta 253:2 0 8.1G 0 lvm
│ └─pve-data-tpool 253:4 0 794.8G 0 lvm
│ ├─pve-data 253:5 0 794.8G 1 lvm
│ ├─pve-vm--105--disk--0 253:6 0 8G 0 lvm
│ ├─pve-vm--301--disk--0 253:7 0 32G 0 lvm
│ ├─pve-vm--333--disk--0 253:8 0 4M 0 lvm
│ ├─pve-vm--333--disk--1 253:9 0 70G 0 lvm
│ ├─pve-vm--101--disk--0 253:10 0 50G 0 lvm
│ ├─pve-vm--998--disk--0 253:11 0 32G 0 lvm
│ ├─pve-vm--900--disk--0 253:12 0 32G 0 lvm
│ ├─pve-vm--901--disk--0 253:13 0 20G 0 lvm
│ ├─pve-vm--902--disk--0 253:14 0 20G 0 lvm
│ ├─pve-vm--999--disk--0 253:15 0 4M 0 lvm
│ ├─pve-vm--999--disk--1 253:16 0 32G 0 lvm
│ ├─pve-vm--901--state--upd 253:17 0 8.5G 0 lvm
│ └─pve-vm--333--state--upd 253:18 0 8.5G 0 lvm
└─pve-data_tdata 253:3 0 794.8G 0 lvm
└─pve-data-tpool 253:4 0 794.8G 0 lvm
├─pve-data 253:5 0 794.8G 1 lvm
├─pve-vm--105--disk--0 253:6 0 8G 0 lvm
├─pve-vm--301--disk--0 253:7 0 32G 0 lvm
├─pve-vm--333--disk--0 253:8 0 4M 0 lvm
├─pve-vm--333--disk--1 253:9 0 70G 0 lvm
├─pve-vm--101--disk--0 253:10 0 50G 0 lvm
├─pve-vm--998--disk--0 253:11 0 32G 0 lvm
├─pve-vm--900--disk--0 253:12 0 32G 0 lvm
├─pve-vm--901--disk--0 253:13 0 20G 0 lvm
├─pve-vm--902--disk--0 253:14 0 20G 0 lvm
├─pve-vm--999--disk--0 253:15 0 4M 0 lvm
├─pve-vm--999--disk--1 253:16 0 32G 0 lvm
├─pve-vm--901--state--upd 253:17 0 8.5G 0 lvm
└─pve-vm--333--state--upd 253:18 0 8.5G 0 lvm



LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-aotz-- <794.79g 24.77 1.09
root pve -wi-ao---- 96.00g
snap_vm-333-disk-0_upd pve Vri---tz-k 4.00m data vm-333-disk-0
snap_vm-333-disk-1_upd pve Vri---tz-k 70.00g data vm-333-disk-1
snap_vm-901-disk-0_upd pve Vri---tz-k 20.00g data vm-901-disk-0
swap pve -wi-ao---- 8.00g
vm-101-disk-0 pve Vwi-a-tz-- 50.00g data 82.09
vm-105-disk-0 pve Vwi-a-tz-- 8.00g data 70.31
vm-301-disk-0 pve Vwi-a-tz-- 32.00g data 93.83
vm-333-disk-0 pve Vwi-a-tz-- 4.00m data 3.12
vm-333-disk-1 pve Vwi-a-tz-- 70.00g data 87.18
vm-333-state-upd pve Vwi-a-tz-- <8.49g data 46.25
vm-900-disk-0 pve Vwi-a-tz-- 32.00g data 49.48
vm-901-disk-0 pve Vwi-a-tz-- 20.00g data 42.91
vm-901-state-upd pve Vwi-a-tz-- <8.49g data 15.98
vm-902-disk-0 pve Vwi-a-tz-- 20.00g data 91.52
vm-998-disk-0 pve Vwi-a-tz-- 32.00g data 0.01
vm-999-disk-0 pve Vwi-a-tz-- 4.00m data 0.00
vm-999-disk-1 pve Vwi-a-tz-- 32.00g data 12.03

 

[mvank]

I have decided to reinstall which solved the problem. Not the best way but the quickest