Hi all,
Trying upgrading my server from 8.4 to 9.1 but after the upgrade the server fails somehow and starts the server with root in ro mode. Starting same server with the 8.4 kernel and everything works as expected.
Old: pve-manager/8.4.16/368e3c45c15b895c (running kernel: 6.8.12-17-pve)
New: pve-manager/9.1.4/5ac30304265fbd8e (running kernel: 6.17.4-2-pve)
I have tried changing root option in grub from UUID to /dev/xx
After server boots this is the result:
sudo systemctl list-units --failed
[sudo] password for mir:
UNIT LOAD ACTIVE SUB DESCRIPTION >
● apparmor.service loaded failed failed Load AppArmor profiles
● corosync.service loaded failed failed Corosync Cluster Engine
● grub-common.service loaded failed failed Record successful boot>
● opensm@0x0008f104039a3505.service loaded failed failed Starts the OpenSM Infi>
● opensm@0x0008f104039a3506.service loaded failed failed Starts the OpenSM Infi>
● postfix.service loaded failed failed Postfix Mail Transport>
● pve-cluster.service loaded failed failed The Proxmox VE cluster>
● pve-firewall.service loaded failed failed Proxmox VE firewall
● pve-guests.service loaded failed failed PVE guests
● pve-ha-crm.service loaded failed failed PVE Cluster HA Resourc>
● pve-ha-lrm.service loaded failed failed PVE Local HA Resource >
● pvefw-logger.service loaded failed failed Proxmox VE firewall lo>
● pvescheduler.service loaded failed failed Proxmox VE scheduler
● pvestatd.service loaded failed failed PVE Status Daemon
● systemd-remount-fs.service loaded failed failed Remount Root and Kerne>
Sudo journalctl -k gives this interesting error message:
Jan 02 23:52:12 esx2 systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Jan 02 23:52:12 esx2 systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Jan 02 23:52:12 esx2 systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
sudo dmesg show this:
[ 11.010202] EXT4-fs (sda1): can't disable delalloc during remount
[ 11.019399] systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
[ 11.022056] systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
[ 11.025012] systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
Hardware:
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: AuthenticAMD
Model name: AMD Opteron(tm) Processor 3365
CPU family: 21
Model: 2
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 1
Stepping: 0
Frequency boost: enabled
CPU(s) scaling MHz: 80%
CPU max MHz: 2300.0000
CPU min MHz: 1100.0000
BogoMIPS: 4611.33
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 ss
e4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lo
ck nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
Virtualization features:
Virtualization: AMD-V
Caches (sum of all):
L1d: 128 KiB (8 instances)
L1i: 256 KiB (4 instances)
L2: 8 MiB (4 instances)
L3: 8 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerabilities:
Gather data sampling: Not affected
Ghostwrite: Not affected
Indirect target selection: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Old microcode: Not affected
Reg file data sampling: Not affected
Retbleed: Mitigation; untrained return thunk; SMT disabled
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Srbds: Not affected
Tsa: Not affected
Tsx async abort: Not affected
Vmscape: Not affected
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Supermicro
Product Name: H8SML
Version: 1234567890
Serial Number: 1234567890
UUID: 49434d53-0200-90dd-2500-dd9025005ef3
Wake-up Type: Power Switch
SKU Number: To Be Filled By O.E.M.
Family: To Be Filled By O.E.M.
Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: Supermicro
Product Name: H8SML
Version: 1234567890
Serial Number: ZM13CS600464
Asset Tag: To Be Filled By O.E.M.
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: To Be Filled By O.E.M.
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0
Handle 0x0003, DMI type 3, 21 bytes
Chassis Information
Manufacturer: Supermicro
Type: Main Server Chassis
Lock: Not Present
Version: 1234567890
Serial Number: 1234567890
Asset Tag: To Be Filled By O.E.M.
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: None
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: 1
Contained Elements: 0
Help is much appreciated.
Trying upgrading my server from 8.4 to 9.1 but after the upgrade the server fails somehow and starts the server with root in ro mode. Starting same server with the 8.4 kernel and everything works as expected.
Old: pve-manager/8.4.16/368e3c45c15b895c (running kernel: 6.8.12-17-pve)
New: pve-manager/9.1.4/5ac30304265fbd8e (running kernel: 6.17.4-2-pve)
I have tried changing root option in grub from UUID to /dev/xx
After server boots this is the result:
sudo systemctl list-units --failed
[sudo] password for mir:
UNIT LOAD ACTIVE SUB DESCRIPTION >
● apparmor.service loaded failed failed Load AppArmor profiles
● corosync.service loaded failed failed Corosync Cluster Engine
● grub-common.service loaded failed failed Record successful boot>
● opensm@0x0008f104039a3505.service loaded failed failed Starts the OpenSM Infi>
● opensm@0x0008f104039a3506.service loaded failed failed Starts the OpenSM Infi>
● postfix.service loaded failed failed Postfix Mail Transport>
● pve-cluster.service loaded failed failed The Proxmox VE cluster>
● pve-firewall.service loaded failed failed Proxmox VE firewall
● pve-guests.service loaded failed failed PVE guests
● pve-ha-crm.service loaded failed failed PVE Cluster HA Resourc>
● pve-ha-lrm.service loaded failed failed PVE Local HA Resource >
● pvefw-logger.service loaded failed failed Proxmox VE firewall lo>
● pvescheduler.service loaded failed failed Proxmox VE scheduler
● pvestatd.service loaded failed failed PVE Status Daemon
● systemd-remount-fs.service loaded failed failed Remount Root and Kerne>
Sudo journalctl -k gives this interesting error message:
Jan 02 23:52:12 esx2 systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Jan 02 23:52:12 esx2 systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Jan 02 23:52:12 esx2 systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
sudo dmesg show this:
[ 11.010202] EXT4-fs (sda1): can't disable delalloc during remount
[ 11.019399] systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
[ 11.022056] systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
[ 11.025012] systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
Hardware:
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: AuthenticAMD
Model name: AMD Opteron(tm) Processor 3365
CPU family: 21
Model: 2
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 1
Stepping: 0
Frequency boost: enabled
CPU(s) scaling MHz: 80%
CPU max MHz: 2300.0000
CPU min MHz: 1100.0000
BogoMIPS: 4611.33
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 ss
e4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lo
ck nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
Virtualization features:
Virtualization: AMD-V
Caches (sum of all):
L1d: 128 KiB (8 instances)
L1i: 256 KiB (4 instances)
L2: 8 MiB (4 instances)
L3: 8 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerabilities:
Gather data sampling: Not affected
Ghostwrite: Not affected
Indirect target selection: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Old microcode: Not affected
Reg file data sampling: Not affected
Retbleed: Mitigation; untrained return thunk; SMT disabled
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Srbds: Not affected
Tsa: Not affected
Tsx async abort: Not affected
Vmscape: Not affected
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Supermicro
Product Name: H8SML
Version: 1234567890
Serial Number: 1234567890
UUID: 49434d53-0200-90dd-2500-dd9025005ef3
Wake-up Type: Power Switch
SKU Number: To Be Filled By O.E.M.
Family: To Be Filled By O.E.M.
Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: Supermicro
Product Name: H8SML
Version: 1234567890
Serial Number: ZM13CS600464
Asset Tag: To Be Filled By O.E.M.
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: To Be Filled By O.E.M.
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0
Handle 0x0003, DMI type 3, 21 bytes
Chassis Information
Manufacturer: Supermicro
Type: Main Server Chassis
Lock: Not Present
Version: 1234567890
Serial Number: 1234567890
Asset Tag: To Be Filled By O.E.M.
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: None
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: 1
Contained Elements: 0
Help is much appreciated.
