[SOLVED] Why do I get "pve-root needs manual fsck" at INITRAMFS?

[JonMikelV]

I've been using Proxmox VE 7.0.1 for about 6 months and just now ran into the second instance of the GUI not working (it starts to load but nothing is displayed & ext js errors show in browser console).

Using SSH to connect then systemctl to try and restart the UI just returned errors (sorry, I don't have them avail. now) and attempting to update complained about a read-only drive somewhere (again, no actual message avail. to share).

From the physical console I am able to reboot, but it just shows these messages:

/dev/mapper/pve-root contains a file system with errors, check forced.
/dev/mapper/pve-root:
Inodes that were part of a corrupted orphan linked list found.

/dev/mapper/pve-root: UNEXPECTED INCONCISTENCY; RUN fsck MANUALY.
(i.e., without -a or -p options)
fsck exited with status code 54
the root filesystem on /dev/mapper/pve-root requires a manual fsck

I ran fsck -f -c -y /dev/mapper/pve-root and it scrolled a bunch of Free blocks count wrong and Inode bitmap differences messages all with a Fix? yes following them and ultimately said /dev/mapper/pve-root: ***** FILE SYSTEM WAS MODIFIED *****.

I ran it again just to be sure and I didn't get any of the scary "count wrong" or "bitmap differences" stuff, but did get the finale FILE SYSTEM WAS MODIFIED for some reason. The console reboot command didn't work at that point so I did a manual power cycle.

While this does seem to have resolved my current issue (GUI & VMs work and I'm now on v7.1-10) without having to re-install, does anybody have thoughts on why this has happened to me twice now and/or how I can avoid it in the future?

Thanks.

 

[mira]

This sounds like a failing disk.
Have you checked your disks with smartctl?

 

[JonMikelV]

Thanks for the smart suggestion. All disks come up with no errors which aligns with the PASSED result that I'm seeing in the Disks UI.

I'm still new to Proxmox so don't know all the places to look for stuff, but I did find the Syslog in the GUI and found some not-recent instances of:

proxmox kernel: EDAC MC0: 1 UE x38 UE on mc#0csrow#1channel#0 or mc#0csrow#1channel#1 (csrow:1 page:0x0 offset:0x0 grain:1073741824)

If I'm reading things correctly that indicates a memory issue, which is odd as I've run memtest+ at boot recently without any errors. Hopefully it's a thermal issue with a bad DIMM and not a motherboard problem.

[A little while later....] I installed memtest and ran it on avail. memory while the system was actively hosting a VM and it reported failures (along with more entries in syslog). Guess it's time for a new stick.

Any other thoughts / suggestions (or corrections on my assumptions) are welcome.

 

[mira]

So the memtest errors were while the host was running with a typical load?
Usually it takes a lot of passes of the complete memory to find something (booting the memtest to make sure the whole memory can be tested).

I'd suggest replacing your faulty DIMM(s) as soon as possible.

 

[JonMikelV]

It's odd - multiple runs of Grub Memtest86+ didn't find any issue but using memtester in Proxmox console I was able to trigger the error enough that I'm confident I got the bad DIMM replaced (no more memtester error).

While the #csrow part of the error helped pinpoint the specific stick with issues, I did the following to ensure error was triggerable before and NOT-triggerable after replacing.
- log into console (local or shell via UI)
- run "vmstat" (to determine how much memory I could play with)
- run "memtester #####k 1" (##### is value from memory "free" column, in kb, run once)
- ensure "EDAC mc0" errors did (before mem swap) or did not (after mem swap) appear

I'm considering this SOLVED under assumption that memory issues were causing bad writes thus screwing up pve-root mention in original error.

Thanks for your assistance!