Random Read-Only Filesystem Error

alerej

New Member
Oct 20, 2023
8
1
3
Every 1-3 months on average, all resources on my Proxmox server will become unresponsive/ unavailable, as is the web UI. The error that is usually coming back when I try to access the resources from the Proxmox app on my phone is


Starting worker failed. Unable to create output file read only file system.

My first instinct to troubleshoot this is to check the journal for the server, however I don't see any useful or related logs to the issue.

Screenshot 2024-06-26 165351.png

I was wondering if somebody else may have experienced this before or know where I could start troubleshooting. The only related proxmox forum answers are related to disk issues which I don't see any of.
 
Stand-alone PVE node or cluster?

Usually it is full root filesystem, storage problems or quorum lost.
 
Boot from a live linux system on usb and run 'fsck /dev/mapper/pve-root' to see what the problem is with your root partition.
Did you resize the partition before ?
 
Last edited:
  • Like
Reactions: Kingneutron
If an ext4 rootfs starts remounting itself read-only, it's due to errors. It's defined that way in /etc/fstab

Do a SMART long test on the drive and be ready to replace it / restore from backup
 
  • Like
Reactions: intrax
If an ext4 rootfs starts remounting itself read-only, it's due to errors. It's defined that way in /etc/fstab

Do a SMART long test on the drive and be ready to replace it / restore from backup
Sorry for the delay, but this has happened again so I went ahead and ran the test on my host drive (/dev/nvme0n1). As far as I can tell, the test passed without any issues.
 

Attachments

  • Screenshot_20240803_091607_Termius.png
    Screenshot_20240803_091607_Termius.png
    318.1 KB · Views: 11
My Lexar NM790 1TB is still at 0% wear, running mostly 24/7 since Feb. I turn off 'atime' everywhere (including in-guest) and run log2ram
Just to make sure it's the drive, I'm running a Samsung 970 Evo 1TB. I'm surprised that that might be the problem. Only mentioning that because I see the drive name isn't in my screenshot.
 
Evo is desktop-class. Stay away from those for serious applications, and also stay away from the 990 "pro" - terrible firmware issues
 
Evo is desktop-class. Stay away from those for serious applications, and also stay away from the 990 "pro" - terrible firmware issues
Lexar NM790 1TB is enterprise? Are you satisfied with this series of disks? Do you have them running under ZFS in some raidz or mirror configuration?
I'm looking for some drives with the size M.2 2280 max.
 
Lexar NM790 1TB is enterprise? Are you satisfied with this series of disks? Do you have them running under ZFS in some raidz or mirror configuration?
I'm looking for some drives with the size M.2 2280 max.

No probably not enterprise, but 0% wear indicator since Feb running 24/7 is pretty good for homelab. It has a ~1000TBW rating, which is several years of continuous use. With log2ram, zram and ~7 virtual disks on it, it's not getting anywhere near that kind of write I/O on a daily basis

My personal drive is 100% single-disk zpool with atime turned off everywhere and no clustering services

https://www.amazon.com/gp/product/B0CGKPPZY9
 
No probably not enterprise, but 0% wear indicator since Feb running 24/7 is pretty good for homelab. It has a ~1000TBW rating, which is several years of continuous use. With log2ram, zram and ~7 virtual disks on it, it's not getting anywhere near that kind of write I/O on a daily basis

My personal drive is 100% single-disk zpool with atime turned off everywhere and no clustering services

https://www.amazon.com/gp/product/B0CGKPPZY9
Just ordered this. I'll update if I start seeing the same behavior after I install it!
 
  • Like
Reactions: Kingneutron

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!