Random Read-Only Filesystem Error

[alerej]

Every 1-3 months on average, all resources on my Proxmox server will become unresponsive/ unavailable, as is the web UI. The error that is usually coming back when I try to access the resources from the Proxmox app on my phone is

Starting worker failed. Unable to create output file read only file system.

My first instinct to troubleshoot this is to check the journal for the server, however I don't see any useful or related logs to the issue.



I was wondering if somebody else may have experienced this before or know where I could start troubleshooting. The only related proxmox forum answers are related to disk issues which I don't see any of.

 

[Dunuin]

Stand-alone PVE node or cluster?

Usually it is full root filesystem, storage problems or quorum lost.

 

[alerej]

Stand-alone PVE node or cluster?

Usually it is full root filesystem, storage problems or quorum lost.

Stand-alone PVE. Looks like I have a good amount of space left still

 

[intrax]

What filesystem?

 

[alerej]

What filesystem?

 

[intrax]

Boot from a live linux system on usb and run 'fsck /dev/mapper/pve-root' to see what the problem is with your root partition.
Did you resize the partition before ?

 

[Kingneutron]

If an ext4 rootfs starts remounting itself read-only, it's due to errors. It's defined that way in /etc/fstab

Do a SMART long test on the drive and be ready to replace it / restore from backup

 

[alerej]

If an ext4 rootfs starts remounting itself read-only, it's due to errors. It's defined that way in /etc/fstab

Do a SMART long test on the drive and be ready to replace it / restore from backup

Sorry for the delay, but this has happened again so I went ahead and ran the test on my host drive (/dev/nvme0n1). As far as I can tell, the test passed without any issues.

 

[intrax]

Well 39 unsafe shutdowns ! Not so good...

 

[Kingneutron]

The drive is not suitable for Proxmox. Make sure everything is on UPS power and replace it with something that has a high TBW rating

 

[alerej]

The drive is not suitable for Proxmox. Make sure everything is on UPS power and replace it with something that has a high TBW rating

Do you have any recommendations?

 

[Kingneutron]

My Lexar NM790 1TB is still at 0% wear, running mostly 24/7 since Feb. I turn off 'atime' everywhere (including in-guest) and run log2ram

 

[alerej]

My Lexar NM790 1TB is still at 0% wear, running mostly 24/7 since Feb. I turn off 'atime' everywhere (including in-guest) and run log2ram

Just to make sure it's the drive, I'm running a Samsung 970 Evo 1TB. I'm surprised that that might be the problem. Only mentioning that because I see the drive name isn't in my screenshot.

 

[Kingneutron]

Evo is desktop-class. Stay away from those for serious applications, and also stay away from the 990 "pro" - terrible firmware issues

 

[milew]

Evo is desktop-class. Stay away from those for serious applications, and also stay away from the 990 "pro" - terrible firmware issues

Lexar NM790 1TB is enterprise? Are you satisfied with this series of disks? Do you have them running under ZFS in some raidz or mirror configuration?
I'm looking for some drives with the size M.2 2280 max.

 

[leesteken]

Lexar NM790 1TB is enterprise?

I don't think it has PLP or a particular high DWPD.

I'm looking for some drives with the size M.2 2280 max.

There are not many M.2 with PLP but there is Micron 7400 and 7450 (which I do enjoy) but also maybe Kingston DC? Samsung PM is 22110. I think most enterprise drives use U.2 or U.3

 

[Kingneutron]

Lexar NM790 1TB is enterprise? Are you satisfied with this series of disks? Do you have them running under ZFS in some raidz or mirror configuration?
I'm looking for some drives with the size M.2 2280 max.

No probably not enterprise, but 0% wear indicator since Feb running 24/7 is pretty good for homelab. It has a ~1000TBW rating, which is several years of continuous use. With log2ram, zram and ~7 virtual disks on it, it's not getting anywhere near that kind of write I/O on a daily basis

My personal drive is 100% single-disk zpool with atime turned off everywhere and no clustering services

https://www.amazon.com/gp/product/B0CGKPPZY9

 

[alerej]

No probably not enterprise, but 0% wear indicator since Feb running 24/7 is pretty good for homelab. It has a ~1000TBW rating, which is several years of continuous use. With log2ram, zram and ~7 virtual disks on it, it's not getting anywhere near that kind of write I/O on a daily basis

My personal drive is 100% single-disk zpool with atime turned off everywhere and no clustering services

https://www.amazon.com/gp/product/B0CGKPPZY9

Just ordered this. I'll update if I start seeing the same behavior after I install it!