Read-only file system

Popalmox · Jan 31, 2023

Hi,

I'm a new user.

I've got several hints that my file system is read-only since about January 9th. The PVE has been up for a couple of months with no issues until then. The lastbackup has been made Jan 8 for instance.

I could't access the WebGUI for some reason but could successfully connect over SSH. I've made some searches in the forum but could not get anything that would match my issue.

Fortunately, every app I installed are still working (because they use another disk I guess) and I'm wondering if it's safe for me to reboot it or do something else.

Here are some quick specs about my system :

Based on a 1 liter PC (Dell Optiplex USFF 3080)
PVE is hosted on a 120gbs SATA SSD
The apps are installed on a second 2 TB NVME SSD
Backups and remote ressources (media, etc) are hosted on a Synology NAS

Can you help me on this ? Thanks !

Regards

Popalmox · Jan 31, 2023

Tried :

Code:

mount -o remount,rw /

got :

Code:

mount: /: cannot remount /dev/mapper/pve-root read-write, is write-protected.

bbgeek17 · Jan 31, 2023

run " journalctl --since "2023-01-09" or earlier and examine the log closely to find out what happened. May be reboot will help, or may be it will never come back again.
There are thousand and one reason for the state you are in, only log examination can help you zoom in.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Popalmox · Jan 31, 2023

bbgeek17 said:
run " journalctl --since "2023-01-09" or earlier and examine the log closely to find out what happened. May be reboot will help, or may be it will never come back again.
There are thousand and one reason for the state you are in, only log examination can help you zoom in.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

* Edit : added a couple of lines before.

Good call, here's what I've got :

Code:

Jan 09 01:10:51 altair systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab...
Jan 09 01:10:51 altair fstrim[1891277]: /boot/efi: 510.6 MiB (535449600 bytes) trimmed on /dev/sda2
Jan 09 01:10:51 altair fstrim[1891277]: /: 23.3 GiB (25059258368 bytes) trimmed on /dev/pve/root
Jan 09 01:10:51 altair systemd[1]: fstrim.service: Succeeded.
Jan 09 01:10:51 altair systemd[1]: Finished Discard unused blocks on filesystems from /etc/fstab.
Jan 09 01:17:01 altair CRON[1892652]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 09 01:17:01 altair CRON[1892653]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jan 09 01:17:01 altair CRON[1892652]: pam_unix(cron:session): session closed for user root
Jan 09 01:37:20 altair kernel: sd 0:0:0:0: [sda] tag#13 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=30s
Jan 09 01:37:20 altair kernel: sd 0:0:0:0: [sda] tag#13 CDB: Write(10) 2a 00 02 97 98 c8 00 00 18 00
Jan 09 01:37:20 altair kernel: sd 0:0:0:0: [sda] tag#13 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=30s
Jan 09 01:37:20 altair kernel: sd 0:0:0:0: [sda] tag#13 CDB: Write(10) 2a 00 02 97 98 c8 00 00 18 00
Jan 09 01:37:20 altair kernel: blk_update_request: I/O error, dev sda, sector 43489480 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
Jan 09 01:37:20 altair kernel: Aborting journal on device dm-4-8.
Jan 09 01:37:20 altair kernel: EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm pmxcfs: Detected aborted journal
Jan 09 01:37:20 altair kernel: EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm rs:main Q:Reg: Detected aborted journal
Jan 09 01:37:20 altair kernel: EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal
Jan 09 01:37:20 altair kernel: EXT4-fs (dm-4): Remounting filesystem read-only
Jan 09 01:37:20 altair rsyslogd[765]: file '/var/log/kern.log': open error: Read-only file system [v8.2102.0 try https://www.rsyslog.com/e/2433 ]
Jan 09 01:37:20 altair pvescheduler[1897088]: replication: can't lock file '/var/lib/pve-manager/pve-replication-state.lck' - can't open file - Read-only file system
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Jan 09 01:37:20 altair rsyslogd[765]: action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]

bbgeek17 · Jan 31, 2023

Popalmox said:
Jan 09 01:37:20 altair kernel: blk_update_request: I/O error, dev sda, sector 43489480 op 0x1WRITE) flags 0x800 phys_seg 3 prio class 0

disk error, reboot will likely bring it back up until you hit the next bad block. Time to look into changing the disk.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Popalmox · Jan 31, 2023

bbgeek17 said:
disk error, reboot will likely bring it back up until you hit the next bad block. Time to look into changing the disk.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thanks. Do you think I could just clone the disk to another one and use this new one ?

bbgeek17 · Jan 31, 2023

May be, if you get lucky. Or may be it will cause total disk failure due to increased access, who know...

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Popalmox · Feb 1, 2023

I've made some progress and wanna give you some updates.

The OS completely froze over night and was offline this morning when I woke up. I took care of it after work with a cold reboot at first.
Still offline, I plugged a monitor in and could see something like that

WhatsApp Image 2023-02-01 à 21.43.41.jpg

The fsck command and the soft reboot could push it back online.

I'm now running my backups.

What are my next actions ? The host SSD could be gone for good pretty quickly. Isn't there a way to properly backup the PVE and restore everything after a fresh install on a new disk ?

Dunuin · Feb 1, 2023

Popalmox said:
Isn't there a way to properly backup the PVE and restore everything after a fresh install on a new disk ?

For backing up/restorong the PVE host configs in case you can access the data but PVE won'T boot any longer or pve-cluster service wont work anymore: https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)#_recovery

Search

Search

Read-only file system

Popalmox

New Member

Popalmox

New Member

bbgeek17

Distinguished Member

Popalmox

New Member

bbgeek17

Distinguished Member

Popalmox

New Member

bbgeek17

Distinguished Member

Popalmox

New Member

Dunuin

Distinguished Member