[SOLVED] Read-only filesystem

[proxman4]

Hi there.

I got a problem with a node on my cluster.

Here is the conf:

Proxmox 7.1 .

I tries to remove a lvm-thin provisionning via Datacenter Webui interface. I rebooted one of the nodes where the lvm thin is and i got a read-only filesystem.

the system is installed like this :

softraid1 on nvme :

nvme1n1                                              259:0    0   477G  0 disk                                                                                                               

├─nvme1n1p1                                          259:1    0   511M  0 part  /boot/efi                                                                                                     

├─nvme1n1p2                                          259:2    0  58,6G  0 part                                                                                                               

│ └─md2                                                9:2    0  58,6G  0 raid1 /                                                                                                             

├─nvme1n1p3                                          259:3    0     8G  0 part  [SWAP]                                                                                                       

└─nvme1n1p4                                          259:4    0 409,9G  0 part                                                                                                               

  └─md0                                                9:0    0 409,7G  0 raid1

    ├─vg_nvme-lv_varlibvz                            253:5    0   100G  0 lvm   /var/lib/vz               


nvme0n1                                              259:5    0   477G  0 disk                                                                                                               

├─nvme0n1p1                                          259:6    0   511M  0 part                                                                                                               

├─nvme0n1p2                                          259:7    0  58,6G  0 part                                                                                                               

│ └─md2                                                9:2    0  58,6G  0 raid1 /                                                                                                             

├─nvme0n1p3                                          259:8    0     8G  0 part  [SWAP]                                                                                                       

├─nvme0n1p4                                          259:9    0     1M  0 part                                                                                                               

└─nvme0n1p5                                          259:10   0 409,9G  0 part                                                                                                               

  └─md0                                                9:0    0 409,7G  0 raid1                                                                                                               

    ├─vg_nvme-lv_varlibvz                            253:5    0   100G  0 lvm   /var/lib/vz

my /proc/mdstat :

md1 : active raid1 sda1[0] sdb1[1]
      3906885440 blocks super 1.2 [2/2] [UU]
      bitmap: 0/30 pages [0KB], 65536KB chunk

md0 : active raid1 nvme0n1p5[0] nvme1n1p4[1]
      429618176 blocks super 1.2 [2/2] [UU]
      bitmap: 0/4 pages [0KB], 65536KB chunk

md2 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
      61438912 blocks [2/2] [UU]

Boot looks ok :

mars 21 02:41:53 OP1 kernel: md/raid1:md1: active with 2 out of 2 mirrors
mars 21 02:41:53 OP1 kernel: md1: detected capacity change from 0 to 4000650690560
mars 21 02:41:53 OP1 kernel: EXT4-fs (md2): mounted filesystem with ordered data mode. Opts: (null)

Then :

kernel: EXT4-fs (md2): Unrecognized mount option "0" or missing value

And the filesystem in in read-Only mode .

TARGET                                SOURCE                                FSTYPE     OPTIONS                                                                                               
/                                     /dev/md2                              ext4       ro,relatime

For now i can access ssh read the logs i'm providing to you.

Server in prod i feel like the night will be long.

So i'm asking here if some kind stranger have some clue for me. Secretlely hoping being told : "just fsck /dev/md2 from a livecd dude ."

thanks.

 

[proxman4]

There is an update :

I managed to reboot the proxmox via an ipmi rescue mode.

did fsck on /dev/md2 and it's clean.

Tried to mount / as rw manually on /mnt


I could remove the

errors=remount-ro

option but i really don't feel comfortable with this.
Besides with ro i can access all my data and that is a very convenient way to approach this problem.

I'm pretty sure i can explore other tracks to find out the real reason /dev/md2 is mounting as ro.

 

[fabian]

maybe post the full boot journal and the output of mount?

 

[proxman4]

Thanks for you reply:

Here is the /etc/fstab

UUID="3a4c20fe-2711-41f9-b972-dbcd93b70ae0" / errors=remount-ro 0       1

# swap-nvme0n1p3
UUID="ff3c6a30-f814-4682-ae1b-f45db059b911" swap swap defaults 0 0

# swap-nvme1n1p3
UUID="a90daf35-0517-4024-8655-7fb78d48076c" swap swap defaults 0 0

LABEL=EFI_SYSPART /boot/efi vfat defaults 0 0

# /var/lib/vz
UUID="0761202a-8aa9-47e1-9de1-64df20517e66"     /var/lib/vz     ext4    defaults 0 0

# /opt/nextcloud-data
#UUID="9297a6e2-a1c4-4151-bcf7-24f2d2880426"    /opt/nextcloud-data     btrfs   defaults 0 0

Here is the mount output :

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32787680k,nr_inodes=8196920,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6563108k,mode=755)
/dev/md2 on / type ext4 (ro,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/nvme1n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mapper/vg_nvme-lv_varlibvz on /var/lib/vz type ext4 (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=6563104k,mode=700,uid=1001,gid=1001)

I checked the disks via the rescue-ipmi and smart test is good.
It looks like the raid 1 and the FS are ok

I'm a little puzzled.

 

[fabian]

the fstab line for '/' is missing the 'ext4' as type, so the options are parsed as fstype, and the 0 as options

 

[proxman4]

Right i didn't see it even if i noticed the error in kernel logs.

Working with few sleep is really a bad choice.

It worked !!!

If you come to Paris one day PM me i'm buying you a restaurant.

Thank you very much .

Long live to proxmox.

 

[sobatputra]

hai... you can help

I have a problem very similar to you

my server proxmox error file system read only after reboot

i used user root for login my server

 

[fabian]

you need to find out why it's read only..

 

[sobatputra]

that's why I asked

I just restarted the server and suddenly it happened

can you help m

you need to find out why it's read only..

 

[fabian]

then check the boot log (journalctl -b) - a filesystem doesn't just "go read-only"