Somehow my file system got set to read only, and now I don't know what to do.

bobloadmire

New Member
May 17, 2024
16
2
3
Installing a Plex LXC and im getting this error: E: Sub-process /usr/bin/dpkg returned an error code (2) W: Problem unlinking the file /var/cache/apt/pkgcache.bin - pkgDPkgPM::Go (30: Read-only file system)

Additionally proxmox locks up entirely and needs a reboot. Anyone have any suggestions?

I have no idea how this happened unfortunately, I didn't manually mess with any folder permissions, to my knowledge.
 
[ 2.485863] AES CTR mode by8 optimization enabled
[ 2.500081] snd_hda_intel 0000:00:1f.3: enabling device (0000 -> 0002)
[ 2.612536] Bluetooth: hci0: RTL: fw version 0xdfc6d922
[ 3.035051] i915 0000:00:02.0: [drm] VT-d active for gfx access
[ 3.035086] i915 0000:00:02.0: vgaarb: deactivate vga console
[ 3.035144] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[ 3.035568] i915 0000:00:02.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 3.036211] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_ops [i915])
[ 3.038522] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc.bin (v2.20)
[ 3.044652] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/adlp_guc_70.bin version 70.20.0
[ 3.044658] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[ 3.061872] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads
[ 3.062289] i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
[ 3.062291] i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
[ 3.062667] i915 0000:00:02.0: [drm] GT0: GUC: RC enabled
[ 3.077952] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
[ 3.078132] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[ 3.137267] [drm] Initialized i915 1.6.0 20230929 for 0000:00:02.0 on minor 0
[ 3.138258] ACPI: video: Video Device [GFX0] (multi-head: yes rom: no post: no)
[ 3.138729] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input7
[ 3.138836] i915 display info: display version: 13
[ 3.138839] i915 display info: cursor_needs_physical: no
[ 3.138840] i915 display info: has_cdclk_crawl: yes
[ 3.138841] i915 display info: has_cdclk_squash: no
[ 3.138841] i915 display info: has_ddi: yes
[ 3.138842] i915 display info: has_dp_mst: yes
[ 3.138843] i915 display info: has_dsb: yes
[ 3.138844] i915 display info: has_fpga_dbg: yes
[ 3.138845] i915 display info: has_gmch: no
[ 3.138845] i915 display info: has_hotplug: yes
[ 3.138846] i915 display info: has_hti: no
[ 3.138847] i915 display info: has_ipc: yes
[ 3.138847] i915 display info: has_overlay: no
[ 3.138848] i915 display info: has_psr: yes
[ 3.138849] i915 display info: has_psr_hw_tracking: no
[ 3.138849] i915 display info: overlay_needs_physical: no
[ 3.138850] i915 display info: supports_tv: no
[ 3.138851] i915 display info: has_hdcp: yes
[ 3.138852] i915 display info: has_dmc: yes
[ 3.138852] i915 display info: has_dsc: yes
[ 3.138918] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[ 3.139022] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[ 3.139071] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[ 3.139024] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[ 3.152713] intel_tcc_cooling: TCC Offset locked
[ 3.171293] input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input8
[ 3.171334] input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9
[ 3.171367] input: HDA Intel PCH HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10
[ 3.171398] input: HDA Intel PCH HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11
[ 3.437668] intel_rapl_msr: PL4 support detected.
[ 3.443399] intel_rapl_common: Found RAPL domain package
[ 3.443405] intel_rapl_common: Found RAPL domain core
[ 3.443405] intel_rapl_common: Found RAPL domain uncore
[ 3.443407] intel_rapl_common: Found RAPL domain psys
[ 3.491001] ZFS: Loaded module v2.2.3-pve2, ZFS pool version 5000, ZFS filesystem version 5
[ 3.543095] audit: type=1400 audit(1715956410.613:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-copy" pid=643 comm="apparmor_parser"
[ 3.543101] audit: type=1400 audit(1715956410.613:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="pve-container-mounthotplug" pid=641 comm="apparmor_parser"
[ 3.544568] audit: type=1400 audit(1715956410.615:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=638 comm="apparmor_parser"
[ 3.544963] audit: type=1400 audit(1715956410.615:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=647 comm="apparmor_parser"
[ 3.545397] audit: type=1400 audit(1715956410.615:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=640 comm="apparmor_parser"
[ 3.545403] audit: type=1400 audit(1715956410.615:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=640 comm="apparmor_parser"
[ 3.546725] audit: type=1400 audit(1715956410.617:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=648 comm="apparmor_parser"
[ 3.546818] audit: type=1400 audit(1715956410.617:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=648 comm="apparmor_parser"
[ 3.546821] audit: type=1400 audit(1715956410.617:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=648 comm="apparmor_parser"
[ 3.582926] RPC: Registered named UNIX socket transport module.
[ 3.582930] RPC: Registered udp transport module.
[ 3.582931] RPC: Registered tcp transport module.
[ 3.582932] RPC: Registered tcp-with-tls transport module.
[ 3.582933] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 4.055311] vmbr0: port 1(enp4s0) entered blocking state
[ 4.055316] vmbr0: port 1(enp4s0) entered disabled state
[ 4.055326] igc 0000:04:00.0 enp4s0: entered allmulticast mode
[ 4.055361] igc 0000:04:00.0 enp4s0: entered promiscuous mode
[ 7.232811] igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
[ 7.232930] vmbr0: port 1(enp4s0) entered blocking state
[ 7.232932] vmbr0: port 1(enp4s0) entered forwarding state
[ 7.868229] VFIO - User Level meta-driver version: 0.3
[ 7.872919] igc 0000:02:00.0 enp2s0: PHC removed
[ 8.081125] igc 0000:03:00.0 enp3s0: PHC removed
[ 20.478782] EXT4-fs warning (device dm-15): ext4_multi_mount_protect:328: MMP interval 42 higher than expected, please wait.

[ 62.732016] EXT4-fs (dm-15): recovery complete
[ 62.734113] EXT4-fs (dm-15): mounted filesystem f51a504d-b6bf-4641-80c9-70ffc795aa43 r/w with ordered data mode. Quota mode: none.
[ 62.777812] kauditd_printk_skb: 17 callbacks suppressed
[ 62.777815] audit: type=1400 audit(1715956470.214:28): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=1230 comm="apparmor_parser"
[ 63.138255] vmbr0: port 2(veth102i0) entered blocking state
[ 63.138260] vmbr0: port 2(veth102i0) entered disabled state
[ 63.138277] veth102i0: entered allmulticast mode
[ 63.138312] veth102i0: entered promiscuous mode
[ 63.170967] eth0: renamed from vethY3nKIY
[ 63.528464] vmbr0: port 2(veth102i0) entered blocking state
[ 63.528468] vmbr0: port 2(veth102i0) entered forwarding state
[ 63.624574] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[ 63.624810] Loaded X.509 cert 'benh@debian.org: 577e021cb980e0e820821ba7b54b4961b8b4fadf'
[ 63.624965] Loaded X.509 cert 'romain.perier@gmail.com: 3abbc6ec146e09d1b6016ab9d6cf71dd233f0328'
[ 63.625110] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[ 63.625250] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
[ 63.625301] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[ 63.625305] cfg80211: failed to load regulatory.db
[ 73.915940] EXT4-fs warning (device dm-10): ext4_multi_mount_protect:328: MMP interval 42 higher than expected, please wait.

[ 116.481069] EXT4-fs (dm-10): recovery complete
[ 116.483034] EXT4-fs (dm-10): mounted filesystem eb7fabb5-cca0-40a6-a425-8e3defbc57d5 r/w with ordered data mode. Quota mode: none.
[ 116.515234] audit: type=1400 audit(1715956523.953:29): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-105_</var/lib/lxc>" pid=1724 comm="apparmor_parser"
[ 116.873213] vmbr0: port 3(veth105i0) entered blocking state
[ 116.873218] vmbr0: port 3(veth105i0) entered disabled state
[ 116.873228] veth105i0: entered allmulticast mode
[ 116.873252] veth105i0: entered promiscuous mode
[ 116.898590] eth0: renamed from vethVjS28n
[ 117.352546] vmbr0: port 3(veth105i0) entered blocking state
[ 117.352551] vmbr0: port 3(veth105i0) entered forwarding state
[ 117.989318] tap101i0: entered promiscuous mode
[ 118.004565] vmbr0: port 4(tap101i0) entered blocking state
[ 118.004571] vmbr0: port 4(tap101i0) entered disabled state
[ 118.004583] tap101i0: entered allmulticast mode
[ 118.004639] vmbr0: port 4(tap101i0) entered blocking state
[ 118.004641] vmbr0: port 4(tap101i0) entered forwarding state
[ 118.295495] x86/split lock detection: #AC: CPU 2/KVM/1936 took a split_lock trap at address: 0xbfe7d050
[ 118.295530] x86/split lock detection: #AC: CPU 1/KVM/1935 took a split_lock trap at address: 0xbfe7d050
[ 118.295532] x86/split lock detection: #AC: CPU 3/KVM/1937 took a split_lock trap at address: 0xbfe7d050
[ 118.295540] x86/split lock detection: #AC: CPU 5/KVM/1939 took a split_lock trap at address: 0xbfe7d050
[ 118.295542] x86/split lock detection: #AC: CPU 4/KVM/1938 took a split_lock trap at address: 0xbfe7d050
[ 121.934427] EXT4-fs warning (device dm-11): ext4_multi_mount_protect:328: MMP interval 42 higher than expected, please wait.

[ 123.729718] usb 1-3: reset full-speed USB device number 2 using xhci_hcd
[ 124.236650] usb 1-4: reset full-speed USB device number 3 using xhci_hcd
[ 131.503609] hrtimer: interrupt took 4161 ns
[ 164.651975] EXT4-fs (dm-11): 1 orphan inode deleted
[ 164.651980] EXT4-fs (dm-11): recovery complete
[ 164.654150] EXT4-fs (dm-11): mounted filesystem 44937be1-a133-4fc5-9a87-1a89443ca029 r/w with ordered data mode. Quota mode: none.
[ 164.692783] audit: type=1400 audit(1715956572.133:30): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-106_</var/lib/lxc>" pid=2304 comm="apparmor_parser"
[ 165.047127] vmbr0: port 5(veth106i0) entered blocking state
[ 165.047131] vmbr0: port 5(veth106i0) entered disabled state
[ 165.047148] veth106i0: entered allmulticast mode
[ 165.047177] veth106i0: entered promiscuous mode
[ 165.073739] eth0: renamed from vethiWfcto
[ 165.936270] EXT4-fs warning (device dm-9): ext4_multi_mount_protect:328: MMP interval 42 higher than expected, please wait.

[ 176.069547] vmbr0: port 5(veth106i0) entered blocking state
[ 176.069552] vmbr0: port 5(veth106i0) entered forwarding state
[ 176.537257] netfs: FS-Cache loaded
[ 176.614597] Key type cifs.spnego registered
[ 176.614612] Key type cifs.idmap registered
[ 176.614931] CIFS: Attempting to mount //192.168.0.116/I
[ 208.637576] EXT4-fs (dm-9): recovery complete
[ 208.637912] EXT4-fs (dm-9): mounted filesystem b851d8ab-24a2-4ee8-99bd-87a166201d9d r/w with ordered data mode. Quota mode: none.
[ 208.671579] audit: type=1400 audit(1715956616.114:31): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-104_</var/lib/lxc>" pid=2877 comm="apparmor_parser"
[ 209.266302] vmbr0: port 6(veth104i0) entered blocking state
[ 209.266307] vmbr0: port 6(veth104i0) entered disabled state
[ 209.266319] veth104i0: entered allmulticast mode
[ 209.266348] veth104i0: entered promiscuous mode
[ 209.299087] eth0: renamed from vethfI9HPE
[ 209.706374] vmbr0: port 6(veth104i0) entered blocking state
[ 209.706380] vmbr0: port 6(veth104i0) entered forwarding state
[ 210.062797] EXT4-fs warning (device dm-16): ext4_multi_mount_protect:328: MMP interval 42 higher than expected, please wait.

[ 252.678610] EXT4-fs (dm-16): recovery complete
[ 252.678949] EXT4-fs (dm-16): mounted filesystem 4d676855-bd37-4c31-b6a5-777d6dcf31f0 r/w with ordered data mode. Quota mode: none.
[ 252.691186] CIFS: No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3.1.1), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3.1.1 (or even SMB3 or SMB2.1) specify vers=1.0 on mount.
[ 252.691190] CIFS: Attempting to mount //192.168.0.116/i
[ 252.762116] audit: type=1400 audit(1715956660.206:32): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-108_</var/lib/lxc>" pid=3389 comm="apparmor_parser"
[ 253.227185] vmbr0: port 7(veth108i0) entered blocking state
[ 253.227191] vmbr0: port 7(veth108i0) entered disabled state
[ 253.227209] veth108i0: entered allmulticast mode
[ 253.227247] veth108i0: entered promiscuous mode
[ 253.253350] eth0: renamed from vethFyGFHP
[ 253.632993] vmbr0: port 7(veth108i0) entered blocking state
[ 253.632999] vmbr0: port 7(veth108i0) entered forwarding state
[ 254.033435] EXT4-fs warning (device dm-13): ext4_multi_mount_protect:328: MMP interval 42 higher than expected, please wait.

[ 296.735646] EXT4-fs (dm-13): recovery complete
[ 296.735959] EXT4-fs (dm-13): mounted filesystem de9df10d-f4b5-46cd-baf8-a34813b1eb7c r/w with ordered data mode. Quota mode: none.
[ 296.776609] audit: type=1400 audit(1715956704.223:33): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-109_</var/lib/lxc>" pid=4060 comm="apparmor_parser"
[ 297.155012] vmbr0: port 8(veth109i0) entered blocking state
[ 297.155017] vmbr0: port 8(veth109i0) entered disabled state
[ 297.155030] veth109i0: entered allmulticast mode
[ 297.155068] veth109i0: entered promiscuous mode
[ 297.184400] eth0: renamed from vethGHvW7y
[ 297.617604] vmbr0: port 8(veth109i0) entered blocking state
[ 297.617610] vmbr0: port 8(veth109i0) entered forwarding state
[ 298.046131] EXT4-fs warning (device dm-14): ext4_multi_mount_protect:328: MMP interval 42 higher than expected, please wait.

[ 340.756528] EXT4-fs (dm-14): recovery complete
[ 340.756878] EXT4-fs (dm-14): mounted filesystem 03b2e2cc-f9cc-46ad-843d-919eb0d8dccb r/w with ordered data mode. Quota mode: none.
[ 340.789475] audit: type=1400 audit(1715956748.238:34): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-110_</var/lib/lxc>" pid=4803 comm="apparmor_parser"
[ 341.154947] vmbr0: port 9(veth110i0) entered blocking state
[ 341.154952] vmbr0: port 9(veth110i0) entered disabled state
[ 341.154977] veth110i0: entered allmulticast mode
[ 341.155036] veth110i0: entered promiscuous mode
[ 341.185212] eth0: renamed from vethqLurmA
[ 341.573596] vmbr0: port 9(veth110i0) entered blocking state
[ 341.573602] vmbr0: port 9(veth110i0) entered forwarding state
 
here is one of the lxc confs,

I have a mountpoint for the network share that is mounted to proxmox itself.

Code:
  GNU nano 7.2                                                        108.conf                                                                 
#<div align='center'><a href='https%3A//Helper-Scripts.com' target='_blank' rel='noopener noreferrer'><img src='https%3A//raw.githubuserconten>
#
#  # Sonarr LXC
#
#  <a href='https%3A//ko-fi.com/D1D7EP4GF'><img src='https%3A//img.shields.io/badge/&#x2615;-Buy me a coffee-blue' /></a>
#  </div>
arch: amd64
cores: 2
features: keyctl=1,nesting=1
hostname: sonarr
memory: 1024
mp0: /mnt/lxc_shares/coopnet/,mp=/mnt/coopnet
net0: name=eth0,bridge=vmbr0,gw=192.168.0.1,hwaddr=BC:24:11:08:DF:E0,ip=192.168.0.167/24,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-108-disk-0,size=4G
swap: 512
tags: proxmox-helper-scripts
unprivileged: 1
 
next step is to look at dmesg on the host. since I understand that local-lvm is on nvme, you can grep for it like so

dmesg | grep nvm
Code:
[    1.044794] nvme 0000:01:00.0: platform quirk: setting simple suspend
[    1.045183] nvme nvme0: pci function 0000:01:00.0
[    1.050652] nvme nvme0: missing or invalid SUBNQN field.
[    1.052939] nvme nvme0: 6/0/0 default/read/poll queues
[    1.054978]  nvme0n1: p1 p2 p3

is missing or invalid SUBNQN field. an issue?
 
Not that I'm aware of; if those are the only messages there isnt anything obvious thats causing your file system to error out :( perhaps a health check on your drive is warranted.
I just backed up everything and nuked it from orbit and installed a new nvme drive, ran memtest86 and reinstalled. Seems good so far. ‍♂️
 
  • Like
Reactions: Kingneutron

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!