[SOLVED] No acces webGUI / LXC containers after update

J

jfamiens

New Member
Aug 16, 2023
9
0
1
Good evening

Yesterday I launched an update of my single node and I can't access the web interface and I can't start my containers.

I can access my host via SSH but that's almost it.

I regained access to https://192.168.0.90:8006 following this thread [removing the line starting with 127.0.1.1 in /etc/hosts, and do systemctl restart pve-cluster] but I lost it after the reboot (my node was in the state "unknown" after being able to restart all my LXC containers), I am back to square one.

Any idea please ?
  • /etc/hosts and /etc/hostname seem correct
1748677031368.png
  • No entries in /etc/pve

Code: 
root@ms01:~# pct start 100

ipcc_send_rec[1] failed: Connection refused

ipcc_send_rec[2] failed: Connection refused

ipcc_send_rec[3] failed: Connection refused

Unable to load access control list: Connection refused


Code: 
root@ms01:~# journalctl -eu pve-cluster

May 30 20:34:45 ms01 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...

May 30 20:34:45 ms01 pmxcfs[1343]: [main] notice: resolved node name 'ms01' to '192.168.0.90' for default node IP address

May 30 20:34:45 ms01 pmxcfs[1343]: [main] notice: resolved node name 'ms01' to '192.168.0.90' for default node IP address

May 30 20:34:45 ms01 pmxcfs[1343]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system

May 30 20:34:45 ms01 pmxcfs[1343]: [main] notice: exit proxmox configuration filesystem (-1)

May 30 20:34:45 ms01 pmxcfs[1343]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system


Code: 
root@ms01:~# systemctl restart pve-cluster

Job for pve-cluster.service failed because the control process exited with error code.

See "systemctl status pve-cluster.service" and "journalctl -xeu pve-cluster.service" for details.


Code: 
root@ms01:~# pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.8.12-10-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.8: 6.8.12-11
proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10
ceph-fuse: 17.2.7-pve3
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
intel-microcode: 3.20241112.1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.1-1
proxmox-backup-file-restore: 3.4.1-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.11
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.1
pve-firmware: 3.15-4
pve-ha-manager: 4.0.7
pve-i18n: 3.4.4
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2

Code: 
root@ms01:~# systemctl status pveproxy.service
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-05-30 20:29:50 CEST; 12h ago
    Process: 984 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=111)
    Process: 985 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 988 (pveproxy)
      Tasks: 4 (limit: 38103)
     Memory: 156.5M
        CPU: 12min 629ms
     CGroup: /system.slice/pveproxy.service
             ├─  988 pveproxy
             ├─45325 "pveproxy worker"
             ├─45326 "pveproxy worker"
             └─45331 "pveproxy worker"

May 31 09:11:39 ms01 pveproxy[45294]: worker exit
May 31 09:11:39 ms01 pveproxy[988]: worker 45294 finished
May 31 09:11:39 ms01 pveproxy[988]: starting 1 worker(s)
May 31 09:11:39 ms01 pveproxy[988]: worker 45326 started
May 31 09:11:39 ms01 pveproxy[45326]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2088.
May 31 09:11:39 ms01 pveproxy[45295]: worker exit
May 31 09:11:39 ms01 pveproxy[988]: worker 45295 finished
May 31 09:11:39 ms01 pveproxy[988]: starting 1 worker(s)
May 31 09:11:39 ms01 pveproxy[988]: worker 45331 started
May 31 09:11:39 ms01 pveproxy[45331]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2088.
 
Last edited:
System is back online... the service pvestatd had to be restarted.

However, if i reboot, i lost once again the connection to webGUI
 
Last edited:
I can stop the pve-cluster service but i cannot restart it.

1748705910722.png
bbgeek17 said:
You have multiple errors related to your root filesystem being Read Only. Do you have space left? Perhaps the fs is not healthy and needs to be fsck'ed.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...

Code: 
root@ms01:~# lvchange -ay /dev/pve/root && fsck /dev/pve/root
fsck from util-linux 2.38.1
fsck: /etc/fstab: parse error at line 5 -- ignored
e2fsck 1.47.0 (5-Feb-2023)
/dev/mapper/pve-root: clean, 64353/6291456 files, 3553857/25165824 blocks
 

Attachments

  • 1748704843898.png
    1748704843898.png
    158.5 KB · Views: 3
Last edited:
bbgeek17 said:
have you investigated this?

what happens when you: touch /var/lib/pve-cluster/touch.file
touch /etc/touch.file
?
what does "mount" output look like?


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
yes, I check three times my fstab file : no space / I haven't modified it for a while ! then I save / mount -a but no change

Code: 
root@ms01:~# touch /var/lib/pve-cluster/touch.file
touch: cannot touch '/var/lib/pve-cluster/touch.file': Read-only file system
root@ms01:~#
 
jfamiens said:
yes, I check three times my fstab file : no space / I haven't modified it for a while ! then I save / mount -a but no change
Click to expand...
Perhaps you should post it here and someone will assist you.

jfamiens said:
touch /var/lib/pve-cluster/touch.file touch: cannot touch '/var/lib/pve-cluster/touch.file': Read-only file system
Click to expand...
That would be an issue. If you run the remaining requested commands - someone pay spot more information and provide help.

You can also plug this into the search engine : linux read only root file system
Many good discussions and a lot of advice on troubleshooting is available online. This is not a PVE problem per-se, its a general Linux system issue.

You can also examine your boot log to find the moment when your rootfs goes RO: journalctl -b0

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
@bbgeek17 thanks for your help !

using the command journalctl -b0, the first message appearing in red refers to the line 11 of my fstab that is now with # but i cannot save it, the file is in read only mode immediately after the restart.

May 31 18:01:08 ms01 systemd-fstab-generator[449]: Failed to create unit file '/run/systemd/generator/mnt-lxc_shares-dd_externe.mount', as it already exists. Duplicate entry in '/etc/fstab'?
May 31 18:01:08 ms01 kernel: usb 2-1.4: new SuperSpeed USB device number 3 using xhci_hcd
May 31 18:01:08 ms01 (sd-execut[443]: /usr/lib/systemd/system-generators/systemd-fstab-generator failed with exit status 1.


Code: 
# Mount CIFS share MEDIA_WEB on DS1522 with rwx permissions for use in LXCs
//192.168.0.95/MEDIA_WEB/ /mnt/lxc_shares/ds1522 cifs _netdev,x-systemd.automount,noatime,uid=100000,gid=110000,dir_mode=0770,file_mode=0770,user=media_homelab,pass=XXXXXX 0 0

# Mount CIFS share EBOOKS on demand with rwx permissions for use in LXCs
//192.168.0.95/EBOOKS/ /mnt/lxc_shares/EBOOKS cifs _netdev,x-systemd.automount,noatime,uid=100000,gid=110000,dir_mode=0770,file_mode=0770,user=media_homelab,pass= XXXXXX 0 0

# Mount CIFS share on DS918 with rwx permissions for use in LXCs
//192.168.0.45/PLEXMEDIA/ /mnt/lxc_shares/ds918 cifs _netdev,x-systemd.automount,noatime,uid=100000,gid=110000,dir_mode=0770,file_mode=0770,user=media_homelab,pass=XXXXXX 0 0

# Mount CIFS Disque externe on DS918 with rwx permissions for use in LXCs
#//192.168.0.45/volumeUSB1/usbshare/ /mnt/lxc_shares/dd_externe cifs _netdev,x-systemd.automount,noatime,uid=100000,gid=110000,dir_mode=0770,file_mode=0770,user=media_homelab,pass=XXXXXX 0 0
//192.168.0.45/DD_EXTERNE /mnt/lxc_shares/dd_externe cifs _netdev,x-systemd.automount,noatime,uid=100000,gid=110000,dir_mode=0770,file_mode=0770,user=media_homelab,pass=XXXXXX 0 0

#Montage dossier partage FRIGATE sur DS1522
//192.168.0.95/volume1/FRIGATE/ /mnt/frigate nfs defaults 0 0

The second message in red in the journal

May 31 18:01:09 ms01 systemd[1]: Failed to start pvefw-logger.service - Proxmox VE firewall logger.
May 31 18:01:09 ms01 systemd-tmpfiles[732]: rm_rf(/tmp): Read-only file system
May 31 18:01:09 ms01 systemd-tmpfiles[732]: Setting access ACL "u::rwx,g::r-x,g:adm:r-x,m::r-x,o::r-x" on /var/log/journal failed: Read-only file system
May 31 18:01:09 ms01 systemd-tmpfiles[732]: Cannot set file attributes for '/var/log/journal', value=0x00800000, mask=0x00800000, ignoring: Read-only file system
May 31 18:01:09 ms01 systemd-tmpfiles[732]: Setting access ACL "u::rwx,g::r-x,g:adm:r-x,m::r-x,o::r-x" on /var/log/journal/d4250146f13540ca8988ac08738b583b failed: Read-only file system
May 31 18:01:09 ms01 systemd-tmpfiles[732]: Cannot set file attributes for '/var/log/journal/d4250146f13540ca8988ac08738b583b', value=0x00800000, mask=0x00800000, ignoring: Read-only file system
May 31 18:01:09 ms01 systemd-tmpfiles[732]: Setting access ACL "u::rw-,g::r-x,g:adm:r--,m::r--,o::---" on /var/log/journal/d4250146f13540ca8988ac08738b583b/system.journal failed: Read-only file system

I made some progress, at least i know how to change the permission after a reboot

  1. command mount -o remount,rw /
  2. command systemctl restart pvefw-logger.service successfully and i was able to make the modification in my /etc/fstab file - then
  3. mount -a
  4. command systemctl restart pve-cluster and i my webGUI is now accessible.
  5. command systemctl restart pvestatd to get the status of my node and LXC
I did run the command sudo fsck -Af -M followed by dmesg but no message in red about read-only
 
Last edited:
What an idiot. Good thing I kept an old backup of my fstab file.

For some reason that escapes me now, I had deleted the first lines... leaving only those for sharing.

SOLVED
 
You must log in or register to reply here.
Top Bottom