Can't connect to GUI after restart

typeronin · Feb 27, 2019

After a reboot, I can't connect to the web GUI anymore. It's running and I can ping the system but I always can't establish a connection.

I'm getting weird errors saying that pve cluster hasn't started and a bunch of other services like Samba have failed to start. The error I'm getting says a key file is missing and it cannot create a lock file because of a read only file system?

How do I start troubleshooting this?

Stoiko Ivanov · Feb 27, 2019

seems your cluster-filesystem isn't running.
Is this a cluster or a single node?
* check the status of pve-cluster and corosync
`systemctl status pve-cluster`
`systemctl status corosync`

check your logs `journalctl -r`

This should get you started!

typeronin · Feb 27, 2019

For systemctl status pve-cluster:

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Feb 27 01:13:14 2019
root@pve1:~# systemctl status pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-02-27 01:52:31 PST; 1h 12min ago
Process: 4238 ExecStart=/usr/bin/pmxcfs (code=exited, status=255)
CPU: 8ms
Feb 27 01:52:31 pve1 systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 27 01:52:31 pve1 pmxcfs[4238]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only
Feb 27 01:52:31 pve1 pmxcfs[4238]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 27 01:52:31 pve1 systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Feb 27 01:52:31 pve1 systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 27 01:52:31 pve1 systemd[1]: pve-cluster.service: Unit entered failed state.
Feb 27 01:52:31 pve1 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.

For systemctl status corosync:

root@pve1:~# systemctl status corosync
● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Wed 2019-02-27 01:52:31 PST; 1h 13min ago
└─ ConditionPathExists=/etc/corosync/corosync.conf was not met
Docs: man:corosync
man:corosync.conf
man:corosync_overview

There's a lot in the journal that's failing but the two most common errors look to be:

Feb 27 03:05:42 pve1 pveproxy[9273]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 16

and

Feb 27 03:05:37 pve1 spiceproxy[9272]: unable to open log file '/var/log/pveproxy/access.log' - Read-only file system

I've done very little so far. In order, I installed SnapRAID, MergerFS, activated Samba, copied a bunch of files, sync, scrub, setup Plex and just went to reboot.

typeronin · Feb 27, 2019

Oh right, I set up Plex on Ubuntu 16.04 and it was working before I rebooted. I had added a bind mount to my MergerFS /storage/ folder to my Plex LXC.

Did that mess it up?

Stoiko Ivanov · Feb 27, 2019

typeronin said:
Feb 27 01:52:31 pve1 pmxcfs[4238]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only

seems your filesystem where '/var' is located is read-only?
maybe a broken disk?

check the output of mount and df

typeronin · Feb 27, 2019

Mount:

Code:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32940424k,nr_inodes=8235106,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6592292k,mode=755)
/dev/mapper/pve-root on / type ext4 (ro,relatime,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18113)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
mergerfsPool on /storage type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,allow_other)
/dev/sdf on /mnt/data/disk1 type ext4 (rw,relatime,data=ordered)
/dev/sdd on /mnt/data/disk2 type ext4 (rw,relatime,data=ordered)
/dev/sdb on /mnt/parity/1-parity type ext4 (rw,relatime,data=ordered)
/dev/sdc on /mnt/data/disk3 type ext4 (rw,relatime,data=ordered)
/dev/sde on /mnt/data/disk4 type ext4 (rw,relatime,data=ordered)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

df:

Filesystem 1K-blocks Used Available Use% Mounted on
udev 32940424 0 32940424 0% /dev
tmpfs 6592292 42244 6550048 1% /run
/dev/mapper/pve-root 28510260 2552732 24486248 10% /
tmpfs 32961456 0 32961456 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 32961456 0 32961456 0% /sys/fs/cgroup
mergerfsPool 6832215024 905490612 5926658876 14% /storage
/dev/sdf 975585792 905128620 70440788 93% /mnt/data/disk1
/dev/sdd 1952209744 120664 1952072696 1% /mnt/data/disk2
/dev/sdb 2928829408 688780196 2240032828 24% /mnt/parity/1-parity
/dev/sdc 1952209744 120664 1952072696 1% /mnt/data/disk3
/dev/sde 1952209744 120664 1952072696 1% /mnt/data/disk4

typeronin · Feb 27, 2019

Stoiko Ivanov said:
seems your filesystem where '/var' is located is read-only?
maybe a broken disk?

check the output of mount and df

Proxmox is running off a brand new 120GB Kingston A400 SSD.

I run dual E5-2650v2 with 64GB RAM. I had just made a Ubuntu 16.04 CT with 8 cores and 16GB RAM with a 32GB share of local-lvm (which was entirely empty).

I'm a Proxmox noob so if I'm doing something completely stupid, I appreciate the help.

Stoiko Ivanov · Feb 27, 2019

typeronin said:
/dev/mapper/pve-root on / type ext4 (ro,relatime,data=ordered)

your root filesystem (/) is read-only!
you need to fix this - check dmesg for potential disk-errors and the journal for other problems of how/why the filesystem got mounted only in read-only mode

typeronin · Feb 27, 2019

What am I looking for in dmesg and journal?

Dmesg:

Code:

 1.963402] mpt2sas_cm0: MSI-X vectors supported: 1, no of cores: 32, max_msix_vectors: -1
[    1.963460] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 43
[    1.963462] mpt2sas_cm0: iomem(0x00000000df900000), mapped(0x        (ptrval)), size(16384)
[    1.963463] mpt2sas_cm0: ioport(0x0000000000008000), size(256)
[    1.980562] scsi host1: ahci
[    1.980676] scsi host2: ahci
[    1.980765] scsi host3: ahci
[    1.980854] scsi host4: ahci
[    1.980945] scsi host5: ahci
[    1.981037] scsi host6: ahci
[    1.981071] ata1: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02100 irq 42
[    1.981073] ata2: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02180 irq 42
[    1.981075] ata3: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02200 irq 42
[    1.981076] ata4: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02280 irq 42
[    1.981078] ata5: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02300 irq 42
[    1.981080] ata6: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02380 irq 42
[    1.981313] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
[    2.019058] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    2.019061] mpt2sas_cm0: sending message unit reset !!
[    2.020580] mpt2sas_cm0: message unit reset: SUCCESS
[    2.059730] mpt2sas_cm0: Allocated physical memory: size(6959 kB)
[    2.059731] mpt2sas_cm0: Current Controller Queue Depth(3088),Max Controller Queue Depth(3152)
[    2.059732] mpt2sas_cm0: Scatter Gather Elements per IO(128)
[    2.104767] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.04.00), ChipRevision(0x03), BiosVersion(07.39.00.00)
[    2.104768] mpt2sas_cm0: Protocol=(
[    2.104769] Initiator
[    2.104769] ,Target
[    2.104770] ),
[    2.104770] Capabilities=(
[    2.104771] TLR
[    2.104771] ,EEDP
[    2.104771] ,Snapshot Buffer
[    2.104772] ,Diag Trace Buffer
[    2.104772] ,Task Set Full
[    2.104773] ,NCQ
[    2.104773] )
[    2.104827] scsi host0: Fusion MPT SAS Host
[    2.105085] mpt2sas_cm0: sending port enable !!
[    2.107242] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x50030480001f14c0), phys(8)
[    2.120038] usb 1-1.5: new low-speed USB device number 3 using ehci-pci
[    2.124030] mpt2sas_cm0: port enable: SUCCESS
[    2.168084] usb 2-1-port7: over-current condition
[    2.193251] ixgbe 0000:04:00.0: Multiqueue Enabled: Rx Queue count = 32, Tx Queue count = 32 XDP Queue count = 0
[    2.228761] ixgbe 0000:04:00.0: PCI Express bandwidth of 16GT/s available
[    2.228765] ixgbe 0000:04:00.0: (Speed:5.0GT/s, Width: x4, Encoding Loss:20%)
[    2.228768] ixgbe 0000:04:00.0: This is not sufficient for optimal performance of this card.
[    2.228770] ixgbe 0000:04:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[    2.228786] ixgbe 0000:04:00.0: A slot with more lanes and/or higher speed is suggested.
[    2.236102] tsc: Refined TSC clocksource calibration: 2599.999 MHz
[    2.236123] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x257a3b2ad7e, max_idle_ns: 440795282337 ns
[    2.236593] usb 1-1.5: New USB device found, idVendor=046d, idProduct=c328
[    2.236595] usb 1-1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    2.236597] usb 1-1.5: Product: USB Keyboard
[    2.236598] usb 1-1.5: Manufacturer: Logitech
[    2.240206] hidraw: raw HID events driver (C) Jiri Kosina
[    2.248453] usbcore: registered new interface driver usbhid
[    2.248453] usbhid: USB HID core driver
[    2.248718] usbcore: registered new interface driver usbkbd
[    2.249231] input: Logitech USB Keyboard as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.5/1-1.5:1.0/0003:046D:C328.0001/input/input2
[    2.252043] ixgbe 0000:04:00.0: MAC: 3, PHY: 0, PBA No: 120416-002
[    2.252045] ixgbe 0000:04:00.0: 90:2b:34:3c:42:da
[    2.294610] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    2.294630] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.294666] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.294705] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.294727] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    2.294750] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.294761] ata1.00: ATA-10: KINGSTON SA400S37120G, SBFK61K1, max UDMA/133
[    2.294763] ata1.00: 234441648 sectors, multi 1: LBA48 NCQ (depth 31/32), AA
[    2.294849] ata1.00: configured for UDMA/133
[    2.295022] scsi 1:0:0:0: Direct-Access     ATA      KINGSTON SA400S3 61K1 PQ: 0 ANSI: 5
[    2.295274] sd 1:0:0:0: Attached scsi generic sg0 type 0
[    2.295295] ata2.00: ATA-8: ST3000DM001-9YN166, CC82, max UDMA/133
[    2.295296] ata2.00: 5860533168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    2.295320] sd 1:0:0:0: [sda] 234441648 512-byte logical blocks: (120 GB/112 GiB)
[    2.295336] sd 1:0:0:0: [sda] Write Protect is off
[    2.295339] sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.295359] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.295858] ata2.00: configured for UDMA/133
[    2.295976] scsi 2:0:0:0: Direct-Access     ATA      ST3000DM001-9YN1 CC82 PQ: 0 ANSI: 5
[    2.296008] ata5.00: ATA-8: Hitachi HDS722020ALA330, JKAOA3MA, max UDMA/133
[    2.296032] ata5.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    2.296043] ata3.00: ATA-8: Hitachi HDS722020ALA330, JKAOA3MA, max UDMA/133
[    2.296045] ata3.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    2.296073] ata4.00: ATA-8: Hitachi HDS722020ALA330, JKAOA3MA, max UDMA/133
[    2.296075] ata4.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    2.296126] ata6.00: ATA-8: WDC WD1002FAEX-00Y9A0, 05.01D05, max UDMA/133
[    2.296128] ata6.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    2.296249] sd 2:0:0:0: Attached scsi generic sg1 type 0
[    2.296287] sd 2:0:0:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.73 TiB)
[    2.296290] sd 2:0:0:0: [sdb] 4096-byte physical blocks
[    2.296315] sd 2:0:0:0: [sdb] Write Protect is off
[    2.296318] sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    2.296355] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.297504] ata5.00: configured for UDMA/133
[    2.297514] ata3.00: configured for UDMA/133
[    2.297533] ata4.00: configured for UDMA/133
[    2.297600]  sda: sda1 sda2 sda3
[    2.297680] scsi 3:0:0:0: Direct-Access     ATA      Hitachi HDS72202 A3MA PQ: 0 ANSI: 5
[    2.297922] sd 1:0:0:0: [sda] Attached SCSI disk
[    2.297946] sd 3:0:0:0: Attached scsi generic sg2 type 0
[    2.297966] sd 3:0:0:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[    2.297983] sd 3:0:0:0: [sdc] Write Protect is off
[    2.297986] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[    2.298019] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.298073] scsi 4:0:0:0: Direct-Access     ATA      Hitachi HDS72202 A3MA PQ: 0 ANSI: 5
[    2.298258] sd 4:0:0:0: Attached scsi generic sg3 type 0
[    2.298294] sd 4:0:0:0: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[    2.298310] sd 4:0:0:0: [sdd] Write Protect is off
[    2.298312] sd 4:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[    2.298347] sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.298376] scsi 5:0:0:0: Direct-Access     ATA      Hitachi HDS72202 A3MA PQ: 0 ANSI: 5
[    2.298513] ata6.00: configured for UDMA/133
[    2.298542] sd 5:0:0:0: [sde] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[    2.298551] sd 5:0:0:0: [sde] Write Protect is off
[    2.298552] sd 5:0:0:0: [sde] Mode Sense: 00 3a 00 00
[    2.298565] sd 5:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.298596] sd 5:0:0:0: Attached scsi generic sg4 type 0
[    2.298760] scsi 6:0:0:0: Direct-Access     ATA      WDC WD1002FAEX-0 1D05 PQ: 0 ANSI: 5
[    2.298946] sd 6:0:0:0: [sdf] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[    2.298955] sd 6:0:0:0: [sdf] Write Protect is off
[    2.298956] sd 6:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[    2.298968] sd 6:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.298987] sd 6:0:0:0: Attached scsi generic sg5 type 0
[    2.304670] sd 4:0:0:0: [sdd] Attached SCSI disk
[    2.305547] sd 5:0:0:0: [sde] Attached SCSI disk
[    2.305817] sd 3:0:0:0: [sdc] Attached SCSI disk
[    2.308134] hid-generic 0003:046D:C328.0001: input,hidraw0: USB HID v1.10 Keyboard [Logitech USB Keyboard] on usb-0000:00:1a.0-1.5/input0
[    2.308356] input: Logitech USB Keyboard as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.5/1-1.5:1.1/0003:046D:C328.0002/input/input3
[    2.309478] random: fast init done
[    2.315041] sd 6:0:0:0: [sdf] Attached SCSI disk
[    2.316291] sd 2:0:0:0: [sdb] Attached SCSI disk
[    2.368244] hid-generic 0003:046D:C328.0002: input,hiddev0,hidraw1: USB HID v1.10 Device [Logitech USB Keyboard] on usb-0000:00:1a.0-1.5/input1
[    2.384087] usb 2-1-port8: over-current condition
[    2.397198] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection
[    2.672449] ixgbe 0000:04:00.1: Multiqueue Enabled: Rx Queue count = 32, Tx Queue count = 32 XDP Queue count = 0
[    2.708870] ixgbe 0000:04:00.1: PCI Express bandwidth of 16GT/s available
[    2.708875] ixgbe 0000:04:00.1: (Speed:5.0GT/s, Width: x4, Encoding Loss:20%)
[    2.708878] ixgbe 0000:04:00.1: This is not sufficient for optimal performance of this card.
[    2.708879] ixgbe 0000:04:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[    2.708882] ixgbe 0000:04:00.1: A slot with more lanes and/or higher speed is suggested.
[    2.733149] ixgbe 0000:04:00.1: MAC: 3, PHY: 0, PBA No: 120416-002
[    2.733151] ixgbe 0000:04:00.1: 90:2b:34:3c:42:db
[    2.882010] ixgbe 0000:04:00.1: Intel(R) 10 Gigabit Network Connection
[    2.882920] ixgbe 0000:04:00.0 enp4s0f0: renamed from eth0
[    2.896223] ixgbe 0000:04:00.1 enp4s0f1: renamed from eth1
[    3.088031] raid6: sse2x1   gen()  7924 MB/s
[    3.136022] raid6: sse2x1   xor()  6336 MB/s
[    3.184027] raid6: sse2x2   gen() 10046 MB/s
[    3.232024] raid6: sse2x2   xor()  7037 MB/s
[    3.280026] raid6: sse2x4   gen()  9755 MB/s
[    3.328024] raid6: sse2x4   xor()  5287 MB/s
[    3.328025] raid6: using algorithm sse2x2 gen() 10046 MB/s
[    3.328026] raid6: .... xor() 7037 MB/s, rmw enabled
[    3.328026] raid6: using ssse3x2 recovery algorithm
[    3.328044] clocksource: Switched to clocksource tsc
[    3.328614] xor: automatically using best checksumming function   avx
[    3.334889] Btrfs loaded, crc32c=crc32c-intel
[    3.459929] random: crng init done
[    3.459930] random: 7 urandom warning(s) missed due to ratelimiting
[    3.501983] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[    3.599707] ip_tables: (C) 2000-2006 Netfilter Core Team
[    3.606787] systemd[1]: systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[    3.624687] systemd[1]: Detected architecture x86-64.
[    3.624853] systemd[1]: Set hostname to <pve1>.
[    3.733725] systemd[1]: Listening on Journal Audit Socket.
[    3.733792] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    3.733833] systemd[1]: Listening on Journal Socket (/dev/log).
[    3.733865] systemd[1]: Listening on LVM2 poll daemon socket.
[    3.733889] systemd[1]: Listening on udev Kernel Socket.
[    3.733928] systemd[1]: Listening on Journal Socket.
[    3.733976] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    3.755981] Loading iSCSI transport class v2.0-870.
[    3.761997] iscsi: registered transport (tcp)
[    3.766035] RPC: Registered named UNIX socket transport module.
[    3.766035] RPC: Registered udp transport module.
[    3.766036] RPC: Registered tcp transport module.
[    3.766036] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.777549] iscsi: registered transport (iser)
[    3.785639] spl: loading out-of-tree module taints kernel.
[    3.787227] SPL: Loaded module v0.7.12-1
[    3.789963] znvpair: module license 'CDDL' taints kernel.
[    3.789964] Disabling lock debugging due to kernel taint
[    3.816950] systemd-journald[583]: Received request to flush runtime journal from PID 1
[    4.301099] ipmi message handler version 39.2
[    4.302298] ipmi device interface
[    4.303697] IPMI System Interface driver.
[    4.303716] ipmi_si IPI0001:00: ipmi_platform: probing via ACPI
[    4.303895] ipmi_si IPI0001:00: [io  0x0ca2] regsize 1 spacing 1 irq 0
[    4.303897] ipmi_si: Adding ACPI-specified kcs state machine
[    4.303921] ipmi_platform: probing via SPMI
[    4.303922] ipmi_si: SPMI: io 0xca2 regsize 1 spacing 1 irq 0
[    4.303923] (NULL device *): SPMI-specified kcs state machine: duplicate
[    4.303962] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
[    4.318275] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    4.321146] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
[    4.377417] input: PC Speaker as /devices/platform/pcspkr/input/input4
[    4.378007] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer
[    4.378008] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules
[    4.378009] RAPL PMU: hw unit of domain package 2^-16 Joules
[    4.378009] RAPL PMU: hw unit of domain dram 2^-16 Joules
[    4.395680] Error: Driver 'pcspkr' is already registered, aborting...
[    4.397083] [drm] Using P2A bridge for configuration
[    4.397086] [drm] AST 2300 detected
[    4.397100] [drm] Analog VGA only
[    4.397121] [drm] dram MCLK=408 Mhz type=1 bus_width=16 size=01000000
[    4.397178] [TTM] Zone  kernel: Available graphics memory: 32961456 kiB
[    4.397179] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
[    4.397179] [TTM] Initializing pool allocator
[    4.397183] [TTM] Initializing DMA pool allocator
[    4.399394] AVX version of gcm_enc/dec engaged.
[    4.399394] AES CTR mode by8 optimization enabled
[    4.488054] fbcon: astdrmfb (fb0) is primary device
[    4.560801] Console: switching to colour frame buffer device 160x64
[    4.590874] ast 0000:07:00.0: fb0: astdrmfb frame buffer device
[    4.628045] [drm] Initialized ast 0.1.0 20120228 for 0000:07:00.0 on minor 0
[    4.640058] ipmi_si IPI0001:00: Found new BMC (man_id: 0x003c0a, prod_id: 0x0000, dev_id: 0x20)
[    4.658958] ipmi_si IPI0001:00: IPMI kcs interface initialized
[    4.660576] IPMI SSIF Interface driver
[    4.930374] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[    4.930387] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[    4.930392] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[    4.930393] EDAC sbridge: Seeking for: PCI ID 8086:0e60
[    4.930397] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[    4.930401] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[    4.930404] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[    4.930405] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[    4.930408] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[    4.930412] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[    4.930413] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[    4.930416] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[    4.930420] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[    4.930421] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[    4.930424] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[    4.930428] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[    4.930429] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[    4.930432] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[    4.930436] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[    4.930437] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[    4.930440] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[    4.930443] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[    4.930444] EDAC sbridge: Seeking for: PCI ID 8086:0e68
[    4.930448] EDAC sbridge: Seeking for: PCI ID 8086:0e79
[    4.930452] EDAC sbridge: Seeking for: PCI ID 8086:0e6a
[    4.930456] EDAC sbridge: Seeking for: PCI ID 8086:0e6b
[    4.930460] EDAC sbridge: Seeking for: PCI ID 8086:0e6c
[    4.930464] EDAC sbridge: Seeking for: PCI ID 8086:0e6d
[    4.930467] EDAC sbridge: Seeking for: PCI ID 8086:0eb8
[    4.930471] EDAC sbridge: Seeking for: PCI ID 8086:0ebc
[    4.930475] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[    4.930479] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[    4.930482] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[    4.930483] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[    4.930486] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[    4.930489] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[    4.930490] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[    4.930494] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[    4.930497] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[    4.930658] EDAC MC0: Giving out device to module sb_edac controller Ivy Bridge SrcID#0_Ha#0: DEV 0000:7f:0e.0 (INTERRUPT)
[    4.930805] EDAC MC1: Giving out device to module sb_edac controller Ivy Bridge SrcID#1_Ha#0: DEV 0000:ff:0e.0 (INTERRUPT)
[    4.930806] EDAC sbridge:  Ver: 1.1.2
[    4.932892] intel_rapl: Found RAPL domain package
[    4.932893] intel_rapl: Found RAPL domain core
[    4.932896] intel_rapl: Found RAPL domain dram
[    4.933421] intel_rapl: Found RAPL domain package
[    4.933422] intel_rapl: Found RAPL domain core
[    4.933424] intel_rapl: Found RAPL domain dram
[    5.458749] ZFS: Loaded module v0.7.12-1, ZFS pool version 5000, ZFS filesystem version 5
[    5.867733] EXT4-fs (sdf): mounted filesystem with ordered data mode. Opts: (null)
[    5.935985] EXT4-fs (sdd): mounted filesystem with ordered data mode. Opts: (null)
[    6.018613] EXT4-fs (sdb): mounted filesystem with ordered data mode. Opts: (null)
[    6.020860] EXT4-fs (sdc): mounted filesystem with ordered data mode. Opts: (null)
[    6.037241] EXT4-fs (sde): mounted filesystem with ordered data mode. Opts: (null)
[    6.115402] audit: type=1400 audit(1551259329.234:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=1307 comm="apparmor_parser"
[    6.119050] audit: type=1400 audit(1551259329.238:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=1306 comm="apparmor_parser"
[    6.119069] audit: type=1400 audit(1551259329.238:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=1306 comm="apparmor_parser"
[    6.119072] audit: type=1400 audit(1551259329.238:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=1306 comm="apparmor_parser"
[    6.119074] audit: type=1400 audit(1551259329.238:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=1306 comm="apparmor_parser"
[    6.130881] softdog: initialized. soft_noboot=0 soft_margin=60 sec soft_panic=0 (nowayout=0)
[    6.138948] new mount options do not match the existing superblock, will be ignored
[    6.262127] vmbr0: port 1(enp4s0f0) entered blocking state
[    6.262129] vmbr0: port 1(enp4s0f0) entered disabled state
[    6.262229] device enp4s0f0 entered promiscuous mode
[    6.398097] pps pps0: new PPS source ptp0
[    6.398100] ixgbe 0000:04:00.0: registered PHC device on enp4s0f0
[    6.522752] IPv6: ADDRCONF(NETDEV_UP): vmbr0: link is not ready
[    6.849645] audit: type=1400 audit(1551259329.970:7): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/bin/lxc-start" pid=1651 comm="apparmor_parser"
[    6.857184] audit: type=1400 audit(1551259329.978:8): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default" pid=1656 comm="apparmor_parser"
[    6.857199] audit: type=1400 audit(1551259329.978:9): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-cgns" pid=1656 comm="apparmor_parser"
[    6.857201] audit: type=1400 audit(1551259329.978:10): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-with-mounting" pid=1656 comm="apparmor_parser"
[    6.857203] audit: type=1400 audit(1551259329.978:11): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-with-nesting" pid=1656 comm="apparmor_parser"
[    9.539448] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 1 Gbps, Flow Control: RX/TX
[    9.539908] vmbr0: port 1(enp4s0f0) entered blocking state
[    9.539910] vmbr0: port 1(enp4s0f0) entered forwarding state
[    9.539962] IPv6: ADDRCONF(NETDEV_CHANGE): vmbr0: link becomes ready

typeronin · Feb 27, 2019

Journal...the errors I listed above. It won't let me post a message that long.

Can you post a generic, default fstab for me, please? I think I deleted something useful.

Stoiko Ivanov · Feb 27, 2019

dmesg does look okay so far.

maybe try to remount / rw:
* `mount -o remount,rw /`

else check your '/etc/fstab' that you did not enter something which made / mount ro

also check the output of
`vgs`
`lvs`
`pvs`

typeronin · Feb 27, 2019

I'm guessing /dev/root needs to be in fstab? Yeah, that's missing.

Code:

  GNU nano 2.7.4                                                               File: /etc/fstab                                                                        
# SnapRAID Disks
/dev/disk/by-id/ata-WDC_WD1002FAEX-00Y9A0_SERIAL /mnt/data/disk1 ext4 defaults 0 2
/dev/disk/by-id/ata-Hitachi_HDS722020ALA330_SERIAL1 /mnt/data/disk2 ext4 defaults 0 2
/dev/disk/by-id/ata-Hitachi_HDS722020ALA330_SERIAL2 /mnt/data/disk3 ext4 defaults 0 2
/dev/disk/by-id/ata-Hitachi_HDS722020ALA330_SERIAL3 /mnt/data/disk4 ext4 defaults 0 2

# Parity Disks
/dev/disk/by-id/ata-ST3000DM001-SERIALPARITY /mnt/parity/1-parity ext4 defaults 0 2

/mnt/data/* /storage fuse.mergerfs defaults,allow_other,direct_io,use_ino,category.create=lfs,moveonenospc=true,minfreespace=20G,fsname=mergerfsPool 0 0

Stoiko Ivanov · Feb 27, 2019

* that's probably the most central cause of problems
* it's '/dev/pve/root' not '/dev/root'

this should be the example fstab of a fresh install on lvm-thin:

Code:

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

typeronin · Feb 27, 2019

...well, I know what I'm doing after work...

Thanks, Stoiko!

typeronin · Feb 27, 2019

Would it be a good idea to use a ZFS mirror as a Proxmox boot drive to protect against drive failure?

Can I add an identical SSD now after the fact to mirror my boot drive? Or should I just make regular backups of my boot drive and restore it to a new drive in the event of failure?

I guess it depends on how Proxmox boots off a ZFS mirror vdev.

typeronin · Feb 28, 2019

Stoiko Ivanov said:
* that's probably the most central cause of problems
* it's '/dev/pve/root' not '/dev/root'

this should be the example fstab of a fresh install on lvm-thin:

Code:

# <file system> <mount point> <type> <options> <dump> <pass> /dev/pve/root / ext4 errors=remount-ro 0 1 /dev/pve/swap none swap sw 0 0 proc /proc proc defaults 0 0

This fixed it. I'm just a dummy that overwrote his f-stab setting up SnapRAID.

Thanks so much for the help, Stoiko.

farse · Sep 22, 2022

help me pleas :

root@Proxmox:~# journalctl -b -u pve-cluster
-- Logs begin at Thu 2022-09-22 14:49:56 CET, end at Thu 2022-09-22 15:23:50 CET. --
Sep 22 14:51:28 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Sep 22 14:51:28 Proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Unit entered failed state.
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 22 15:05:12 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 22 15:05:12 Proxmox pmxcfs[1087]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 15:05:12 Proxmox pmxcfs[1087]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Sep 22 15:05:12 Proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Unit entered failed state.
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 22 15:17:13 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...

mn_box · Nov 6, 2022

farse said:
help me pleas :

root@Proxmox:~# journalctl -b -u pve-cluster
-- Logs begin at Thu 2022-09-22 14:49:56 CET, end at Thu 2022-09-22 15:23:50 CET. --
Sep 22 14:51:28 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Sep 22 14:51:28 Proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Unit entered failed state.
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 22 15:05:12 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 22 15:05:12 Proxmox pmxcfs[1087]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 15:05:12 Proxmox pmxcfs[1087]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Sep 22 15:05:12 Proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Unit entered failed state.
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 22 15:17:13 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...

Ran into the same issue, did you find a solution?

89east · Oct 6, 2023

Hi there, I have the same issue as above, which happened to me after a power cut.

After following this thread, I noticed that the below file, the #/dev/pve/root / ext4 errors=remount-ro 0 1 was remarked!.

BUT i can not save the file after modifying it since i get

[ Error writing fstaby: Read-only file system ]

File: /etc/fstab

GNU nano 7.2 fstab
# <file system> <mount point> <type> <options> <dump> <pass>
#/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
871d4bb1-b971-4b4d-a9d2-5fae897de5e4 /mnt/USB ext4 defaults,noatime,nofail 0 2

Any guidance, please?

Thank you in advance

BobhWasatch · Oct 6, 2023

Code:

mount -o remount,rw /dev/pve/root /

Then you can edit the /etc/fstab file.

Can't connect to GUI after restart

New Member

Proxmox Staff Member

New Member

New Member

Proxmox Staff Member

New Member

New Member

Proxmox Staff Member

New Member

New Member

Proxmox Staff Member

New Member

Proxmox Staff Member

New Member

New Member

New Member

New Member

New Member

New Member

Renowned Member