Can't connect to GUI after restart

T

typeronin

Active Member
Feb 27, 2019
17
0
41
34
After a reboot, I can't connect to the web GUI anymore. It's running and I can ping the system but I always can't establish a connection.

I'm getting weird errors saying that pve cluster hasn't started and a bunch of other services like Samba have failed to start. The error I'm getting says a key file is missing and it cannot create a lock file because of a read only file system?

How do I start troubleshooting this?
 
seems your cluster-filesystem isn't running.
Is this a cluster or a single node?
* check the status of pve-cluster and corosync
`systemctl status pve-cluster`
`systemctl status corosync`

check your logs `journalctl -r`

This should get you started!
 
For systemctl status pve-cluster:

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Feb 27 01:13:14 2019
root@pve1:~# systemctl status pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-02-27 01:52:31 PST; 1h 12min ago
Process: 4238 ExecStart=/usr/bin/pmxcfs (code=exited, status=255)
CPU: 8ms
Feb 27 01:52:31 pve1 systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 27 01:52:31 pve1 pmxcfs[4238]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only
Feb 27 01:52:31 pve1 pmxcfs[4238]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 27 01:52:31 pve1 systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Feb 27 01:52:31 pve1 systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 27 01:52:31 pve1 systemd[1]: pve-cluster.service: Unit entered failed state.
Feb 27 01:52:31 pve1 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.

For systemctl status corosync:

root@pve1:~# systemctl status corosync
● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Wed 2019-02-27 01:52:31 PST; 1h 13min ago
└─ ConditionPathExists=/etc/corosync/corosync.conf was not met
Docs: man:corosync
man:corosync.conf
man:corosync_overview

There's a lot in the journal that's failing but the two most common errors look to be:

Feb 27 03:05:42 pve1 pveproxy[9273]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 16

and

Feb 27 03:05:37 pve1 spiceproxy[9272]: unable to open log file '/var/log/pveproxy/access.log' - Read-only file system

I've done very little so far. In order, I installed SnapRAID, MergerFS, activated Samba, copied a bunch of files, sync, scrub, setup Plex and just went to reboot.
 
Oh right, I set up Plex on Ubuntu 16.04 and it was working before I rebooted. I had added a bind mount to my MergerFS /storage/ folder to my Plex LXC.

Did that mess it up?
 
typeronin said:
Feb 27 01:52:31 pve1 pmxcfs[4238]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only
Click to expand...
seems your filesystem where '/var' is located is read-only?
maybe a broken disk?

check the output of mount and df
 
Mount:

Code: 
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32940424k,nr_inodes=8235106,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6592292k,mode=755)
/dev/mapper/pve-root on / type ext4 (ro,relatime,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18113)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
mergerfsPool on /storage type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,allow_other)
/dev/sdf on /mnt/data/disk1 type ext4 (rw,relatime,data=ordered)
/dev/sdd on /mnt/data/disk2 type ext4 (rw,relatime,data=ordered)
/dev/sdb on /mnt/parity/1-parity type ext4 (rw,relatime,data=ordered)
/dev/sdc on /mnt/data/disk3 type ext4 (rw,relatime,data=ordered)
/dev/sde on /mnt/data/disk4 type ext4 (rw,relatime,data=ordered)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

df:

Filesystem 1K-blocks Used Available Use% Mounted on
udev 32940424 0 32940424 0% /dev
tmpfs 6592292 42244 6550048 1% /run
/dev/mapper/pve-root 28510260 2552732 24486248 10% /
tmpfs 32961456 0 32961456 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 32961456 0 32961456 0% /sys/fs/cgroup
mergerfsPool 6832215024 905490612 5926658876 14% /storage
/dev/sdf 975585792 905128620 70440788 93% /mnt/data/disk1
/dev/sdd 1952209744 120664 1952072696 1% /mnt/data/disk2
/dev/sdb 2928829408 688780196 2240032828 24% /mnt/parity/1-parity
/dev/sdc 1952209744 120664 1952072696 1% /mnt/data/disk3
/dev/sde 1952209744 120664 1952072696 1% /mnt/data/disk4
 
Stoiko Ivanov said:
seems your filesystem where '/var' is located is read-only?
maybe a broken disk?

check the output of mount and df
Click to expand...

Proxmox is running off a brand new 120GB Kingston A400 SSD.

I run dual E5-2650v2 with 64GB RAM. I had just made a Ubuntu 16.04 CT with 8 cores and 16GB RAM with a 32GB share of local-lvm (which was entirely empty).

I'm a Proxmox noob so if I'm doing something completely stupid, I appreciate the help.
 
typeronin said:
/dev/mapper/pve-root on / type ext4 (ro,relatime,data=ordered)
Click to expand...

your root filesystem (/) is read-only!
you need to fix this - check dmesg for potential disk-errors and the journal for other problems of how/why the filesystem got mounted only in read-only mode
 
What am I looking for in dmesg and journal?

Dmesg:

Code: 
 1.963402] mpt2sas_cm0: MSI-X vectors supported: 1, no of cores: 32, max_msix_vectors: -1
[    1.963460] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 43
[    1.963462] mpt2sas_cm0: iomem(0x00000000df900000), mapped(0x        (ptrval)), size(16384)
[    1.963463] mpt2sas_cm0: ioport(0x0000000000008000), size(256)
[    1.980562] scsi host1: ahci
[    1.980676] scsi host2: ahci
[    1.980765] scsi host3: ahci
[    1.980854] scsi host4: ahci
[    1.980945] scsi host5: ahci
[    1.981037] scsi host6: ahci
[    1.981071] ata1: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02100 irq 42
[    1.981073] ata2: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02180 irq 42
[    1.981075] ata3: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02200 irq 42
[    1.981076] ata4: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02280 irq 42
[    1.981078] ata5: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02300 irq 42
[    1.981080] ata6: SATA max UDMA/133 abar m2048@0xdff02000 port 0xdff02380 irq 42
[    1.981313] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
[    2.019058] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    2.019061] mpt2sas_cm0: sending message unit reset !!
[    2.020580] mpt2sas_cm0: message unit reset: SUCCESS
[    2.059730] mpt2sas_cm0: Allocated physical memory: size(6959 kB)
[    2.059731] mpt2sas_cm0: Current Controller Queue Depth(3088),Max Controller Queue Depth(3152)
[    2.059732] mpt2sas_cm0: Scatter Gather Elements per IO(128)
[    2.104767] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.04.00), ChipRevision(0x03), BiosVersion(07.39.00.00)
[    2.104768] mpt2sas_cm0: Protocol=(
[    2.104769] Initiator
[    2.104769] ,Target
[    2.104770] ),
[    2.104770] Capabilities=(
[    2.104771] TLR
[    2.104771] ,EEDP
[    2.104771] ,Snapshot Buffer
[    2.104772] ,Diag Trace Buffer
[    2.104772] ,Task Set Full
[    2.104773] ,NCQ
[    2.104773] )
[    2.104827] scsi host0: Fusion MPT SAS Host
[    2.105085] mpt2sas_cm0: sending port enable !!
[    2.107242] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x50030480001f14c0), phys(8)
[    2.120038] usb 1-1.5: new low-speed USB device number 3 using ehci-pci
[    2.124030] mpt2sas_cm0: port enable: SUCCESS
[    2.168084] usb 2-1-port7: over-current condition
[    2.193251] ixgbe 0000:04:00.0: Multiqueue Enabled: Rx Queue count = 32, Tx Queue count = 32 XDP Queue count = 0
[    2.228761] ixgbe 0000:04:00.0: PCI Express bandwidth of 16GT/s available
[    2.228765] ixgbe 0000:04:00.0: (Speed:5.0GT/s, Width: x4, Encoding Loss:20%)
[    2.228768] ixgbe 0000:04:00.0: This is not sufficient for optimal performance of this card.
[    2.228770] ixgbe 0000:04:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[    2.228786] ixgbe 0000:04:00.0: A slot with more lanes and/or higher speed is suggested.
[    2.236102] tsc: Refined TSC clocksource calibration: 2599.999 MHz
[    2.236123] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x257a3b2ad7e, max_idle_ns: 440795282337 ns
[    2.236593] usb 1-1.5: New USB device found, idVendor=046d, idProduct=c328
[    2.236595] usb 1-1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    2.236597] usb 1-1.5: Product: USB Keyboard
[    2.236598] usb 1-1.5: Manufacturer: Logitech
[    2.240206] hidraw: raw HID events driver (C) Jiri Kosina
[    2.248453] usbcore: registered new interface driver usbhid
[    2.248453] usbhid: USB HID core driver
[    2.248718] usbcore: registered new interface driver usbkbd
[    2.249231] input: Logitech USB Keyboard as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.5/1-1.5:1.0/0003:046D:C328.0001/input/input2
[    2.252043] ixgbe 0000:04:00.0: MAC: 3, PHY: 0, PBA No: 120416-002
[    2.252045] ixgbe 0000:04:00.0: 90:2b:34:3c:42:da
[    2.294610] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    2.294630] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.294666] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.294705] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.294727] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    2.294750] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.294761] ata1.00: ATA-10: KINGSTON SA400S37120G, SBFK61K1, max UDMA/133
[    2.294763] ata1.00: 234441648 sectors, multi 1: LBA48 NCQ (depth 31/32), AA
[    2.294849] ata1.00: configured for UDMA/133
[    2.295022] scsi 1:0:0:0: Direct-Access     ATA      KINGSTON SA400S3 61K1 PQ: 0 ANSI: 5
[    2.295274] sd 1:0:0:0: Attached scsi generic sg0 type 0
[    2.295295] ata2.00: ATA-8: ST3000DM001-9YN166, CC82, max UDMA/133
[    2.295296] ata2.00: 5860533168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    2.295320] sd 1:0:0:0: [sda] 234441648 512-byte logical blocks: (120 GB/112 GiB)
[    2.295336] sd 1:0:0:0: [sda] Write Protect is off
[    2.295339] sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.295359] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.295858] ata2.00: configured for UDMA/133
[    2.295976] scsi 2:0:0:0: Direct-Access     ATA      ST3000DM001-9YN1 CC82 PQ: 0 ANSI: 5
[    2.296008] ata5.00: ATA-8: Hitachi HDS722020ALA330, JKAOA3MA, max UDMA/133
[    2.296032] ata5.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    2.296043] ata3.00: ATA-8: Hitachi HDS722020ALA330, JKAOA3MA, max UDMA/133
[    2.296045] ata3.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    2.296073] ata4.00: ATA-8: Hitachi HDS722020ALA330, JKAOA3MA, max UDMA/133
[    2.296075] ata4.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    2.296126] ata6.00: ATA-8: WDC WD1002FAEX-00Y9A0, 05.01D05, max UDMA/133
[    2.296128] ata6.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    2.296249] sd 2:0:0:0: Attached scsi generic sg1 type 0
[    2.296287] sd 2:0:0:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.73 TiB)
[    2.296290] sd 2:0:0:0: [sdb] 4096-byte physical blocks
[    2.296315] sd 2:0:0:0: [sdb] Write Protect is off
[    2.296318] sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    2.296355] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.297504] ata5.00: configured for UDMA/133
[    2.297514] ata3.00: configured for UDMA/133
[    2.297533] ata4.00: configured for UDMA/133
[    2.297600]  sda: sda1 sda2 sda3
[    2.297680] scsi 3:0:0:0: Direct-Access     ATA      Hitachi HDS72202 A3MA PQ: 0 ANSI: 5
[    2.297922] sd 1:0:0:0: [sda] Attached SCSI disk
[    2.297946] sd 3:0:0:0: Attached scsi generic sg2 type 0
[    2.297966] sd 3:0:0:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[    2.297983] sd 3:0:0:0: [sdc] Write Protect is off
[    2.297986] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[    2.298019] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.298073] scsi 4:0:0:0: Direct-Access     ATA      Hitachi HDS72202 A3MA PQ: 0 ANSI: 5
[    2.298258] sd 4:0:0:0: Attached scsi generic sg3 type 0
[    2.298294] sd 4:0:0:0: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[    2.298310] sd 4:0:0:0: [sdd] Write Protect is off
[    2.298312] sd 4:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[    2.298347] sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.298376] scsi 5:0:0:0: Direct-Access     ATA      Hitachi HDS72202 A3MA PQ: 0 ANSI: 5
[    2.298513] ata6.00: configured for UDMA/133
[    2.298542] sd 5:0:0:0: [sde] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[    2.298551] sd 5:0:0:0: [sde] Write Protect is off
[    2.298552] sd 5:0:0:0: [sde] Mode Sense: 00 3a 00 00
[    2.298565] sd 5:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.298596] sd 5:0:0:0: Attached scsi generic sg4 type 0
[    2.298760] scsi 6:0:0:0: Direct-Access     ATA      WDC WD1002FAEX-0 1D05 PQ: 0 ANSI: 5
[    2.298946] sd 6:0:0:0: [sdf] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[    2.298955] sd 6:0:0:0: [sdf] Write Protect is off
[    2.298956] sd 6:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[    2.298968] sd 6:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.298987] sd 6:0:0:0: Attached scsi generic sg5 type 0
[    2.304670] sd 4:0:0:0: [sdd] Attached SCSI disk
[    2.305547] sd 5:0:0:0: [sde] Attached SCSI disk
[    2.305817] sd 3:0:0:0: [sdc] Attached SCSI disk
[    2.308134] hid-generic 0003:046D:C328.0001: input,hidraw0: USB HID v1.10 Keyboard [Logitech USB Keyboard] on usb-0000:00:1a.0-1.5/input0
[    2.308356] input: Logitech USB Keyboard as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.5/1-1.5:1.1/0003:046D:C328.0002/input/input3
[    2.309478] random: fast init done
[    2.315041] sd 6:0:0:0: [sdf] Attached SCSI disk
[    2.316291] sd 2:0:0:0: [sdb] Attached SCSI disk
[    2.368244] hid-generic 0003:046D:C328.0002: input,hiddev0,hidraw1: USB HID v1.10 Device [Logitech USB Keyboard] on usb-0000:00:1a.0-1.5/input1
[    2.384087] usb 2-1-port8: over-current condition
[    2.397198] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection
[    2.672449] ixgbe 0000:04:00.1: Multiqueue Enabled: Rx Queue count = 32, Tx Queue count = 32 XDP Queue count = 0
[    2.708870] ixgbe 0000:04:00.1: PCI Express bandwidth of 16GT/s available
[    2.708875] ixgbe 0000:04:00.1: (Speed:5.0GT/s, Width: x4, Encoding Loss:20%)
[    2.708878] ixgbe 0000:04:00.1: This is not sufficient for optimal performance of this card.
[    2.708879] ixgbe 0000:04:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[    2.708882] ixgbe 0000:04:00.1: A slot with more lanes and/or higher speed is suggested.
[    2.733149] ixgbe 0000:04:00.1: MAC: 3, PHY: 0, PBA No: 120416-002
[    2.733151] ixgbe 0000:04:00.1: 90:2b:34:3c:42:db
[    2.882010] ixgbe 0000:04:00.1: Intel(R) 10 Gigabit Network Connection
[    2.882920] ixgbe 0000:04:00.0 enp4s0f0: renamed from eth0
[    2.896223] ixgbe 0000:04:00.1 enp4s0f1: renamed from eth1
[    3.088031] raid6: sse2x1   gen()  7924 MB/s
[    3.136022] raid6: sse2x1   xor()  6336 MB/s
[    3.184027] raid6: sse2x2   gen() 10046 MB/s
[    3.232024] raid6: sse2x2   xor()  7037 MB/s
[    3.280026] raid6: sse2x4   gen()  9755 MB/s
[    3.328024] raid6: sse2x4   xor()  5287 MB/s
[    3.328025] raid6: using algorithm sse2x2 gen() 10046 MB/s
[    3.328026] raid6: .... xor() 7037 MB/s, rmw enabled
[    3.328026] raid6: using ssse3x2 recovery algorithm
[    3.328044] clocksource: Switched to clocksource tsc
[    3.328614] xor: automatically using best checksumming function   avx
[    3.334889] Btrfs loaded, crc32c=crc32c-intel
[    3.459929] random: crng init done
[    3.459930] random: 7 urandom warning(s) missed due to ratelimiting
[    3.501983] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[    3.599707] ip_tables: (C) 2000-2006 Netfilter Core Team
[    3.606787] systemd[1]: systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[    3.624687] systemd[1]: Detected architecture x86-64.
[    3.624853] systemd[1]: Set hostname to <pve1>.
[    3.733725] systemd[1]: Listening on Journal Audit Socket.
[    3.733792] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    3.733833] systemd[1]: Listening on Journal Socket (/dev/log).
[    3.733865] systemd[1]: Listening on LVM2 poll daemon socket.
[    3.733889] systemd[1]: Listening on udev Kernel Socket.
[    3.733928] systemd[1]: Listening on Journal Socket.
[    3.733976] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    3.755981] Loading iSCSI transport class v2.0-870.
[    3.761997] iscsi: registered transport (tcp)
[    3.766035] RPC: Registered named UNIX socket transport module.
[    3.766035] RPC: Registered udp transport module.
[    3.766036] RPC: Registered tcp transport module.
[    3.766036] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.777549] iscsi: registered transport (iser)
[    3.785639] spl: loading out-of-tree module taints kernel.
[    3.787227] SPL: Loaded module v0.7.12-1
[    3.789963] znvpair: module license 'CDDL' taints kernel.
[    3.789964] Disabling lock debugging due to kernel taint
[    3.816950] systemd-journald[583]: Received request to flush runtime journal from PID 1
[    4.301099] ipmi message handler version 39.2
[    4.302298] ipmi device interface
[    4.303697] IPMI System Interface driver.
[    4.303716] ipmi_si IPI0001:00: ipmi_platform: probing via ACPI
[    4.303895] ipmi_si IPI0001:00: [io  0x0ca2] regsize 1 spacing 1 irq 0
[    4.303897] ipmi_si: Adding ACPI-specified kcs state machine
[    4.303921] ipmi_platform: probing via SPMI
[    4.303922] ipmi_si: SPMI: io 0xca2 regsize 1 spacing 1 irq 0
[    4.303923] (NULL device *): SPMI-specified kcs state machine: duplicate
[    4.303962] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
[    4.318275] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    4.321146] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
[    4.377417] input: PC Speaker as /devices/platform/pcspkr/input/input4
[    4.378007] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer
[    4.378008] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules
[    4.378009] RAPL PMU: hw unit of domain package 2^-16 Joules
[    4.378009] RAPL PMU: hw unit of domain dram 2^-16 Joules
[    4.395680] Error: Driver 'pcspkr' is already registered, aborting...
[    4.397083] [drm] Using P2A bridge for configuration
[    4.397086] [drm] AST 2300 detected
[    4.397100] [drm] Analog VGA only
[    4.397121] [drm] dram MCLK=408 Mhz type=1 bus_width=16 size=01000000
[    4.397178] [TTM] Zone  kernel: Available graphics memory: 32961456 kiB
[    4.397179] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
[    4.397179] [TTM] Initializing pool allocator
[    4.397183] [TTM] Initializing DMA pool allocator
[    4.399394] AVX version of gcm_enc/dec engaged.
[    4.399394] AES CTR mode by8 optimization enabled
[    4.488054] fbcon: astdrmfb (fb0) is primary device
[    4.560801] Console: switching to colour frame buffer device 160x64
[    4.590874] ast 0000:07:00.0: fb0: astdrmfb frame buffer device
[    4.628045] [drm] Initialized ast 0.1.0 20120228 for 0000:07:00.0 on minor 0
[    4.640058] ipmi_si IPI0001:00: Found new BMC (man_id: 0x003c0a, prod_id: 0x0000, dev_id: 0x20)
[    4.658958] ipmi_si IPI0001:00: IPMI kcs interface initialized
[    4.660576] IPMI SSIF Interface driver
[    4.930374] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[    4.930387] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[    4.930392] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[    4.930393] EDAC sbridge: Seeking for: PCI ID 8086:0e60
[    4.930397] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[    4.930401] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[    4.930404] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[    4.930405] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[    4.930408] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[    4.930412] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[    4.930413] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[    4.930416] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[    4.930420] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[    4.930421] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[    4.930424] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[    4.930428] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[    4.930429] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[    4.930432] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[    4.930436] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[    4.930437] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[    4.930440] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[    4.930443] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[    4.930444] EDAC sbridge: Seeking for: PCI ID 8086:0e68
[    4.930448] EDAC sbridge: Seeking for: PCI ID 8086:0e79
[    4.930452] EDAC sbridge: Seeking for: PCI ID 8086:0e6a
[    4.930456] EDAC sbridge: Seeking for: PCI ID 8086:0e6b
[    4.930460] EDAC sbridge: Seeking for: PCI ID 8086:0e6c
[    4.930464] EDAC sbridge: Seeking for: PCI ID 8086:0e6d
[    4.930467] EDAC sbridge: Seeking for: PCI ID 8086:0eb8
[    4.930471] EDAC sbridge: Seeking for: PCI ID 8086:0ebc
[    4.930475] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[    4.930479] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[    4.930482] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[    4.930483] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[    4.930486] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[    4.930489] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[    4.930490] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[    4.930494] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[    4.930497] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[    4.930658] EDAC MC0: Giving out device to module sb_edac controller Ivy Bridge SrcID#0_Ha#0: DEV 0000:7f:0e.0 (INTERRUPT)
[    4.930805] EDAC MC1: Giving out device to module sb_edac controller Ivy Bridge SrcID#1_Ha#0: DEV 0000:ff:0e.0 (INTERRUPT)
[    4.930806] EDAC sbridge:  Ver: 1.1.2
[    4.932892] intel_rapl: Found RAPL domain package
[    4.932893] intel_rapl: Found RAPL domain core
[    4.932896] intel_rapl: Found RAPL domain dram
[    4.933421] intel_rapl: Found RAPL domain package
[    4.933422] intel_rapl: Found RAPL domain core
[    4.933424] intel_rapl: Found RAPL domain dram
[    5.458749] ZFS: Loaded module v0.7.12-1, ZFS pool version 5000, ZFS filesystem version 5
[    5.867733] EXT4-fs (sdf): mounted filesystem with ordered data mode. Opts: (null)
[    5.935985] EXT4-fs (sdd): mounted filesystem with ordered data mode. Opts: (null)
[    6.018613] EXT4-fs (sdb): mounted filesystem with ordered data mode. Opts: (null)
[    6.020860] EXT4-fs (sdc): mounted filesystem with ordered data mode. Opts: (null)
[    6.037241] EXT4-fs (sde): mounted filesystem with ordered data mode. Opts: (null)
[    6.115402] audit: type=1400 audit(1551259329.234:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=1307 comm="apparmor_parser"
[    6.119050] audit: type=1400 audit(1551259329.238:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=1306 comm="apparmor_parser"
[    6.119069] audit: type=1400 audit(1551259329.238:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=1306 comm="apparmor_parser"
[    6.119072] audit: type=1400 audit(1551259329.238:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=1306 comm="apparmor_parser"
[    6.119074] audit: type=1400 audit(1551259329.238:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=1306 comm="apparmor_parser"
[    6.130881] softdog: initialized. soft_noboot=0 soft_margin=60 sec soft_panic=0 (nowayout=0)
[    6.138948] new mount options do not match the existing superblock, will be ignored
[    6.262127] vmbr0: port 1(enp4s0f0) entered blocking state
[    6.262129] vmbr0: port 1(enp4s0f0) entered disabled state
[    6.262229] device enp4s0f0 entered promiscuous mode
[    6.398097] pps pps0: new PPS source ptp0
[    6.398100] ixgbe 0000:04:00.0: registered PHC device on enp4s0f0
[    6.522752] IPv6: ADDRCONF(NETDEV_UP): vmbr0: link is not ready
[    6.849645] audit: type=1400 audit(1551259329.970:7): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/bin/lxc-start" pid=1651 comm="apparmor_parser"
[    6.857184] audit: type=1400 audit(1551259329.978:8): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default" pid=1656 comm="apparmor_parser"
[    6.857199] audit: type=1400 audit(1551259329.978:9): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-cgns" pid=1656 comm="apparmor_parser"
[    6.857201] audit: type=1400 audit(1551259329.978:10): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-with-mounting" pid=1656 comm="apparmor_parser"
[    6.857203] audit: type=1400 audit(1551259329.978:11): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-with-nesting" pid=1656 comm="apparmor_parser"
[    9.539448] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 1 Gbps, Flow Control: RX/TX
[    9.539908] vmbr0: port 1(enp4s0f0) entered blocking state
[    9.539910] vmbr0: port 1(enp4s0f0) entered forwarding state
[    9.539962] IPv6: ADDRCONF(NETDEV_CHANGE): vmbr0: link becomes ready
 
Journal...the errors I listed above. It won't let me post a message that long.

Can you post a generic, default fstab for me, please? I think I deleted something useful.
 
Last edited:
dmesg does look okay so far.

maybe try to remount / rw:
* `mount -o remount,rw /`

else check your '/etc/fstab' that you did not enter something which made / mount ro

also check the output of
`vgs`
`lvs`
`pvs`
 
  • Like
Reactions: TiagoLardaCrianca
I'm guessing /dev/root needs to be in fstab? Yeah, that's missing.

Code: 
  GNU nano 2.7.4                                                               File: /etc/fstab                                                                        
# SnapRAID Disks
/dev/disk/by-id/ata-WDC_WD1002FAEX-00Y9A0_SERIAL /mnt/data/disk1 ext4 defaults 0 2
/dev/disk/by-id/ata-Hitachi_HDS722020ALA330_SERIAL1 /mnt/data/disk2 ext4 defaults 0 2
/dev/disk/by-id/ata-Hitachi_HDS722020ALA330_SERIAL2 /mnt/data/disk3 ext4 defaults 0 2
/dev/disk/by-id/ata-Hitachi_HDS722020ALA330_SERIAL3 /mnt/data/disk4 ext4 defaults 0 2

# Parity Disks
/dev/disk/by-id/ata-ST3000DM001-SERIALPARITY /mnt/parity/1-parity ext4 defaults 0 2

/mnt/data/* /storage fuse.mergerfs defaults,allow_other,direct_io,use_ino,category.create=lfs,moveonenospc=true,minfreespace=20G,fsname=mergerfsPool 0 0
 
* that's probably the most central cause of problems
* it's '/dev/pve/root' not '/dev/root'

this should be the example fstab of a fresh install on lvm-thin:
Code: 
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
 
Would it be a good idea to use a ZFS mirror as a Proxmox boot drive to protect against drive failure?

Can I add an identical SSD now after the fact to mirror my boot drive? Or should I just make regular backups of my boot drive and restore it to a new drive in the event of failure?

I guess it depends on how Proxmox boots off a ZFS mirror vdev.
 
Stoiko Ivanov said:
* that's probably the most central cause of problems
* it's '/dev/pve/root' not '/dev/root'

this should be the example fstab of a fresh install on lvm-thin:
Code: 
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
Click to expand...

This fixed it. I'm just a dummy that overwrote his f-stab setting up SnapRAID.

Thanks so much for the help, Stoiko.
 
help me pleas :

root@Proxmox:~# journalctl -b -u pve-cluster
-- Logs begin at Thu 2022-09-22 14:49:56 CET, end at Thu 2022-09-22 15:23:50 CET. --
Sep 22 14:51:28 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Sep 22 14:51:28 Proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Unit entered failed state.
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 22 15:05:12 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 22 15:05:12 Proxmox pmxcfs[1087]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 15:05:12 Proxmox pmxcfs[1087]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Sep 22 15:05:12 Proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Unit entered failed state.
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 22 15:17:13 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
 
farse said:
help me pleas :

root@Proxmox:~# journalctl -b -u pve-cluster
-- Logs begin at Thu 2022-09-22 14:49:56 CET, end at Thu 2022-09-22 15:23:50 CET. --
Sep 22 14:51:28 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 14:51:28 Proxmox pmxcfs[973]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Sep 22 14:51:28 Proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Unit entered failed state.
Sep 22 14:51:28 Proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 22 15:05:12 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Sep 22 15:05:12 Proxmox pmxcfs[1087]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Sep 22 15:05:12 Proxmox pmxcfs[1087]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Sep 22 15:05:12 Proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Unit entered failed state.
Sep 22 15:05:12 Proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 22 15:17:13 Proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Click to expand...
Ran into the same issue, did you find a solution?
 
Hi there, I have the same issue as above, which happened to me after a power cut.

After following this thread, I noticed that the below file, the #/dev/pve/root / ext4 errors=remount-ro 0 1 was remarked!.

BUT i can not save the file after modifying it since i get

[ Error writing fstaby: Read-only file system ]


File: /etc/fstab

GNU nano 7.2 fstab
# <file system> <mount point> <type> <options> <dump> <pass>
#/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
871d4bb1-b971-4b4d-a9d2-5fae897de5e4 /mnt/USB ext4 defaults,noatime,nofail 0 2


Any guidance, please?

Thank you in advance
 
You must log in or register to reply here.
Top Bottom