Failed to start proxmox ve cluster file system

[sheebz]

Good day everyone,

Hoping this is a simple fix. Not sure what happened.

Updated my UniFi AP’s and switches last night, so I did a reboot of my entire network this morning here and now all of a sudden proxmox won’t boot properly.

When grub boots, it gets to the screen right before the login and everything system related says failed. Proxmox ve cluster file system and about 5 other things, then goes to login screen.

I searched a few threads and seems the answer to the ones I found were all /etc/hosts related issues. I’ve double checked mine and everything looks as it should.

I’m running my pfsense on a vm, so I’ve got no internet. Luckily, I can ssh into it from one of the pc’s on the network.

Trying to upload photos of the journal log, but keep getting a “file is too large” error… can’t post screenshots or copy and paste logs since I have no internet on the pc’s. May have to get to a pc with internet to post.

Will update with pics as soon as I can.

In the meantime, if anyone has any possible ideas why this would be happening, please let me know!

**nothing was changed on server/hypervisor side. Only updates to UniFi, then network reboot**

 

[sheebz]

forgot one of my pc's has wifi on the mobo. thank god for hotspots on my phone lol

this is what comes up in the journal logs. any help is greatly appreciated!

root@pve:~# journalctl -b0 -u pve-cluster.service
-- Journal begins at Thu 2022-05-12 16:12:25 MDT, ends at Tue 2022-06-21 11:07:23 MDT. --
Jun 21 10:34:00 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jun 21 10:34:00 pve pmxcfs[2372]: [database] crit: unable to set WAL mode: disk I/O error#010
Jun 21 10:34:00 pve pmxcfs[2372]: [database] crit: unable to set WAL mode: disk I/O error#010
Jun 21 10:34:00 pve pmxcfs[2372]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jun 21 10:34:00 pve pmxcfs[2372]: [main] notice: exit proxmox configuration filesystem (-1)
Jun 21 10:34:00 pve pmxcfs[2372]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jun 21 10:34:00 pve pmxcfs[2372]: [main] notice: exit proxmox configuration filesystem (-1)
Jun 21 10:34:00 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jun 21 10:34:00 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jun 21 10:34:00 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jun 21 10:34:00 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Jun 21 10:34:00 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jun 21 10:34:00 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jun 21 10:34:00 pve pmxcfs[2459]: [database] crit: unable to set WAL mode: disk I/O error#010
Jun 21 10:34:00 pve pmxcfs[2459]: [database] crit: unable to set WAL mode: disk I/O error#010
Jun 21 10:34:00 pve pmxcfs[2459]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jun 21 10:34:00 pve pmxcfs[2459]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jun 21 10:34:00 pve pmxcfs[2459]: [main] notice: exit proxmox configuration filesystem (-1)
Jun 21 10:34:00 pve pmxcfs[2459]: [main] notice: exit proxmox configuration filesystem (-1)
Jun 21 10:34:00 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jun 21 10:34:00 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jun 21 10:34:00 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jun 21 10:34:00 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 2.
Jun 21 10:34:00 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jun 21 10:34:00 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jun 21 10:34:00 pve pmxcfs[2468]: [database] crit: unable to set WAL mode: disk I/O error#010
Jun 21 10:34:00 pve pmxcfs[2468]: [database] crit: unable to set WAL mode: disk I/O error#010
Jun 21 10:34:00 pve pmxcfs[2468]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jun 21 10:34:00 pve pmxcfs[2468]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jun 21 10:34:00 pve pmxcfs[2468]: [main] notice: exit proxmox configuration filesystem (-1)
Jun 21 10:34:00 pve pmxcfs[2468]: [main] notice: exit proxmox configuration filesystem (-1)
Jun 21 10:34:00 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jun 21 10:34:00 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jun 21 10:34:00 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jun 21 10:34:01 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 3.
Jun 21 10:34:01 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jun 21 10:34:01 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jun 21 10:34:01 pve pmxcfs[2477]: [database] crit: unable to set WAL mode: disk I/O error#010
Jun 21 10:34:01 pve pmxcfs[2477]: [database] crit: unable to set WAL mode: disk I/O error#010
Jun 21 10:34:01 pve pmxcfs[2477]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jun 21 10:34:01 pve pmxcfs[2477]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jun 21 10:34:01 pve pmxcfs[2477]: [main] notice: exit proxmox configuration filesystem (-1)
Jun 21 10:34:01 pve pmxcfs[2477]: [main] notice: exit proxmox configuration filesystem (-1)
Jun 21 10:34:01 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jun 21 10:34:01 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jun 21 10:34:01 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jun 21 10:34:01 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 4.
Jun 21 10:34:01 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jun 21 10:34:01 pve systemd[1]: Starting The Proxmox VE cluster filesystem...
Jun 21 10:34:01 pve pmxcfs[2492]: [database] crit: unable to set WAL mode: disk I/O error#010
Jun 21 10:34:01 pve pmxcfs[2492]: [database] crit: unable to set WAL mode: disk I/O error#010
Jun 21 10:34:01 pve pmxcfs[2492]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jun 21 10:34:01 pve pmxcfs[2492]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Jun 21 10:34:01 pve pmxcfs[2492]: [main] notice: exit proxmox configuration filesystem (-1)
Jun 21 10:34:01 pve pmxcfs[2492]: [main] notice: exit proxmox configuration filesystem (-1)
Jun 21 10:34:01 pve systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jun 21 10:34:01 pve systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jun 21 10:34:01 pve systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jun 21 10:34:01 pve systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Jun 21 10:34:01 pve systemd[1]: Stopped The Proxmox VE cluster filesystem.

 

[tstrand]

Maybe it's the same issue as this: https://forum.proxmox.com/threads/proxmox-ve-cluster-not-start-after-a-power-fail.51870/

 

[sheebz]

Maybe it's the same issue as this: https://forum.proxmox.com/threads/proxmox-ve-cluster-not-start-after-a-power-fail.51870/

Trying this now. Will post update when finished

 

[sheebz]

Maybe it's the same issue as this: https://forum.proxmox.com/threads/proxmox-ve-cluster-not-start-after-a-power-fail.51870/

didnt work. ended up having to reinstall proxmox