Proxmox VE cluster not start after a power fail

Feb 22, 2019
2
0
6
39
After a power fail the server dont start properly, i can connect to the server by ssh but i don't have access to the Web UI and the virtual machines doesn't start. I could see that the /etc/pve/ directory is empty, I tried to start the pve-cluster service manually but it showme the fallowing output:

root@nodo:~# systemctl restart pve-cluster

Job for pve-cluster.service failed because the control process exited with error code.
See "systemctl status pve-cluster.service" and "journalctl -xe" for details

Then i run the suggested commands..

root@nodo:~# systemctl status pve-cluster.service
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-02-22 17:17:48 CST; 8min ago
Process: 7614 ExecStart=/usr/bin/pmxcfs (code=exited, status=255)
CPU: 9ms

Feb 22 17:17:36 nodo systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 22 17:17:42 nodo pmxcfs[7614]: [database] crit: unable to set WAL mode: disk I/O error#010
Feb 22 17:17:42 nodo pmxcfs[7614]: [database] crit: unable to set WAL mode: disk I/O error#010
Feb 22 17:17:48 nodo pmxcfs[7614]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 22 17:17:48 nodo pmxcfs[7614]: [main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
Feb 22 17:17:48 nodo pmxcfs[7614]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 22 17:17:48 nodo systemd[1]: pve-cluster.service: Control process exited, code=exited status=255
Feb 22 17:17:48 nodo systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 22 17:17:48 nodo systemd[1]: pve-cluster.service: Unit entered failed state.
Feb 22 17:17:48 nodo systemd[1]: pve-cluster.service: Failed with result 'exit-code'.


root@nodo:~# journalctl -xe
Feb 22 17:27:37 nodo pveproxy[1837]: worker 8161 finished
Feb 22 17:27:37 nodo pveproxy[1837]: starting 1 worker(s)
Feb 22 17:27:37 nodo pveproxy[1837]: worker 8164 started
Feb 22 17:27:37 nodo pveproxy[8164]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/
Feb 22 17:27:38 nodo pveproxy[8162]: worker exit
Feb 22 17:27:38 nodo pveproxy[8163]: worker exit
Feb 22 17:27:38 nodo pveproxy[1837]: worker 8162 finished
Feb 22 17:27:38 nodo pveproxy[1837]: starting 1 worker(s)
Feb 22 17:27:38 nodo pveproxy[1837]: worker 8163 finished
Feb 22 17:27:38 nodo pveproxy[1837]: worker 8165 started
Feb 22 17:27:38 nodo pveproxy[8165]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/
Feb 22 17:27:42 nodo pveproxy[8164]: worker exit

It seem a file system fail, to me in the file: config.db-wal i tried to make a backup to this file and give me an i/o error:
root@nodo:/var/lib/pve-cluster# cp config.db-wal config.db-wal.bak
cp: error reading 'config.db-wal': Input/output error

Any idea of how to solve this?, I have several vm here and many information can be lost...
Please any help would be apreciated
Thanks
 
an input/output error is most likely connected to a broken hard disk -> check the disk
thanks for reply, it was a bad sector, after spend some time saving all configuration that I could. When I connect a monitor to the server the first thing I saw was the server telling me that i/o error occurred because a bad sector with the sector number... I prepare flash drive with HirenBootCD and repair the bad sector in a couple of minutes... restart and all ok now
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!