Proxmox Crash and 'unable to activate storage'

[Benoît Fontaine]

Hi,

My Proxmox (ve 5.1) server was not working a few days ago and I hard-reboot it.
It did not work and I had to use 'rescue boot' with the boot CD to fix that.
Since then, I have a storage that does not work anymore.

Here is the log file (/var/log/daemon.log) just before the crash:

Apr  9 16:33:53 proxmox pveproxy[20245]: worker exit
Apr  9 16:34:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Apr  9 16:34:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Apr  9 16:35:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Apr  9 16:35:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Apr  9 16:36:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Apr  9 16:36:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Apr  9 16:37:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Apr  9 16:37:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Apr  9 16:38:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Apr  9 16:38:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Apr  9 16:39:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Apr  9 16:39:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Apr  9 16:40:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Apr  9 16:40:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Apr  9 16:40:09 proxmox pveproxy[18651]: worker exit
Apr  9 16:40:09 proxmox pveproxy[41371]: worker 18651 finished
Apr  9 16:40:09 proxmox pveproxy[41371]: starting 1 worker(s)
Apr  9 16:40:09 proxmox pveproxy[41371]: worker 24954 started
Apr  9 16:41:50 proxmox systemd[3494]: Time has been changed
Apr  9 16:41:50 proxmox systemd[1]: Time has been changed
Apr  9 16:41:50 proxmox systemd[1]: apt-daily.timer: Adding 2h 4min 2.175313s random time.
Apr  9 16:41:50 proxmox systemd[1]: apt-daily-upgrade.timer: Adding 9min 18.942264s random time.
Apr  9 16:41:50 proxmox systemd[1]: Starting Proxmox VE replication runner...
Apr  9 16:41:50 proxmox watchdog-mux[841]: client watchdog expired - disable watchdog updates
Apr  9 16:41:50 proxmox watchdog-mux[841]: client watchdog expired - disable watchdog updates
Apr  9 16:41:50 proxmox pve-ha-crm[1404]: loop take too long (90 seconds)
Apr  9 16:41:50 proxmox watchdog-mux[841]: exit watchdog-mux with active connections
Apr  9 16:41:51 proxmox pve-ha-lrm[1415]: loop take too

And now :

Apr 12 09:12:47 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:12:57 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:13:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Apr 12 09:13:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Apr 12 09:13:07 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:13:17 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:13:27 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:13:37 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:13:47 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:13:57 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:14:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Apr 12 09:14:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Apr 12 09:14:07 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:14:17 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:14:27 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:14:37 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'
Apr 12 09:14:47 proxmox pvestatd[1505]: unable to activate storage 'local' - directory is expected to be a mount point but is not mounted: '/var/lib/vz'

I would like to know the cause of the crash, and how to fix the storage.
Thanks

 

[wolfgang]

Hi,

this look like a partition is missing.

check fstab and check why /var/lib/vz can't be mounted.

 

[Benoît Fontaine]

This storage is working now, yet there is still the error message in the log, weird...

What does mean "client watchdog expired - disable watchdog updates" ?