VM Fails to start

[revvr]

I have a VM running OpenMediaVault. Overnight, the VM shut down and now I cannot start it again. I have not changed anything on it. I was made aware of this because the daily backup task failed. All other VMs are working correctly.

I've tried restarting Proxmox, but it does not fix the issue.

The VM starts, loads and passes Grub, and then I can see the standard OMV terminal once the machine has booted, like it is ready for action. However, a few seconds later, it just shuts down. I'm not really sure of how to even start troubleshooting this. The VM has worked great for several months.

I did journalctl -r to see if I could find a clue, but I'm not great at this. I suspect something is consuming too much CPU and it shuts down?

May 16 09:03:28 rackmeat qmeventd[10764]: Finished cleanup for 102
May 16 09:03:28 rackmeat qmeventd[10764]: Starting cleanup for 102
May 16 09:03:28 rackmeat systemd[1]: 102.scope: Consumed 28.853s CPU time.
May 16 09:03:28 rackmeat systemd[1]: 102.scope: Deactivated successfully.
May 16 09:03:28 rackmeat pvedaemon[1475]: <root@pam> end task UPID:rackmeat:00002526:0000C6BE:66462E01:vncproxy:102:root@pam: OK
May 16 09:03:28 rackmeat qmeventd[1102]: read: Connection reset by peer
May 16 09:03:28 rackmeat kernel: vmbr0: port 3(fwpr102p0) entered disabled state
May 16 09:03:28 rackmeat kernel: fwpr102p0 (unregistering): left promiscuous mode
May 16 09:03:28 rackmeat kernel: fwpr102p0 (unregistering): left allmulticast mode
May 16 09:03:28 rackmeat kernel: fwbr102i0: port 1(fwln102i0) entered disabled state
May 16 09:03:28 rackmeat kernel: fwln102i0 (unregistering): left promiscuous mode
May 16 09:03:28 rackmeat kernel: fwln102i0 (unregistering): left allmulticast mode
May 16 09:03:28 rackmeat kernel: vmbr0: port 3(fwpr102p0) entered disabled state
May 16 09:03:28 rackmeat kernel: fwbr102i0: port 1(fwln102i0) entered disabled state
May 16 09:03:28 rackmeat kernel: fwbr102i0: port 2(tap102i0) entered disabled state
May 16 09:03:28 rackmeat kernel: tap102i0: left allmulticast mode
May 16 09:03:28 rackmeat kernel:  sdd: sdd1 sdd9
May 16 09:03:28 rackmeat kernel:  sdc: sdc1 sdc9
May 16 09:03:28 rackmeat kernel:  sdb: sdb1 sdb9
May 16 09:03:28 rackmeat kernel:  sda: sda1 sda9
May 16 09:02:09 rackmeat pvedaemon[1475]: <root@pam> starting task UPID:rackmeat:00002526:0000C6BE:66462E01:vncproxy:102:root@pam:
May 16 09:02:09 rackmeat pvedaemon[9510]: starting vnc proxy UPID:rackmeat:00002526:0000C6BE:66462E01:vncproxy:102:root@pam:
May 16 09:02:09 rackmeat pvedaemon[1476]: <root@pam> end task UPID:rackmeat:00002475:0000C631:66462DFF:qmstart:102:root@pam: OK
May 16 09:02:09 rackmeat kernel: fwbr102i0: port 2(tap102i0) entered forwarding state
May 16 09:02:09 rackmeat kernel: fwbr102i0: port 2(tap102i0) entered blocking state
May 16 09:02:09 rackmeat kernel: tap102i0: entered allmulticast mode
May 16 09:02:09 rackmeat kernel: fwbr102i0: port 2(tap102i0) entered disabled state
May 16 09:02:09 rackmeat kernel: fwbr102i0: port 2(tap102i0) entered blocking state
May 16 09:02:09 rackmeat kernel: fwbr102i0: port 1(fwln102i0) entered forwarding state
May 16 09:02:09 rackmeat kernel: fwbr102i0: port 1(fwln102i0) entered blocking state
May 16 09:02:09 rackmeat kernel: fwln102i0: entered promiscuous mode
May 16 09:02:09 rackmeat kernel: fwln102i0: entered allmulticast mode
May 16 09:02:09 rackmeat kernel: fwbr102i0: port 1(fwln102i0) entered disabled state
May 16 09:02:09 rackmeat kernel: fwbr102i0: port 1(fwln102i0) entered blocking state
May 16 09:02:09 rackmeat kernel: vmbr0: port 3(fwpr102p0) entered forwarding state
May 16 09:02:09 rackmeat kernel: vmbr0: port 3(fwpr102p0) entered blocking state
May 16 09:02:09 rackmeat kernel: fwpr102p0: entered promiscuous mode
May 16 09:02:09 rackmeat kernel: fwpr102p0: entered allmulticast mode
May 16 09:02:09 rackmeat kernel: vmbr0: port 3(fwpr102p0) entered disabled state
May 16 09:02:09 rackmeat kernel: vmbr0: port 3(fwpr102p0) entered blocking state
May 16 09:02:09 rackmeat kernel: tap102i0: entered promiscuous mode
May 16 09:02:08 rackmeat systemd[1]: Started 102.scope.
May 16 09:02:08 rackmeat kernel:  sdb: sdb1 sdb9
May 16 09:02:08 rackmeat kernel:  sda: sda1 sda9
May 16 09:02:07 rackmeat pvedaemon[1476]: <root@pam> starting task UPID:rackmeat:00002475:0000C631:66462DFF:qmstart:102:root@pam:
May 16 09:02:07 rackmeat pvedaemon[9333]: start VM 102: UPID:rackmeat:00002475:0000C631:66462DFF:qmstart:102:root@pam:

Can anyone help me figure out what is happening? This VM is important to my house. Thanks!!

Edit: I just tried restoring backups from two different days, same result.

 

[klx]

Can you revert it back from an earlier backup?

 

[revvr]

I tried that, two different backups, same result

 

[klx]

If your backups won't start, and this VM ran for a while, what hardware change if any were made? Do you have another cluster node to migrate the VM to?

 

[revvr]

If your backups won't start, and this VM ran for a while, what hardware change if any were made? Do you have another cluster node to migrate the VM to?

That's the thing. Nothing changed. All hardware is the same, other VMs are running just fine.

I don't have another cluster node to move things to.

 

[ggroke]

When you try to start the VM, are there any messages on its console? Are there enough resources for this VM? For the sake of testing, try starting only this VM, letting all others stopped.

 

[revvr]

Nothing out of the ordinary in the console and resources are plentiful. It has access to 20 cores and 24 gb of ram. It is my main VM within Proxmox. The node is not even close to resource limits.

I tried starting with everything else stopped and got the same result.

 

[ggroke]

It appears to be a problem with your VM, not Proxmox. Maybe it was already broken for some time, but you only found out it crashes when you needed to restart it. Any chance you recently upgrade some packages or something else?

I'm thinking the best solution is to look and try older backups until you find one that works, maybe before some update(s).

 

[revvr]

I've tried backups that go back 2 months, same result. I might have upgraded proxmox packages recently, but that's it.

 

[revvr]

Ok guys, major breakthrough just now.

It appears the problem is with OMV's NUT. This VM is connected to another server which also acts as a UPS server. Somehow, OMV thinks there is no power and we're on UPS battery power and decides to shut down.

I don't know if the problem is with the UPS server or with Open Media Vault's NUT service, but will look into it. I shut down the UPS server to prevent communication to OMV and it booted up perfectly! At least I was able to boot, so now I can look through stuff.