System broke after power outtage

KingTChoka

Member
Sep 4, 2021
39
2
13
28
Situation
My proxmox (v. 7.1-4) was running last night and the power cutoff while it was on and VM's were running.

Problems
  1. Upon restarting my machine, only the VM's without GPU passthrough turned on, while the VM's with GPU passthrough does not turn on. The error code related to this says: TASK ERROR: Cannot bind 0000:01:00.0 to vfio which is weird since it was working before the power cutoff.
  2. I cannot even restart the machine from the command line. Running # shutdown now or poweroff -f in the terminal nothing happens (command either returns right away or just hangs).
Question
1. How do I fix this?
I have no idea what to do.
Under normal circumstances it looks like a GPU passthrough issue, but if the poweroff and shutdown commands aren't even working, I'm guessing there's something else that's broken.


2. So it looks like my nvme drive has broken. Some of my VM's are attached to a external Hard Drive, so will re-installing proxmox on my nvme hard drive and then re-attaching those HDD bring back my VM's to use?

Updates
1. Running the following commands gave the following error that might be hints:
Code:
root@host:~# shutdown -P
Failed to set wall message, ignoring: Transport endpoint is not connected
Failed to call ScheduleShutdown in logind, no action will be taken: Transport endpoint is not connected

2. Ran # reboot, # reboot now, nothing happened so I forced machines shutoff again (held power button for 5 seconds), now on machine start up, no connection can be made - no GUI, no SSH...

3. So I plugged in an HDMI to the machine on boot up and got this error message: Failed fsck check in the initramfs stage
1667703153890.jpeg

4. Looks like the Hard Drive is broken?
1667703819563.jpeg

5. OK, so I think my nvme drive is just dead (hardware failure). Running smartctl -a /dev/nvme0n1 returns the following:
1667714168577.jpeg
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!