[SOLVED] Power Cut to Machine... vm boots but will not login

[keernan.lanismore]

I'm running PVE 8.1.10. I lost power to my machine. Proxmox rebooted fine. Two vms were active at the time power was cut. Both of those vms seemed to boot fine from the proxmox gui. The console presents a login and appears to accept my userid and password, but presents the message: "ping: connect: Network is unreachable"

I confirmed the ping failure when running my own ping from another machine on my home lan.
Both vms are running the most recent Debian.

Output from: sudo cat /etc/pve/qemu-server/100.conf:

boot: order=scsi0;ide2;net0
cores: 1
cpu: x86-64-v2-AES
ide2: none,media=cdrom
memory: 4096
meta: creation-qemu=8.1.2,ctime=1702760897
name: vm1
net0: virtio=BC:24:11:91:AC:80,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-0,iothread=1,size=50G
scsihw: virtio-scsi-single
smbios1: uuid=4ee8b992-6dc9-451a-969d-67d9b1cf4c2a
sockets: 1
vmgenid: e8208d0b-ef2f-4639-a653-5fd14255cc1f

Any and all help would be most appreciated.

 

[leesteken]

It's not uncommon that filesystems get corrupted on unexpected power loss (with consumer SSDs even more often than HDDs). Maybe just restore the VMs from backup (to a new VM, don't overwrite!) to be sure you have a good working copy? And then move the disk from the old VM to the new VM (Reassign Owner), mount it and see what you can recover?

 

[gfngfn256]

In all likelihood your VM's got corrupted during power loss.

Assuming you have backups - try restoring them to a different VMID (so that you can still attempt some data recovery from original VM) you can do this from PVE in the Storage itself by selecting to Restore the backup & choosing a different VMID) - if the new VM works - you know the original is corrupted.

One thing I don't get - do you have some script in place that issues that Ping command automatically upon login? Or maybe you meant that you entered the Ping command yourself?

How does the VM normally get its IP address. Static/DHCP ?
Check your router to see if its trying/connected.

 

[keernan.lanismore]

It's not uncommon that filesystems get corrupted on unexpected power loss (with consumer SSDs even more often than HDDs). Maybe just restore the VMs from backup (to a new VM, don't overwrite!) to be sure you have a good working copy? And then move the disk from the old VM to the new VM (Reassign Owner), mount it and see what you can recover?

haha... backup? what's that? (ps: worse, the vm contains all the code I've been writing for months creating a bash backup system)

 

[gfngfn256]

the vm contains all the code I've been writing for months creating a bash backup system

Well I guess you had a constant reminder of the necessity to backup - now you know how necessary!

 

[keernan.lanismore]

In all likelihood your VM's got corrupted during power loss.

Assuming you have backups - try restoring them to a different VMID (so that you can still attempt some data recovery from original VM) you can do this from PVE in the Storage itself by selecting to Restore the backup & choosing a different VMID) - if the new VM works - you know the original is corrupted.

One thing I don't get - do you have some script in place that issues that Ping command automatically upon login? Or maybe you meant that you entered the Ping command yourself?

How does the VM normally get its IP address. Static/DHCP ?
Check your router to see if its trying/connected.

>Assuming you have backups
Of course not.

>ping
That's coming from .bashrc . It runs sshfs to mount a notes dir from another lan machine so I have access to my notes on whatever machine I am working on. It pings the machine that has the notes before running sshfs.

>I'm running a pfsense router on its own machine (not a vm). It assigns an ip as part of its dhcp server console but is set up to assign a static ip (eg outside its dhcp dynamic range). But checking just now, I see it has assigned new ips to two new mac ids (undoubtedly the vms). pfsense shows both devices online. However ping returns: "Destination Host Unreachable"

 

[keernan.lanismore]

>Assuming you have backups
Of course not.

>ping
That's coming from .bashrc . It runs sshfs to mount a notes dir from another lan machine so I have access to my notes on whatever machine I am working on. It pings the machine that has the notes before running sshfs.

>I'm running a pfsense router on its own machine (not a vm). It assigns an ip as part of its dhcp server console but is set up to assign a static ip (eg outside its dhcp dynamic range). But checking just now, I see it has assigned new ips to two new mac ids (undoubtedly the vms). pfsense shows both devices online. However ping returns: "Destination Host Unreachable"

I'm a bit dense sometimes. I have been logging into the vm via the console. I still can't ping the machine or log in remotely. When I run ip addr in the proxmox console, I do not get an ip address. So, I can see my code, but I can't send it anywhere nor does copy/paste seem to work.

 

[keernan.lanismore]

In all likelihood your VM's got corrupted during power loss.

Assuming you have backups - try restoring them to a different VMID (so that you can still attempt some data recovery from original VM) you can do this from PVE in the Storage itself by selecting to Restore the backup & choosing a different VMID) - if the new VM works - you know the original is corrupted.

One thing I don't get - do you have some script in place that issues that Ping command automatically upon login? Or maybe you meant that you entered the Ping command yourself?

How does the VM normally get its IP address. Static/DHCP ?
Check your router to see if its trying/connected.

I'm thinking of using the proxmox gui to modify hardware / reset the network, but I'm nervous because I do not have a good grasp of networking, especially virtualized networks; nor do I have a clue what resetting the network means/does. I assume it recreates the ethernet emulation (is that even a thing?).

 

[gfngfn256]

So, I can see my code, but I can't send it anywhere nor does copy/paste seem to work.

If I understand you correctly - your in a web-browser logged into PVE - you are in the console for that VM.
You should then be able to select (with mouse) the text you want to copy - right click your mouse & select copy. Other shortcuts (Windows ctrl-c etc. ) should also work.
This maybe browser/OS dependant however.
I just tried it in a Debian LXC - Chrome/Windows 11 it works

 

[gfngfn256]

I'm thinking of using the proxmox gui to modify hardware / reset the network

Not sure what your referring to - maybe the VMs network device settings? Not sure why you would be going there. Your 100.conf file shows you your (current) settings.

I see it has assigned new ips to two new mac ids (undoubtedly the vms)

If this is in fact the said VMs - you've got some major corruption going on. Do these MACs begin with BC:24:11

There is also a possibility that your main PVE is also somehow corrupted.

From the PVE Host console,

cat /etc/network/interfaces

cat /etc/hosts

Enter each line & then post the output here (in CODE tags if possible).

 

[keernan.lanismore]

If I understand you correctly - your in a web-browser logged into PVE - you are in the console for that VM.
You should then be able to select (with mouse) the text you want to copy - right click your mouse & select copy. Other shortcuts (Windows ctrl-c etc. ) should also work.
This maybe browser/OS dependant however.
I just tried it in a Debian VM - Chrome/Windows 11 it works

For some reason I am unable to highlight select with my mouse in the proxmox console window. However, I've solved the problem by adding a new network device and it automatically joined the lan. I can't ssh in yet because it isn't recognizing my keys but I'll turn ssh password back on and reset the ssh keys. Thank you for all your help. I don't think I would have figured this out without you asking about the ping (which led me to realizing I was actually logging in via the proxmox gui console).

 

[leesteken]

haha... backup? what's that? (ps: worse, the vm contains all the code I've been writing for months creating a bash backup system)

Since you don't care about the data (otherwise you would have make backups or copies), just reinstall a new VM? ;-) But seriously, create a new VM with recent Debian first and move the old virtual disk to the new VM to see what you can recover. Also make sure to have a byte-for-byte copy of the old virtual disk (in case the recovery breaks it further).

 

[gfngfn256]

I just tried it in a Debian VM - Chrome/Windows 11 it works

I incorrectly reported this - I tried it in a Debian LXC with the PVE console! (Will edit my original post)
In a VM: THIS WILL NOT WORK
It is possible using SPICE - used many times