[SOLVED] Ubuntu 20.04 - was running now won't boot-Guest Mount Issue

[Pedulla]

Had a Ubuntu 20.04 server running as a qemu VM and truck'n along quite nicely. I got the COVID, came back and it was hung and now when it reboots, the boot hangs at[ OK ] Mounted /boot/efi..

I've never had this happen with a Ubuntu server on proxmox and I'm at a loss as to where to start.
PVE 6.4-13
VM - Ubuntu 20.04

Conf File

cat /etc/pve/nodes/pve/qemu-server/100.conf agent: 1 balloon: 0 boot: order=scsi0;ide2;net0 cores: 12 cpu: kvm64 ide2: none,media=cdrom memory: 32768 name: zm.pdx.ifchurch.com net0: virtio=EA:05:CC:AD:B4:7A,bridge=vmbr0,firewall=1,tag=50 net1: virtio=6A:5C:83:DB:E6:38,bridge=vmbr0,firewall=1,tag=20 net2: virtio=82:F8:97:EC:C4:A4,bridge=vmbr0,firewall=1,tag=99 numa: 1 onboot: 1 ostype: l26 scsi0: local-zfs:vm-100-disk-0,size=32G,ssd=1 scsi1: zmstor:vm-100-disk-0,size=11878G scsihw: virtio-scsi-pci smbios1: uuid=04c5058f-55ae-4b55-8c96-527f8a86bc57 sockets: 2 startup: order=2,up=15 usb0: host=046d:0990 vga: memory=16 vmgenid: 4e16432a-5576-482a-8e7d-40630bc66df7

 

[Moayad]

Do you have a backup for the VM? - if yes, is the last backup works?

May I ask you what is the USB passed through on the Ubuntu VM?

Could you check the Syslog during the start of the Ubuntu VM journalctl -f? and please post the full output of pveversion -v

 

[Pedulla]

Thank you in advance for the help.
The USB is a web cam providing overwatch on the physical server room.
And just FYI, once the VM hangs only a STOP will shut it down. Both reset and shutdown timeout with no affect.

Also it appears in my absence that I've run out of some storage resources.

 

[Pedulla]

I tried booting into a linuxmint live cd in the VM, thinking I could mount the vm's drives from inside the VM and see if I could poke around.
That hangs as well.

Detached the harddrive called zmstor and the system boots now. It appears this zvol is nearly full 38M left on a 12T volume.
How do I mount this zvol from the host and clear out some of the data or fix the issue?

 

[Pedulla]

Still looking for a way to recover this, rather than blow up the zvol and start over with that storage.
If I comment out the mount line in fstab, the system starts fine, so it seems like its an issue mounting the full zvol in the guest.

When I manually mount inside the guest, it hangs the system and timeouts the ssh connection and pve guest terminal. Again, only way to recover is to stop the guest vm.

Is there a way to fix/repair the zvol from the host?

 

[Pedulla]

So I'm down to running sudo dumpe2fs /dev/sdb1 and sudo fsck -b 32768 /dev/sdb1 in the guest.
Both commands hang the guest and a vm hard stop is needed.

When it hangs I get Sep 15 16:15:43 pve pvedaemon[69776]: VM 100 qmp command failed - VM 100 qmp command 'guest-ping' failed - got timeout on the host syslog.

I'm up for any help here. What is being reported back from the zvol that would hang a guest?

 

[Pedulla]

Well thanks for the help everyone.
At the end of the day a hard-drive (ext4 on a zvol) attached to the guest filled and corrupted.
I was unable to repair the drive with e2fsck, so just reformatted it from the host.
The guest is back up and running.

I'll mark this solved, but I'd like to know what I could do different, than an ancillary disk (not the / partition) attached to a guest, to keep the guest from a hard hang should that disk fill.