HELP WITH PFSENSE VM BRICKING AT STARTUP AFTER UPDATING PROXMOX

[sheebz]

So I just updated my proxmox to 7.4 a couple days ago. Restarted my server afterwards and everything was fine. Now, today the power company was doing something with the lines and cut the power off for a few hours and when it kicked back on, I had no internet. Restarted the server, modem, switch and still no change. Ended up having to kick my modem out of bridge mode and change ip addresses on the pc to get access into the proxmox gui and my pfsense vm is stuck on this screen.

i cant shutdown or reset it from here, gives me a "can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout (500)" error. Tried "qm unlock 100", but that timed out as well. Reboot the server, let it boot up and it does the same thing again. Not sure what happened the third time, but it went to a "loading boot options" screen then said boot from ipv4, boot from ipv6, then boot64, then it fails and kicked me into the UEFI shell and says "Failed to start Boot0002 "UEFI QEMU DVD-ROM" Time out".
.

Tried resetting a few more times and couldnt get it to stop booting into that, so i typed reset and it rebooted the vm. When it got back to that boot options loading screen, I hit escape and it brought me into the boot menu. i disabled secure boot and it started booting normal again, but then froze at the same place as before.

Now that I've gotten back to this screen and read everything, I can see it says "cant find '/etc/hostid'. It's 3:30am and im tired after spending the last 4 hours trying to fix this.

any ideas what might have happened and if there is a way to fix it without having to kill the VM and reinstall?

Also posting on netgate forum since I'm not sure if it's a pfsense issue or proxmox....

 

[dcsapak]

sound like something on the disk corrupted maybe?

can you post the vm config (qm config ID) and the host journal since last boot?

 

[ITT]

https://forum.netgate.com/topic/141...zfs-not-generated-from-uuid/4?_=1669295521699

cant see it, but maybe you had accidential activated "pre enrolled keys" on VM creation?

 

[Spoonman2002]

Not trying to be clever (or sarcastic), but did you not take snapshot(s) or backups (PBS)
of this important vm (internet access) ?

 

[sheebz]

Not trying to be clever (or sarcastic), but did you not take snapshot(s) or backups (PBS)
of this important vm (internet access) ?

Unfortunately I did not. I had that set up previously, but set it up wrong and my drives filled up with backups so I turned it off and it totally slipped my mind to set it back up

 

[sheebz]

https://forum.netgate.com/topic/141...zfs-not-generated-from-uuid/4?_=1669295521699

cant see it, but maybe you had accidential activated "pre enrolled keys" on VM creation?

I will look at this when I get home in a bit. Doesn’t sound familiar, but possible

 

[sheebz]

sound like something on the disk corrupted maybe?

can you post the vm config (qm config ID) and the host journal since last boot?

Yes, I will post when I get home.

Do you mean the vm disk corrupted, or the physical disk? Because everything else is working. My other vm’s are running fine. It’s specifically the pfsense vm that’s broken

 

[sheebz]

https://forum.netgate.com/topic/141...zfs-not-generated-from-uuid/4?_=1669295521699

cant see it, but maybe you had accidential activated "pre enrolled keys" on VM creation?

this looks like i would need to be able to ssh into the VM to do this, unless misunderstanding? I can't ssh into it or get into the web gui at any point where i am at.

looking at the output from "qm config 100", it looked like there is a prerolled key on the efi disk... cant find a way to change that from the gui.

efidisk0: local-lvm:vm-100-disk-2,efitype=4m,pre-enrolled-keys=1,size=4M

deleted efi disk with pre enrolled key, booted vm, still no change

 

[sheebz]

one of the members on the pfsense forum mentioned to try this

https://docs.netgate.com/pfsense/en...filesystem-check.html#manual-filesystem-check

i tried booting into "single user mode" to run file system check, but the VM freezes in the exact same spot as the picture from my original post

 

[sheebz]

sound like something on the disk corrupted maybe?

can you post the vm config (qm config ID) and the host journal since last boot?

this is what comes up for the VM config

root@pve:~# qm config 100
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
efidisk0: local-lvm:vm-100-disk-2,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: local:iso/pfSense-CE-2.6.0-RELEASE-amd64.iso,media=cdrom,size=749476K
machine: q35
memory: 8000
meta: creation-qemu=6.2.0,ctime=1655947909
name: PFSense
net0: virtio=5E:1E:19:0A:73:8A,bridge=vmbr0,firewall=1
net1: virtio=8A:23:1A:8C:E0:BE,bridge=vmbr1,firewall=1
numa: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,cache=writethrough,discard=on,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=dc43a313-cdcf-4695-997a-c42253442a18
sockets: 2
startup: order=1
tpmstate0: local-lvm:vm-100-disk-1,size=4M,version=v2.0
vga: memory=32
vmgenid: a8fc113d-ef9a-4db2-8d42-5f108699730c

 

[fiona]

Hi,
just guessing, but does it work if you use CPU type kvm64 instead of host? Does downgrading apt install pve-edk2-firmware=3.20220526-1 make a difference?

 

[mcfly9]

Hi,
just guessing, but does it work if you use CPU type kvm64 instead of host? Does downgrading apt install pve-edk2-firmware=3.20220526-1 make a difference?

I am seeing the same issue after update to 7.4 (coming from 7.3). Setting the CPU to kvm64 allowed pfsense to boot again.

 

[mcfly9]

Oddly, after I have shut down pfsense again and flicked it back to host cpu, now it boots that way again.

 

[Spoonman2002]

pfSense is BSD based, not Linux.
I see a lot of people set the OS type to Linux. I set it to “other”.
This is not the solution to your boot problem, but I just wanted it to mention

 

[mcfly9]

Oddly, after I have shut down pfsense again and flicked it back to host cpu, now it boots that way again.

Scratch that, I now had to travel to a remote site to get pfsense booted properly again (cpu = host). Now switched back to kvm64 cpu. Very frustrating.