I Killed my PVE, by adding wrong device

[intercode]

Hi,

i might have killed my entire PVE setup.

I was trying to add a BT device to a VM - but i might have added the wrong device.

It stalled the system, and after reboot - i am getting a sh!t load of error messeges.
ect. Failed to write to blocx xxxxx
filesystm is in read only mode

on the host, i am promptet with the login "screen" but it just blinks and prompts me for login again.
if i type in wrong password, i get prompted that the password is woing.
but it keeps dumping, "fialed to reloacate journal etc etc...

is there anyway to revover from this mistake?
i have attached a picture
thanks

 

[simonhoffmann]

Did you add a USB device or a PCIe device to the VM? Is your boot disk USB or an internal drive?

If your boot disk is not USB and you tried to add a USB device to your VM, I cant see how this should have bricked your system. This looks more like your boot drive just broke.

 

[intercode]

Thanks for your snap reply

I have added a raw PCIe device
the boot disk is internal nvme disk

 

[simonhoffmann]

Thanks for your snap reply

I have added a raw PCIe device
the boot disk is internal nvme disk

you could try booting from a Linux (eg Ubuntu) Live USB Stick, edit the qemu config and remove the passthrough line

 

[intercode]

thanks...

i will give it a try...

do you by any chance, know where the file is located ?

 

[simonhoffmann]

thanks...

i will give it a try...

do you by any chance, know where the file is located ?

/etc/pve/nodes/<node-name>/qemu-server/<vm-id>.conf

if this node is not part of a cluster.

If it is part of a cluster, you could try the following

- power off the defective node
- on a second node, go to the above path and edit the file
- boot the defective node again.

This might lead corosync to sync the config before the VM is started and get the updated config without the passthrough line.

 

[gfngfn256]

I was trying to add a BT device to a VM

Was that VM setup to Start at boot?

If It was not, & from your output I assume you have rebooted the PVE host (forced power off?), then editing the above passthrough line in the VM config, will make no difference to your issue, since that VM is anyway not running/started when you reboot the host. The damage you are seeing is historic from when the VM was actually running.

If however, it was set to Start at boot, then when you edit that VM's config set it NOT to Start at boot, & then reboot your entire host. You may then have some luck in your PVE coming back to life.

All in all, I'd get your backups out, because unfortunately most likely you are looking at a reinstall.

Did you just add that raw PCIe in the VM's config while attempting that BT passthrough? If you did some other changes to the host's config, you may have to reverse them too.

 

[intercode]

thanks.

i have attached a picture.. i dont know where to look for the file. i guess that i have to mount som sort of disk first ?

no cluster - stand alone setup

thanks

 

[intercode]

Was that VM setup to Start at boot?

If It was not, & from your output I assume you have rebooted the PVE host (forced power off?), then editing the above passthrough line in the VM config, will make no difference to your issue, since that VM is anyway not running/started when you reboot the host. The damage you are seeing is historic from when the VM was actually running.

If however, it was set to Start at boot, then when you edit that VM's config set it NOT to Start at boot, & then reboot your entire host. You may then have some luck in your PVE coming back to life.

All in all, I'd get your backups out, because unfortunately most likely you are looking at a reinstall.

Did you just add that raw PCIe in the VM's config while attempting that BT passthrough? If you did some other changes to the host's config, you may have to reverse them too.

yes they ware set to start at boot... do you know where i edit the file for not start at boot?

 

[leesteken]

Maybe just disable VT-d/IOMMU in the motherboard BIOS. Then the VMs with passthrough will not start and you can edit your files in /etc/pve/, because those are only there when the PVE services start successfully.

 

[simonhoffmann]

thanks.

i have attached a picture.. i dont know where to look for the file. i guess that i have to mount som sort of disk first ?

no cluster - stand alone setup

thanks

your pve-root lvm is mounted under /media/ubuntu/d51...
Navigate to this folder, then etc/pve/nodes/<nodename>/qemu-server.

Edit the .conf file of the VM and remove the passthrough line.

 

[intercode]

Maybe just disable VT-d/IOMMU in the motherboard BIOS. Then the VMs with passthrough will not start and you can edit your files in /etc/pve/, because those are only there when the PVE services start successfully.

damn... that did the host booting. but i can not access the webserver. but i have access now to the filesystem.
but do you know where to look for that config file ? please se attached picture. thanks

 

[leesteken]

damn... that did the host booting. but i can not access the webserver. but i have access now to the filesystem.
but do you know where to look for that config file ? please se attached picture. thanks

What config file? Your VM configurations are in the /etc/pve/qemu-server/ directory.

 

[gfngfn256]

do you know where i edit the file for not start at boot?

In the VM's config file you will find a line like this:
onboot: 1
change that to:
onboot: 0

 

[intercode]

What config file? Your VM configurations are in the /etc/pve/qemu-server/ directory.

wow... commented the lines out, and now everythink works like a charm... thanks mate... ! u saved me a nitht of restoring !

 

[gfngfn256]

Happy you got it solved. Next time be careful what changes you make & secondly with passthrough specifically never set that VM to run at start, till you've fully tested the VM to be running successfully. That way you can (usually!) just reboot the host to start again.

Maybe tag prefix the thread-title with [SOLVED], (upper right hand corner under title).