proxmox boot hangs with libkmod: error: kmod_config_parse: /etc/modprobe.d/vfio.conf

[cjonesinbgwa]

I have a proxmox server that will not boot correctly. It tries to boot normally but hangs with multiple "proxmox boot hangs with libkmod: error: kmod_config_parse: /etc/modprobe.d/vfio.conf " followed by line numbers.

I have tried booting into debug/console mode from the pve installer usb and editing the vfio.conf file and that does not work. I tried deleting the vfio.conf file and updating the initial ram disk using the command "update-initramfs -u" then rebooting. The server tries to reboot again and ends up with the same errors scrolling down the screen and never completes the boot process.

I'd like to avoid starting from scratch with a brand new install on the existing hardware, but I will lose the VMs and LXC containers running on the pve server. I'll probably run into error with the cluster members if I try to use the same hostname and IP address during the fresh install.

Any ideas how I can fix the boot process so it will completely boot successfully?

 

[Khensu]

Hey,

not sure where you ran update-initramfs -u but did you maybe accidentally do it in the live linux of the debug environment?

I think you need to chroot into the pve installation first, otherwise you would regenerate that for the live linux, which is not what you want.
I have never done this before as well, just want to give you a hint what it could be...

As you are in a cluster, do you have shared storage?
If yes, you can simply move the vm configurations from the defective node to the remaining ones.
They reside in /etc/pve/nodes/<node>/qemu-server/<VMID>.conf. Simply move the to another node.

You can simply readd the node (also with same ip and hostname, just follow the warning at the end)
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node

 

[cjonesinbgwa]

Hey,

not sure where you ran update-initramfs -u but did you maybe accidentally do it in the live linux of the debug environment?

I think you need to chroot into the pve installation first, otherwise you would regenerate that for the live linux, which is not what you want.
I have never done this before as well, just want to give you a hint what it could be...

As you are in a cluster, do you have shared storage?
If yes, you can simply move the vm configurations from the defective node to the remaining ones.
They reside in /etc/pve/nodes/<node>/qemu-server/<VMID>.conf. Simply move the to another node.

You can simply readd the node (also with same ip and hostname, just follow the warning at the end)
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node

First off. Thank you for your response.

I was able to follow your suggested instruction.

You need to chroot into the pve installation first. Lets assume you mounted the root of the pve installation at /mnt/pve.

Mount the needed directories to run update-initramfs:

for dir in dev proc sys run; do mount --bind /$dir /mnt/pve/$dir; done

Now you can do "chroot /mnt/pve".
Check if the vfio.conf is present, delete it if necessary.
Then run "update-initramfs -u -k all" and maybe "proxmox-boot-tool refresh".
Leave the chroot with exit and reboot into it.

I had mounted the PVE installation at
/mnt/proxmox
I successfully mounted the needed directories using the command
for dir in dev proc sys run; do mount --bind /$dir /mnt/proxmox/$dir; done
I then used the chroot command that you provided.
chroot /mnt/proxmox
Checked for but did not find the vfio.conf file located at
/mnt/proxmox/etc/modprobe.d/
The I ran update-initramfs -u -k all and when it finished I ran proxmox-boot-tool refresh
Exited chroot with the exit command and rebooted.

The first boot attempt hung with this on the screen.
"Booting 'Proxmox VE GNU?Linux'
"Loading Linux 6.17.4-2pve..."
"Loading initial ramdisk..."
After quite awhile I returned to find this on the screen.
"[ 121.111902] watchdog: CPU9: Watchdog detected hard LOCKUP on cpu 9"
The system remains like that until a reboot. The next reboot halt in the same manner but with a different CPU number. The Dystem passes all the Dell Pre-Boot diagnostics. A A proxmox installer USB booting into "Rescue boot" doe not fix the issue either.


Does anyone have any other ideas?

To answer your other questions...
As you are in a cluster, do you have shared storage?
Yes they are in a cluster, but the VMs are stored in local storage. However, I do have backups of the VMs on a NAS. I think I will restore the VMs to another node in the cluster, then reinstall the PVE server software, configure the storage and migrate the VMs to the new ewly installed PVE server node.
I will take a look at the other URL you provided as well.
If yes, you can simply move the vm configurations from the defective node to the remaining ones.
They reside in /etc/pve/nodes/<node>/qemu-server/<VMID>.conf. Simply move the to another node.
To clarify... I would need to access the /etc/pve/nodes directory on the problematic PVE server and copy the <VMID?.conf files to another drive location (external) and then copy them to another PVE server.



Does anyone have any other ideas?

 

[Khensu]

I had mounted the PVE installation at
/mnt/proxmox
I successfully mounted the needed directories using the command
for dir in dev proc sys run; do mount --bind /$dir /mnt/proxmox/$dir; done
I then used the chroot command that you provided.
chroot /mnt/proxmox
Checked for but did not find the vfio.conf file located at
/mnt/proxmox/etc/modprobe.d/
The I ran update-initramfs -u -k all and when it finished I ran proxmox-boot-tool refresh
Exited chroot with the exit command and rebooted.

Ahh you still saw it.. Yeah I removed how to actually perform it, as I have tried it that way and it did not work as expected, so I quickly removed it again.
You may just have a broken initramfs, which is why it does not start up.

Sorry my knowledge here is not good enough to fix it without trying it out on my side first.

You can still boot the normal installier right?

Is the storage of the VMs on a different disk than the OS?
If yes, you can just reinstall and add the storage back.