[SOLVED] Cannot create VMs in PVE 6.3-2

[mcgnomer]

Today I installed PVE 6.3-2 on to an SSD attached to the SATA 0 port on my R710.
I previously had PVE 5.x installed on a virtual disk provided by the PERC 6/i raid card.

Everything installed fine and I'm able to log into the GUI to start creating VMs.
When I attempt to create a VM I am getting the following:

()
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.15.18-30-pve/modules.dep.bin'
modprobe: FATAL: Module dm-thin-pool not found in directory /lib/modules/4.15.18-30-pve
  /sbin/modprobe failed: 1
  thin: Required device-mapper target(s) not detected in your kernel.
TASK ERROR: unable to create VM 100 - lvcreate 'pve/vm-100-disk-0' error:   Run `lvcreate --help' for more information.

I went and looked in /lib/modules/ and I do not have a dir for 4.15.xxx.

root@pve:/lib/modules# ls
5.4.73-1-pve  lvs

pveversion -v

root@pve:/lib/modules# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 4.15.18-30-pve)
pve-manager: 6.3-2 (running version: 6.3-2/22f57405)
pve-kernel-5.4: 6.3-1
pve-kernel-helper: 6.3-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-6
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-1
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

uname -a

root@pve:/lib/modules# uname -a
Linux pve 4.15.18-30-pve #1 SMP PVE 4.15.18-58 (Fri, 12 Jun 2020 13:53:01 +0200) x86_64 GNU/Linux

apt update

root@pve:/lib/modules# apt update
Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://security.debian.org buster/updates InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.

apt upgrade

root@pve:/lib/modules# apt update
Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://security.debian.org buster/updates InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.

I'm not sure where to go from here... Any help would be appreciated!

 

[mcgnomer]

Added the non-enterprise repo and updated/upgraded
Still same issue:

root@pve:~# apt update
Hit:1 http://security.debian.org buster/updates InRelease
Hit:2 http://deb.debian.org/debian buster InRelease                                               
Hit:3 http://deb.debian.org/debian buster-updates InRelease                                       
Hit:4 http://download.proxmox.com/debian/pve buster InRelease                 
Reading package lists... Done                           
Building dependency tree       
Reading state information... Done
All packages are up to date.

pveversion -v

root@pve:~# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 4.15.18-30-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-8
pve-kernel-helper: 6.3-8
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-1
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-5
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-10
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

 

[rauxon]

That package list definitely includes the newer kernel - that looks like it has booted using an old 5.x kernel using the 6.x root filesystem.

As a first step, I would check the boot order (make sure it is booting off the SSD rather than the PERC).

If you don't mind reinitializing the whole system (the cleanest approach, but losing any VM images you still have from 5.x) I would:

• Through the PERC management utility, destroy the virtual disk
• In the BIOS make sure that the boot order is set to the SSD first
• Reinstall proxmox
• Reboot, and recreate the virtual disk
• Configure the PERC device within proxmox
• Reboot one more time

 

[mcgnomer]

Hmm. It seems as if it was using that virtual disk somehow. Though, not sure how.

I went into the PERC management utility and destroyed the VD.
Went into BIOS and made sure that the embedded SATA was first on the boot order.
Re-installed proxmox on sda (the 500gb ssd)
When I rebooted after install I get the message there is no bootable image found.
BIOS sees the SATA drive - not plugged into PERC.

I booted to a live usb image and was able to see the partitioning that happens during proxmox install on the SATA drive....
I'll try another reinstall... but maybe I'll wipe the ssd manually first.

 

[mcgnomer]

Erased the SSD using dd from the USB linux.
Reinstalled Proxmox on the SSD
Changed the boot method from BIOS to UEFI
Proxmox boots and I can now create VMs.

pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.73-1-pve)
pve-manager: 6.3-2 (running version: 6.3-2/22f57405)
pve-kernel-5.4: 6.3-1
pve-kernel-helper: 6.3-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-6
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-1
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

Thanks for the help!!

 

[Mortifier]

Did you lose any of your VMs? I am also having the issue upgrading from version 5 with ZFS not mounting.

 

[mcgnomer]

I backed up all my existing VMs (that I wanted to save) prior to my upgrade. My upgrade was also planned to be destructive to my 5.x implementation. It was more of a re-install... I was able to restore all my VMs after resolving my issue here.

 

[rauxon]

Did you lose any of your VMs? I am also having the issue upgrading from version 5 with ZFS not mounting.

It would probably make sense to create a new thread - this issue was related to the boot device and kernel from 5.x, rather than an upgrade from 5.x to 6.x in general.

I upgraded my home cluster from 5.x to 6.x (from memory) without any issue.