Custom kernel doesn't boot on proxmox

[pderen]

I have built a 6.10 kernel using clang (with .config from generic ubuntu) on Ubuntu 24.04. I am able to run it on Ubuntu but not on Proxmox.
On Proxmox, I get following error:

Alert! /dev/mapper/pve-root does not exist. Dropping to shell

Running lvm pvscan returns nothin, and adding rootdelay=10 to grub also didn't help.
I was trying to re-generate the initrd image for this kernel on Proxmox but the same problem still exists.
I was also trying to build custom kernel directly on Proxmox using gcc and .confing file from proxmox 6.8 kernel with the same error.
I was trying to enable CONFIG_SCSI_SYM53C8XX_2 and rebuild kernel. The same problem.

It looks like my new kernel is not able to see my lvm partition on Proxmox.

My setup: Xeon x3440, 24GB RAM, Intel S3420GP main board, ssd 512GB
I am using No subscription repo with Proxmox version 8.2.4

Any ideas what to do?

 

[esi_y]

Does cat /proc/misc show device-mapper?

 

[pderen]

yes, it shows device-mapper

 

[esi_y]

yes, it shows device-mapper

Hey! Sorry, I have not followed up here at the time as the low hanging fruit was out of question. No better ideas, I think you would need to provide more details about how you build your kernel. At least for me, nothing to catch on.

BTW Feel free to bump up after several days directly in the thread.

 

[esi_y]

Also see here: https://forum.proxmox.com/threads/proxmox-ve-8-0-mainline-kernel-builds.135375/

 

[pderen]

My building procedure is as below:

git clone --branch linux-6.8.y https://gitlab.com/linux-kernel/stable.git
# repo was cloned into stable folder

cd stable
git pull
cp -v /boot/config-$(uname -r) .config #copy default ubuntu config
make CC=clang-18 localmodconfig
scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS
scripts/config --disable CONFIG_FRAME_WARN
scripts/config --set-str CONFIG_SYSTEM_TRUSTED_KEYS ""
scripts/config --set-str CONFIG_SYSTEM_REVOCATION_KEYS ""
scripts/config --enable CONFIG_SCSI_SYM53C8XX_2
fakeroot make CC=clang-18 -j2

 

[esi_y]

cp -v /boot/config-$(uname -r) .config #copy default ubuntu config

please attach here your specific one (just to be sure) - you can attach as a file to the forum

 

[esi_y]

The other thing, if all you are after is:

scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS
scripts/config --disable CONFIG_FRAME_WARN
scripts/config --set-str CONFIG_SYSTEM_TRUSTED_KEYS ""
scripts/config --set-str CONFIG_SYSTEM_REVOCATION_KEYS ""
scripts/config --enable CONFIG_SCSI_SYM53C8XX_2

... and you intend to use this for real use, why not make use of the "official way" and tweak that instead of Ubuntu one?

https://github.com/proxmox/pve-kernel/blob/master/README

 

[pderen]

here is my .config

 

[esi_y]

Your update-initramfs showed nothing suspicious?

 

[pderen]

My initramfs file is small compared to the generic

drwxr-xr-x 18 root root 4096 May 20 20:24 ..
-rw-r--r-- 1 root root 177489 Aug 7 21:01 config-6.10.3-pve
-rw-r--r-- 1 root root 287593 Aug 7 04:07 config-6.10.3-zabbly+
-rw-r--r-- 1 root root 287593 Aug 11 18:03 config-6.10.4-zabbly+
-rw-r--r-- 1 root root 280345 Jul 26 14:34 config-6.5.13-6-pve
-rw-r--r-- 1 root root 286709 Aug 5 18:17 config-6.8.12-1-pve
-rw-r--r-- 1 root root 286835 Jul 26 13:15 config-6.8.8-4-pve
drwxr-xr-x 2 root root 4096 May 22 2022 efi
drwxr-xr-x 6 root root 4096 Aug 13 17:34 grub
-rw-r--r-- 1 root root 26291204 Aug 7 21:13 initrd.img-6.10.3-pve
-rw-r--r-- 1 root root 58592649 Aug 8 20:10 initrd.img-6.10.3-zabbly+
-rw-r--r-- 1 root root 58583336 Aug 13 17:33 initrd.img-6.10.4-zabbly+
-rw-r--r-- 1 root root 60530371 Jul 26 22:30 initrd.img-6.5.13-6-pve
-rw-r--r-- 1 root root 60161214 Aug 8 20:07 initrd.img-6.8.12-1-pve
-rw-r--r-- 1 root root 61342081 Jul 26 22:30 initrd.img-6.8.8-4-pve
-rw-r--r-- 1 root root 138712 Feb 11 2023 memtest86+ia32.bin
-rw-r--r-- 1 root root 139776 Feb 11 2023 memtest86+ia32.efi
-rw-r--r-- 1 root root 144312 Feb 11 2023 memtest86+x64.bin
-rw-r--r-- 1 root root 145408 Feb 11 2023 memtest86+x64.efi
drwxr-xr-x 2 root root 4096 Aug 8 20:08 pve
-rw-r--r-- 1 root root 7698381 Aug 7 21:01 System.map-6.10.3-pve
-rw-r--r-- 1 root root 7916269 Aug 7 04:07 System.map-6.10.3-zabbly+
-rw-r--r-- 1 root root 7916975 Aug 11 18:03 System.map-6.10.4-zabbly+
-rw-r--r-- 1 root root 7978698 Jul 26 14:34 System.map-6.5.13-6-pve
-rw-r--r-- 1 root root 8037342 Aug 5 18:17 System.map-6.8.12-1-pve
-rw-r--r-- 1 root root 8338194 Jul 26 13:15 System.map-6.8.8-4-pve
-rw-r--r-- 1 root root 12771328 Aug 7 21:01 vmlinuz-6.10.3-pve
-rw-r--r-- 1 root root 13550080 Aug 7 04:07 vmlinuz-6.10.3-zabbly+
-rw-r--r-- 1 root root 13537792 Aug 11 18:03 vmlinuz-6.10.4-zabbly+
-rw-r--r-- 1 root root 13561448 Jul 26 14:34 vmlinuz-6.5.13-6-pve
-rw-r--r-- 1 root root 13683816 Aug 5 18:17 vmlinuz-6.8.12-1-pve
-rw-r--r-- 1 root root 14150760 Jul 26 13:15 vmlinuz-6.8.8-4-pve

 

[esi_y]

What does lsmod show on your running system (the one with PVE kernel)?

 

[pderen]

This is lsmod on "good" kernel that sees lvm

Linux proxmox 6.10.3-zabbly+ #debian12 SMP PREEMPT_DYNAMIC Wed Aug 7 02:53:53 UTC 2024 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Aug 13 17:57:19 2024 from 192.168.178.20
root@proxmox:~# lsmod
Module Size Used by
tcp_diag 12288 0
inet_diag 24576 1 tcp_diag
cfg80211 1224704 0
veth 36864 0
ebtable_filter 12288 0
ebtables 40960 1 ebtable_filter
ip_set 61440 0
ip6table_raw 12288 0
iptable_raw 12288 0
ip6table_filter 12288 0
ip6_tables 32768 2 ip6table_filter,ip6table_raw
iptable_filter 12288 0
scsi_transport_iscsi 167936 1
nf_tables 327680 0
bridge 393216 0
stp 12288 1 bridge
llc 16384 2 bridge,stp
bonding 229376 0
tls 139264 1 bonding
softdog 12288 2
sunrpc 774144 1
nfnetlink_log 20480 1
nfnetlink 20480 4 nf_tables,ip_set,nfnetlink_log
binfmt_misc 24576 1
ipmi_ssif 40960 0
nouveau 2777088 0
snd_hda_codec_hdmi 86016 1
mxm_wmi 12288 1 nouveau
intel_powerclamp 16384 0
drm_gpuvm 36864 1 nouveau
drm_exec 12288 2 drm_gpuvm,nouveau
snd_hda_intel 57344 0
coretemp 16384 0
gpu_sched 57344 1 nouveau
snd_intel_dspcfg 36864 1 snd_hda_intel
drm_ttm_helper 12288 1 nouveau
snd_intel_sdw_acpi 16384 1 snd_intel_dspcfg
ttm 90112 2 drm_ttm_helper,nouveau
snd_hda_codec 192512 2 snd_hda_codec_hdmi,snd_hda_intel
kvm_intel 393216 4
drm_display_helper 225280 1 nouveau
snd_hda_core 131072 3 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec
snd_hwdep 16384 1 snd_hda_codec
drm_kms_helper 233472 2 drm_display_helper,nouveau
snd_pcm 172032 4 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd_hda_core
kvm 1232896 3 kvm_intel
i2c_algo_bit 16384 1 nouveau
video 73728 1 nouveau
sha256_ssse3 32768 0
sha1_ssse3 32768 0
snd_timer 49152 1 snd_pcm
acpi_ipmi 20480 0
snd 131072 6 snd_hda_codec_hdmi,snd_hwdep,snd_hda_intel,snd_hda_codec,snd_timer,snd_pcm
wmi 28672 3 video,mxm_wmi,nouveau
intel_cstate 24576 0
soundcore 16384 1 snd
ipmi_si 77824 1
pcspkr 12288 0
i7core_edac 32768 0
ipmi_devintf 16384 0
ipmi_msghandler 77824 4 ipmi_devintf,ipmi_si,acpi_ipmi,ipmi_ssif
input_leds 12288 0
joydev 24576 0
mac_hid 12288 0
vhost_net 32768 1
vhost 61440 1 vhost_net
vhost_iotlb 16384 1 vhost
tap 28672 1 vhost_net
drm 679936 9 gpu_sched,drm_kms_helper,drm_exec,drm_gpuvm,drm_display_helper,drm_ttm_helper,ttm,nouveau
efi_pstore 12288 0
dmi_sysfs 20480 0
ip_tables 32768 2 iptable_filter,iptable_raw
x_tables 57344 7 ebtables,ip6table_filter,ip6table_raw,iptable_filter,ip6_tables,iptable_raw,ip_tables
autofs4 57344 2
btrfs 1855488 0
blake2b_generic 24576 0
xor 20480 1 btrfs
raid6_pq 118784 1 btrfs
libcrc32c 12288 2 btrfs,nf_tables
simplefb 12288 0
hid_generic 12288 0
gpio_ich 16384 0
usbkbd 12288 0
usbmouse 12288 0
usbhid 69632 0
hid 245760 2 usbhid,hid_generic
i2c_i801 36864 0
i2c_mux 12288 1 i2c_i801
e1000e 327680 0
i2c_smbus 16384 1 i2c_i801
lpc_ich 28672 0
pata_acpi 12288 0
ahci 49152 2
libahci 53248 1 ahci

 

[esi_y]

What does cat /proc/partitions show inside the BB?

(Gotta go now, will be back later though.)

 

[pderen]

On "good" kernel

root@proxmox:~# cat /proc/partitions
major minor #blocks name

7 0 134217728 loop0
7 1 33554432 loop1
11 0 1048575 sr0
8 0 488386584 sda
8 1 1007 sda1
8 2 524288 sda2
8 3 487861255 sda3
252 0 8388608 dm-0
252 1 479469568 dm-1

 

[esi_y]

On "good" kernel

root@proxmox:~# cat /proc/partitions
major minor #blocks name

7 0 134217728 loop0
7 1 33554432 loop1
11 0 1048575 sr0
8 0 488386584 sda
8 1 1007 sda1
8 2 524288 sda2
8 3 487861255 sda3
252 0 8388608 dm-0
252 1 479469568 dm-1

I meant inside the BusyBox on the "bad" one

 

[pderen]

on "bad" kernel shows only

11 0 1048575 sr0

 

[esi_y]

Ok this is probably simpler than I had thought. On the good running system, can you please post (assuming sda is your system drive, seems to be just that one there, alognside a CDROM?):

udevadm info -a -n /dev/sda

Please attach the output as a file or inline into [ CODE ] ... [ /CODE ] tags.

 

[pderen]

on "good" kernel:

root@proxmox:~# udevadm info -a -n /dev/sda >output.txt
root@proxmox:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 128G 0 loop
loop1 7:1 0 32G 0 loop
sda 8:0 0 465.8G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part
└─sda3 8:3 0 465.3G 0 part
├─pve-swap 252:0 0 8G 0 lvm [SWAP]
└─pve-root 252:1 0 457.3G 0 lvm /
sr0 11:0 1 1024M 0 rom

 

[esi_y]

Alert! /dev/mapper/pve-root does not exist. Dropping to shell

Running lvm pvscan returns nothin, and adding rootdelay=10 to grub also didn't help.

Any ideas what to do?

Back to this (screenshot with lsmod from within BusyBox did not get quoted)...

Are you able to modprobe ahci at this point?