kernel panic when creating VMs (centos 9 stream ISO)

[Robson Rissato]

Good morning friends

I have proxmox 7.1-10
updated.

I'm trying to create VM with centers
CentOS-Stream-9-latest-x86_64-dvd1.iso

but when VM starts to install the operating system

enter kernel panic.


jai if this was a problem, i installed this in my local virtual box and everything went well..

but in proxmox it doesn't work... can anyone help me?

 

[oguz]

hi,

I have proxmox 7.1-10
updated.

I'm trying to create VM with centers
CentOS-Stream-9-latest-x86_64-dvd1.iso

but when VM starts to install the operating system

enter kernel panic.

* are you getting the kernel panic inside the VM or on the PVE host?

depending on the answer to my above questions please post:
* pveversion -v
* qm config VMID (change VMID to yours)
* if you can post the trace here (you should be able to find that in dmesg), that would help too, you can use [code][/code] tags

 

[Robson Rissato]

inside the vm when the installation will start




pveversion -v
Linux xorn 5.13.19-4-pve #1 SMP PVE 5.13.19-9 (Mon, 07 Feb 2022 11:01:14 +0100) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Feb 9 11:07:31 -03 2022 on pts/0
root@xorn:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-4-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-10
pve-kernel-5.13: 7.1-7
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-4
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.11.22-3-pve: 5.11.22-7
pve-kernel-5.11.22-2-pve: 5.11.22-4
pve-kernel-5.4.124-1-pve: 5.4.124-1
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-1
pve-xtermjs: 4.16.0-1
pve-zsync: 2.2.1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

qm config VMID (106)

acpi: 1
boot: order=ide2;scsi0;scsi1;net0
cores: 2
ide2: local:iso/CentOS-Stream-9-latest-x86_64-dvd1.iso,media=cdrom,size=8214M
kvm: 1
memory: 4192
meta: creation-qemu=6.1.0,ctime=1644414135
name: VM106
net0: virtio=6E:7D:85:29:AB:76,bridge=vmbr1
numa: 0
ostype: l26
scsi0: local:106/vm-106-disk-0.qcow2,size=100G,ssd=1
scsi1: local:106/vm-106-disk-1.qcow2,size=500G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=51709630-b6c2-41a9-b01d-4b14527cc1e1
sockets: 1
vmgenid: d53fce54-5afc-4a31-af83-cd18ad454b34

 

[oguz]

thanks for sharing the outputs.

inside the vm when the installation will start

have you tried with another VM or another ISO? is this the only one with that issue?

 

[Robson Rissato]

I tested with debian 11.2 and it is normal...only with centos 9

 

[oguz]

I tested with debian 11.2 and it is normal...only with centos 9

okay interesting... could you check if that ISO is not corrupted? you could compare the checksums from where you downloaded it

 

[Robson Rissato]

I already did this 4 x

I downloaded it again

and same iso I used locally and it works.
only on proxmox does not work

 

[Robson Rissato]

sha256sum CentOS-Stream-9-latest-x86_64-dvd1.iso
2688febc19b7abca5423fed70882649b7f95f1a9750873aabae007b04d2ea693 CentOS-Stream-9-latest-x86_64-dvd1.iso

 

[oguz]

sha256sum CentOS-Stream-9-latest-x86_64-dvd1.iso
2688febc19b7abca5423fed70882649b7f95f1a9750873aabae007b04d2ea693 CentOS-Stream-9-latest-x86_64-dvd1.iso

thanks, i can verify that i have the same one here downloaded from [0], and i can reproduce the kernel panic from your screenshot when starting the installation.

in the meantime you could try installing from a different ISO and see if that works for you

[0]: https://www.centos.org/centos-stream/

 

[Robson Rissato]

 

[Robson Rissato]

centos 8 stream

installation works normal

 

[Robson Rissato]

does not work if configured like this centos 9 stream



but that's how it works


I believe that the problem is in the default( kvm64) of the proxmox

 

[ivenae]

You use 5.13.19-4
This Kernel has known bugs.
Switch back to 5.13.19-3 or – if not productive – forward to 5.15.
I would not use that kernel. Search forum for this kernel version.

 

[oguz]

You use 5.13.19-4
This Kernel has known bugs.
Switch back to 5.13.19-3 or – if not productive – forward to 5.15.
I would not use that kernel. Search forum for this kernel version.

that's true, though i was able to replicate the same errors on a 5.15 kernel setup. the panic happens inside the VM because the init process gets killed for some reason (though i haven't got around to debugging this yet). my guess would be that this specific ISO is causing the issue since it doesn't seem to happen with other ones.

centos 8 stream

installation works normal

could you try an older centos 9 ISO?

 

[em3034]

I'm encountering the same issue with the new CentOS Stream 9 ISO. I've tried it on 2 proxmox nodes, one based on Intel and the other one AMD.
The same works fine using a previous CentOS Stream 9 ISO from Begining January. It also works fine using the latest RHEL9 Beta.
It would be interesting to see if it can be installed on other Hypervisor than proxmox. Anybody tried?

 

[oguz]

I'm encountering the same issue with the new CentOS Stream 9 ISO. I've tried it on 2 proxmox nodes, one based on Intel and the other one AMD.
The same works fine using a previous CentOS Stream 9 ISO from Begining January. It also works fine using the latest RHEL9 Beta.

thanks for verifying that the older centos 9 stream ISO is working fine, then it's most likely something they've changed in the ISO that's causing the issue...

 

[oguz]

does not work if configured like this centos 9 stream
View attachment 34154


but that's how it works
View attachment 34155

I believe that the problem is in the default( kvm64) of the proxmox

i've just tried that now too, it seems to work fine with the host CPU type, thanks for checking!

It would be interesting to see if it can be installed on other Hypervisor than proxmox. Anybody tried?

no idea, but it works with the CPU type set to host.

if you have similar issues in different hypervisors then it might be a bug on centos' end (would make sense to report it to them if that's the case)

 

[em3034]

Actually the workaround (spotted by Robson earlier in this thread) to set the CPU type to Host instead of Default (kvm64) solves the problem.
But using the default doesn't while is used to work with previous Stream 9 isos.

 

[sml42]

The error I am getting with CPU Type KVM64

fatal glibc error: cpu does not support x86-64-v2
and kernel panic afterwards

I can confirm that with CPU type host everything works as expected.

 

[Nicox]

I get the exact same issue with Centos 9 Stream. Putting CPU type to host resolved the problem. Is it correct to keep this option for long term deployment ?