8.3 + Debian 12 + Cloud-Init + Expanded Disk + OVMF = Kernel Panic on First Boot

S

SlothCroissant

Active Member
Feb 26, 2019
16
0
41
35
I have a consistent repro of a kernel panic, seemingly occurring due to something between cloud-init and Proxmox running 8.3. Likely not something your average user will find, as it requires expanding a disk on first install (whereas most click-ops would be the ISO installer, I assume). My example used Ansible, found this as well on a Terraform provider: https://github.com/bpg/terraform-provider-proxmox/issues/1639, but I can also reproduce the issue entirely natively in Proxmox:

  1. Download the latest debian-12-generic-amd64.qcow2 from https://cdimage.debian.org/images/cloud/bookworm/ (example is using the image uploaded on 2024-12-02 17:54)
  2. Create a VM without a disk, enable cloud-init (See below for the resulting .conf to know what settings were used):
  3. Import the previously-downloaded qcow2: qm disk import 100 /var/lib/vz/template/vm/debian-12-generic-amd64.qcow2 local-zfs
  4. Associate/mount the disk using default settings in the VM hardware config
  5. Expand the disk (I expanded mine by 64G, though my Ansible example expands by 62G) in the web UI.
  6. Start the VM (below is the final .conf file)
Notes:
  • A reboot/reset makes it past the kernel panic, and it *appears* as though the cloud-init is applied properly.
  • The issue can be worked around by adding serial0: socket. Not sure at all why this would be.
Final VM conf:

Code: 
root@pve01:~# cat /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot:
cipassword: $5$3VyDlYb6$9e.xg94NF1/HWdchmwqKOtcoh3gc5h9UUDu196bmVV5
ciuser: testuser
cores: 2
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,size=1M
ide0: local-zfs:vm-100-cloudinit,media=cdrom
ipconfig0: ip=dhcp
machine: q35
memory: 4096
meta: creation-qemu=9.0.2,ctime=1736116049
name: test
net0: virtio=BC:24:11:06:EF:F5,bridge=vmbr0,tag=10
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-100-disk-2,iothread=1,size=66G
scsihw: virtio-scsi-single
smbios1: uuid=b4913ce3-c7f0-4eaf-9fab-1ab7f7a5738b
sockets: 2
tpmstate0: local-zfs:vm-100-disk-1,size=4M,version=v2.0
vmgenid: 6091fe0e-7915-4bea-833d-161da806c628

Result:

Screenshot 2025-01-05 at 16.43.58.png

pveversion -v:
Code: 
root@pve01:~# pveversion
pve-manager/8.3.2/3e76eec21c4a14a7 (running kernel: 6.8.12-5-pve)
root@pve01:~# pveversion -v
proxmox-ve: 8.3.0 (running kernel: 6.8.12-5-pve)
pve-manager: 8.3.2 (running version: 8.3.2/3e76eec21c4a14a7)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-5
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.3
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-2
pve-ha-manager: 4.0.6
pve-i18n: 3.3.2
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.3
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1

Hardware: Dell R730XD
  • CPU: Xeon E5-2683v4 (2X)
  • RAM: 256GB DDR4
  • Disks: 6x Toshiba 3.84TB SAS SSD in ZFS RAID 10
The kernel panic appears to be due to "Kernel panic - not syncing: Attempted to kill init!", though since I can't scroll back, I can't see more detail.
 
Last edited:
Ali3n said:
Same issue, not working properly...
Click to expand...
Have you tried the solution by SlothCroissant?
Code: 
The issue can be worked around by adding serial0: socket. Not sure at all why this would be.

Did you try a Debian Bookworm cloud image, or a different one?
 
mira said:
Have you tried the solution by SlothCroissant?
Code: 
The issue can be worked around by adding serial0: socket. Not sure at all why this would be.

Did you try a Debian Bookworm cloud image, or a different one?
Click to expand...
Have you tried the solution by SlothCroissant? > Yes, didn't change anything, running:
proxmox-ve: 8.3.0 (running kernel: 6.8.12-7-pve)
pve-manager: 8.3.3 (running version: 8.3.3/f157a38b211595d6)

I've tried with those (https://cdimage.debian.org/images/cloud/bookworm/latest/):
https://cdimage.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2
https://cdimage.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.raw
https://cdimage.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.tar.xz

With or without "serial0: socket" Cloud-Init itself didn't function, e.g. VM Hostname (internal) is localhost, given username+pw+ssh-key don't works.
 
Since your issue is unrelated to the original one, please create a new thread.
In the new thread please provide the complete VM config (qm config <VMID>).
And check the cloud-init logs in the guest.
 
I have the same issue, and what did i found - it happens after resize system drive while VM is stopped.
It seems like kernel & Ext4 filesystem issue, because when main drive is not extended - first boot does correct.
 
  • Like
Reactions: Ali3n
mira said:
Since your issue is unrelated to the original one, please create a new thread.
In the new thread please provide the complete VM config (qm config <VMID>).
And check the cloud-init logs in the guest.
Click to expand...
Are you sure about it? I'm not getting further than others tbh..

Without main drive size increase (Can't login because no Cloud-Init Config was transfered i guess):
(qm importdisk 5000 --format=qcow2 debian-12-genericcloud-amd64.qcow2 CephPool2)
edit: also the hostname didn't set.. should be Cloud-Init.
1739364249025.png

With main drive size increase:
(qm importdisk 5000 --format=qcow2 debian-12-genericcloud-amd64.qcow2 CephPool2 && qm disk resize 5000 virtio0 10G)
1739364456090.png

################################################
qm config 5000:

agent: 1,fstrim_cloned_disks=1,type=virtio
bios: ovmf
boot: order=virtio0
cipassword: x
ciuser: x
cores: 2
cpu: host
efidisk0: CephPool2:vm-5000-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide2: CephPool2:vm-5000-cloudinit,media=cdrom
ipconfig0: ip=172.x.x.x/24,gw=172.x.x.1
machine: q35
memory: 2048
meta: creation-qemu=9.0.2,ctime=1738909293
name: Cloud-Init
net0: virtio=BC:24:11:x:x:x,bridge=vmbr1,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=x
sockets: 1
sshkeys: ssh-rsa-x
 
Last edited:
Ali3n said:
Are you sure about it? I'm not getting further than others tbh..

Without main drive size increase (Can't login because no Cloud-Init Config was transfered i guess):
(qm importdisk 5000 --format=qcow2 debian-12-genericcloud-amd64.qcow2 CephPool2)
edit: also the hostname didn't set.. should be Cloud-Init.
View attachment 82234

With main drive size increase:
(qm importdisk 5000 --format=qcow2 debian-12-genericcloud-amd64.qcow2 CephPool2 && qm disk resize 5000 virtio0 10G)
View attachment 82235

################################################
qm config 5000:

agent: 1,fstrim_cloned_disks=1,type=virtio
bios: ovmf
boot: order=virtio0
cipassword: x
ciuser: x
cores: 2
cpu: host
efidisk0: CephPool2:vm-5000-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide2: CephPool2:vm-5000-cloudinit,media=cdrom
ipconfig0: ip=172.x.x.x/24,gw=172.x.x.1
machine: q35
memory: 2048
meta: creation-qemu=9.0.2,ctime=1738909293
name: Cloud-Init
net0: virtio=BC:24:11:x:x:x,bridge=vmbr1,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=x
sockets: 1
sshkeys: ssh-rsa-x
Click to expand...
Yes, here you need to add a serial socket and set the display to it. That's an issue of the guest image and the same issue the original thread starter mentioned, including the workaround.

With this workaround applied, the config should be used by cloud-init. If not, check the cloud-init logs in the guest, as well as the journal of the boot.
So let's first try to apply the workaround on a new VM and see if the kernel panic still happens.
 
mira said:
Yes, here you need to add a serial socket and set the display to it. That's an issue of the guest image and the same issue the original thread starter mentioned, including the workaround.

With this workaround applied, the config should be used by cloud-init. If not, check the cloud-init logs in the guest, as well as the journal of the boot.
So let's first try to apply the workaround on a new VM and see if the kernel panic still happens.
Click to expand...
The VM boots with and without increasing the main drive size and serial socket, but as mentioned, I currently have no way to log in. From what I’ve seen, there is no "Cloud-Init Default Login," and no credentials seem to be set.

Since I cannot log in, I also cannot check the cloud-init logs inside the VM. I tried checking the cloud-init configuration from Proxmox using "qm cloudinit dump <VMID> user", but the output doesn’t show any valid credentials. (Console: Login incorrect)

Is there another way to verify if Cloud-Init is correctly applying user credentials? Or is there a known issue where the user settings are not passed to the VM? Any advice on how to proceed would be appreciated!
 
Oh, sorry I missed that before.
Add `ide2` (your cloud-init disk) to the boot order, or switch it to SATA or SCSI:
Code: 
 ide2: CephPool2:vm-5000-cloudinit,media=cdrom

You've been running in 2 separate issues. The first one is the kernel panic when resizing the disk without a serial socket, and the 2nd that the IDE drive is not available early enough when booting the first time with OVMF for cloud-init to use it [0].


[0] https://forum.proxmox.com/threads/s...late-vm-uefi-customization.151811/post-687869
 
  • Like
Reactions: fiona and Ali3n
mira said:
Oh, sorry I missed that before.
Add `ide2` (your cloud-init disk) to the boot order, or switch it to SATA or SCSI:
Code: 
 ide2: CephPool2:vm-5000-cloudinit,media=cdrom

You've been running in 2 separate issues. The first one is the kernel panic when resizing the disk without a serial socket, and the 2nd that the IDE drive is not available early enough when booting the first time with OVMF for cloud-init to use it [0].


[0] https://forum.proxmox.com/threads/s...late-vm-uefi-customization.151811/post-687869
Click to expand...
Oh my god, thank you so much!

Yes you're correct, this were two issues. It works now if serial 0 + serial socket & cloudInit-Drive scsi0 is set.

The Kernel Panic error still persists with only "serial 0 + serial socket"

Thank you!!!!
 
You must log in or register to reply here.
Top Bottom