8.3 + Debian 12 + Cloud-Init + Expanded Disk + OVMF = Kernel Panic on First Boot

S

SlothCroissant

Active Member
Feb 26, 2019
16
0
41
35
I have a consistent repro of a kernel panic, seemingly occurring due to something between cloud-init and Proxmox running 8.3. Likely not something your average user will find, as it requires expanding a disk on first install (whereas most click-ops would be the ISO installer, I assume). My example used Ansible, found this as well on a Terraform provider: https://github.com/bpg/terraform-provider-proxmox/issues/1639, but I can also reproduce the issue entirely natively in Proxmox:

  1. Download the latest debian-12-generic-amd64.qcow2 from https://cdimage.debian.org/images/cloud/bookworm/ (example is using the image uploaded on 2024-12-02 17:54)
  2. Create a VM without a disk, enable cloud-init (See below for the resulting .conf to know what settings were used):
  3. Import the previously-downloaded qcow2: qm disk import 100 /var/lib/vz/template/vm/debian-12-generic-amd64.qcow2 local-zfs
  4. Associate/mount the disk using default settings in the VM hardware config
  5. Expand the disk (I expanded mine by 64G, though my Ansible example expands by 62G) in the web UI.
  6. Start the VM (below is the final .conf file)
Notes:
  • A reboot/reset makes it past the kernel panic, and it *appears* as though the cloud-init is applied properly.
  • The issue can be worked around by adding serial0: socket. Not sure at all why this would be.
Final VM conf:

Code: 
root@pve01:~# cat /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot:
cipassword: $5$3VyDlYb6$9e.xg94NF1/HWdchmwqKOtcoh3gc5h9UUDu196bmVV5
ciuser: testuser
cores: 2
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,size=1M
ide0: local-zfs:vm-100-cloudinit,media=cdrom
ipconfig0: ip=dhcp
machine: q35
memory: 4096
meta: creation-qemu=9.0.2,ctime=1736116049
name: test
net0: virtio=BC:24:11:06:EF:F5,bridge=vmbr0,tag=10
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-100-disk-2,iothread=1,size=66G
scsihw: virtio-scsi-single
smbios1: uuid=b4913ce3-c7f0-4eaf-9fab-1ab7f7a5738b
sockets: 2
tpmstate0: local-zfs:vm-100-disk-1,size=4M,version=v2.0
vmgenid: 6091fe0e-7915-4bea-833d-161da806c628

Result:

Screenshot 2025-01-05 at 16.43.58.png

pveversion -v:
Code: 
root@pve01:~# pveversion
pve-manager/8.3.2/3e76eec21c4a14a7 (running kernel: 6.8.12-5-pve)
root@pve01:~# pveversion -v
proxmox-ve: 8.3.0 (running kernel: 6.8.12-5-pve)
pve-manager: 8.3.2 (running version: 8.3.2/3e76eec21c4a14a7)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-5
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.3
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-2
pve-ha-manager: 4.0.6
pve-i18n: 3.3.2
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.3
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1

Hardware: Dell R730XD
  • CPU: Xeon E5-2683v4 (2X)
  • RAM: 256GB DDR4
  • Disks: 6x Toshiba 3.84TB SAS SSD in ZFS RAID 10
The kernel panic appears to be due to "Kernel panic - not syncing: Attempted to kill init!", though since I can't scroll back, I can't see more detail.
 
Last edited:
Ali3n said:
Same issue, not working properly...
Click to expand...
Have you tried the solution by SlothCroissant?
Code: 
The issue can be worked around by adding serial0: socket. Not sure at all why this would be.

Did you try a Debian Bookworm cloud image, or a different one?
 
mira said:
Have you tried the solution by SlothCroissant?
Code: 
The issue can be worked around by adding serial0: socket. Not sure at all why this would be.

Did you try a Debian Bookworm cloud image, or a different one?
Click to expand...
Have you tried the solution by SlothCroissant? > Yes, didn't change anything, running:
proxmox-ve: 8.3.0 (running kernel: 6.8.12-7-pve)
pve-manager: 8.3.3 (running version: 8.3.3/f157a38b211595d6)

I've tried with those (https://cdimage.debian.org/images/cloud/bookworm/latest/):
https://cdimage.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2
https://cdimage.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.raw
https://cdimage.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.tar.xz

With or without "serial0: socket" Cloud-Init itself didn't function, e.g. VM Hostname (internal) is localhost, given username+pw+ssh-key don't works.
 
Since your issue is unrelated to the original one, please create a new thread.
In the new thread please provide the complete VM config (qm config <VMID>).
And check the cloud-init logs in the guest.
 
I have the same issue, and what did i found - it happens after resize system drive while VM is stopped.
It seems like kernel & Ext4 filesystem issue, because when main drive is not extended - first boot does correct.
 
  • Like
Reactions: Ali3n
mira said:
Since your issue is unrelated to the original one, please create a new thread.
In the new thread please provide the complete VM config (qm config <VMID>).
And check the cloud-init logs in the guest.
Click to expand...
Are you sure about it? I'm not getting further than others tbh..

Without main drive size increase (Can't login because no Cloud-Init Config was transfered i guess):
(qm importdisk 5000 --format=qcow2 debian-12-genericcloud-amd64.qcow2 CephPool2)
edit: also the hostname didn't set.. should be Cloud-Init.
1739364249025.png

With main drive size increase:
(qm importdisk 5000 --format=qcow2 debian-12-genericcloud-amd64.qcow2 CephPool2 && qm disk resize 5000 virtio0 10G)
1739364456090.png

################################################
qm config 5000:

agent: 1,fstrim_cloned_disks=1,type=virtio
bios: ovmf
boot: order=virtio0
cipassword: x
ciuser: x
cores: 2
cpu: host
efidisk0: CephPool2:vm-5000-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide2: CephPool2:vm-5000-cloudinit,media=cdrom
ipconfig0: ip=172.x.x.x/24,gw=172.x.x.1
machine: q35
memory: 2048
meta: creation-qemu=9.0.2,ctime=1738909293
name: Cloud-Init
net0: virtio=BC:24:11:x:x:x,bridge=vmbr1,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=x
sockets: 1
sshkeys: ssh-rsa-x
 
Last edited:
Ali3n said:
Are you sure about it? I'm not getting further than others tbh..

Without main drive size increase (Can't login because no Cloud-Init Config was transfered i guess):
(qm importdisk 5000 --format=qcow2 debian-12-genericcloud-amd64.qcow2 CephPool2)
edit: also the hostname didn't set.. should be Cloud-Init.
View attachment 82234

With main drive size increase:
(qm importdisk 5000 --format=qcow2 debian-12-genericcloud-amd64.qcow2 CephPool2 && qm disk resize 5000 virtio0 10G)
View attachment 82235

################################################
qm config 5000:

agent: 1,fstrim_cloned_disks=1,type=virtio
bios: ovmf
boot: order=virtio0
cipassword: x
ciuser: x
cores: 2
cpu: host
efidisk0: CephPool2:vm-5000-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide2: CephPool2:vm-5000-cloudinit,media=cdrom
ipconfig0: ip=172.x.x.x/24,gw=172.x.x.1
machine: q35
memory: 2048
meta: creation-qemu=9.0.2,ctime=1738909293
name: Cloud-Init
net0: virtio=BC:24:11:x:x:x,bridge=vmbr1,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=x
sockets: 1
sshkeys: ssh-rsa-x
Click to expand...
Yes, here you need to add a serial socket and set the display to it. That's an issue of the guest image and the same issue the original thread starter mentioned, including the workaround.

With this workaround applied, the config should be used by cloud-init. If not, check the cloud-init logs in the guest, as well as the journal of the boot.
So let's first try to apply the workaround on a new VM and see if the kernel panic still happens.
 
mira said:
Yes, here you need to add a serial socket and set the display to it. That's an issue of the guest image and the same issue the original thread starter mentioned, including the workaround.

With this workaround applied, the config should be used by cloud-init. If not, check the cloud-init logs in the guest, as well as the journal of the boot.
So let's first try to apply the workaround on a new VM and see if the kernel panic still happens.
Click to expand...
The VM boots with and without increasing the main drive size and serial socket, but as mentioned, I currently have no way to log in. From what I’ve seen, there is no "Cloud-Init Default Login," and no credentials seem to be set.

Since I cannot log in, I also cannot check the cloud-init logs inside the VM. I tried checking the cloud-init configuration from Proxmox using "qm cloudinit dump <VMID> user", but the output doesn’t show any valid credentials. (Console: Login incorrect)

Is there another way to verify if Cloud-Init is correctly applying user credentials? Or is there a known issue where the user settings are not passed to the VM? Any advice on how to proceed would be appreciated!
 
Oh, sorry I missed that before.
Add `ide2` (your cloud-init disk) to the boot order, or switch it to SATA or SCSI:
Code: 
 ide2: CephPool2:vm-5000-cloudinit,media=cdrom

You've been running in 2 separate issues. The first one is the kernel panic when resizing the disk without a serial socket, and the 2nd that the IDE drive is not available early enough when booting the first time with OVMF for cloud-init to use it [0].


[0] https://forum.proxmox.com/threads/s...late-vm-uefi-customization.151811/post-687869
 
  • Like
Reactions: fiona and Ali3n
mira said:
Oh, sorry I missed that before.
Add `ide2` (your cloud-init disk) to the boot order, or switch it to SATA or SCSI:
Code: 
 ide2: CephPool2:vm-5000-cloudinit,media=cdrom

You've been running in 2 separate issues. The first one is the kernel panic when resizing the disk without a serial socket, and the 2nd that the IDE drive is not available early enough when booting the first time with OVMF for cloud-init to use it [0].


[0] https://forum.proxmox.com/threads/s...late-vm-uefi-customization.151811/post-687869
Click to expand...
Oh my god, thank you so much!

Yes you're correct, this were two issues. It works now if serial 0 + serial socket & cloudInit-Drive scsi0 is set.

The Kernel Panic error still persists with only "serial 0 + serial socket"

Thank you!!!!
 
mira said:
You've been running in 2 separate issues. The first one is the kernel panic when resizing the disk without a serial socket
Click to expand...
Ah, interesting, Thx. I've just run into the same kernel panic problem (magically fixed by one VM reset resp. reboot), after learning the cloud-init approach over the last few days.

So, I guess we should have done
Code: 
qm set 9000 --serial0 socket
and maybe
Code: 
qm set 9000 --vga serial0
after all.

I our case I cloned a VM template (created with the cloud-init approach) and resized the disc before starting it.
Should the resize happen after the new VM was started?

PS: Here's how that successful 2nd VM start adapted to the larger disc on the fly:
Code: 
# journalctl -b -1  |egrep -i ext4
Mai 02 10:32:02 debian-12-template kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Quota mode: none.
Mai 02 10:32:02 debian-12-template kernel: EXT4-fs (sda1): re-mounted. Quota mode: none.
Mai 02 10:32:02 debian-12-template kernel: EXT4-fs (sda1): resizing filesystem from 753408 to 19628027 blocks
Mai 02 10:32:02 debian-12-template kernel: EXT4-fs (sda1): resized filesystem to 19628027
 
Last edited:
Christoph Lechleitner said:
I our case I cloned a VM template (created with the cloud-init approach) and resized the disc before starting it.
Should the resize happen after the new VM was started?
Click to expand...
You only resized the external (QEMU) disk from the point of the view of the hypervisor. Think about physical computer - you took a small disk out, put a larger one in. The OS in the PC will not see the increased size until it is started. Kernel needs to update its view, the OS needs to expand the file system, etc.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
> Think about physical computer

Thanks! Indeed. After over 20 years focussing on OpenVZ/LXC my mind is too accustomed to the host owning the containers' rootfs ;-)

We're finally moving to VMs though, because 1. live migration and 2. nasty cornercases with docker under LXC.

Since search engines got me here directly I'll do a test or two and report back here rg. disc enlarging.
 
About the disc enlarging, that's a RTFM on me, see pve.proxmox.com/wiki/Resize_disks.

In my case, Debian 12 Bookwom with cloud-image in .qcow2 variant (see cloud.debian.org/images/cloud/), the disk was not /dev/vda but /dev/sda (of course one calls lsblk and/or df -h before parted), but it was no problem doing the resize online after starting the VM.

No kernel panic. Yay. Thanks & sorry for the noise.

I'm not sure if qm set ${VMID} --serial0 socket --vga serial0 was required or not, probably not, the main difference I noticed was that the Start button in the Console window vanished (I had to start via main GUI, or CLI), but that's understandable.
 
Last edited:
You must log in or register to reply here.
Top Bottom