[SOLVED] PVE 7.0: All VMs with Cloud-init & SeaBIOS fail during boot process (bootloop/disk not found)

YAGA · Oct 2, 2021

Dear forum members,

Probably a few weeks ago, during a PVE 7.x update (PVE 7 was previously installed), all my VMs (QEMU) with Cloud-Init and SeaBIOS stopped starting. During the boot process it looks like the partition is not found and the SeaBIOS remains in an infinite loop: try to start the VM, it fails, try to start the VM, it fails, ...

~~If I recreate a new VM with Qemu and Cloud-Init it works perfectly.~~
Update 1: same issue with a new VM with Qemu with or without Cloud-Init.
Update 2: with the latest PVE update October 7th 2021 -> If I recreate a new VM with Qemu and Cloud-Init it works perfectly.

CTs (LXC) are not affected by this starting problem.

It appears that a reference or link to was broken to initiate the boot phase with Seabios during a recent update to PVE 7.x.

This occurs on different physical nodes which excludes a hardware problem.

Any suggestions are welcome,

Regards,

SeaBIOS snapshot during the boot process (ESC for boot menu)

Code:

SeaBIOS (version rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org)
Machine UUID 6b0065e1-38f3-4d7b-8812-32f35703f890

iPXE (http://ipxe.org) 00:12.0 C000 PCI2.10 PnP PMM+BFF8EDC0+BFEEEDC0 C000

Press ESC for boot menu.

Select boot device:

1. virtio-scsi Drive QEMU QEMU HARDDISK 2.5+
2. Legacy option rom
3. DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
4. iPXE (PCI 00:12.0)

101.conf (example of a non-booting VM)

Code:

#debian-10.3.0-openstack-amd64.qcow2.iso - QEMU VM: S2-UNMS 192.168.2.101/24 – created: 2020/03/25 13:16:25
agent: 1
boot: c
bootdisk: scsi0
citype: nocloud
ciuser: root
cores: 4
cpulimit: 4
ide2: local-lvm:vm-101-cloudinit,media=cdrom
ipconfig0: ip=192.168.2.101/24,gw=192.168.2.1
memory: 8192
name: S2-UNMS
net0: virtio=F6:8E:D5:95:C9:2A,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-101-disk-0,size=40G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=6b0065e1-38f3-4d7b-8812-32f35703f890
sockets: 1
sshkeys: ssh-rsa%.............................
vga: serial0
vmgenid: 7e208451-a6f9-4db1-ab4f-522449513e19

pveversion --verbose

Code:

proxmox-ve: 7.0-2 (running kernel: 5.11.22-4-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-7
pve-kernel-helper: 7.0-7
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.11.22-3-pve: 5.11.22-7
pve-kernel-5.11.22-2-pve: 5.11.22-4
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-4.15: 5.4-9
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-9
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-11
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.10-1
proxmox-backup-file-restore: 2.0.10-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-10
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-3
pve-firmware: 3.3-1
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-14
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

YAGA · Oct 4, 2021

Unfortunately the last PVE updates today didn't solve the issue.

pve-firmware 3.3-2
pve-kernel-5.11 7.0-8
pve-kernel-5.11.22-5-pve 5.11.22-10

VM are always in a bootloop (CDROM of Disk not found).
CT still working properly.

vmware · Oct 7, 2021

Hi all,

My new PVE 7.0 install is having very similar issue. The VM couldn't seem to find the boot drive during, and fell into "boot loop".

My PVE host runs simple ext4 LVM storage. VM was restored from a backup file from old 6.4 install.

Code:

# cat /etc/pve/qemu-server/102.conf

balloon: 1024
bootdisk: virtio0
cores: 2
memory: 3072
name: myvm
net0: virtio=.....,bridge=vmbr20,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=....
sockets: 1
startup: up=69
vga: serial0
virtio0: local-lvm:vm-102-disk-0,discard=on,size=20G
vmgenid: .....

~~Also want to add that I have 2 PVE 7.0 hosts. The other PVE host with Btrfs storage is not affected.~~

Both LVM and Btrfs PVE hosts are affected

YAGA · Oct 7, 2021

Hi there,

I've exactly the same issue, VM with Cloud Init previously created from PVE 6.x are not working with the latest PVE 7.x update from community repository.

VM can't start, there are in a bootloop in SeaBIOS (Disk not found).

My PVE host also uses ext4 LVM storage.

With the latest PVE update (October 7th 2021), If I recreate a new VM with Qemu and Cloud-Init it works perfectly.

Only previously created VM from PVE 6.4 can't boot anymore with the latest PVE 7.x update.

Any suggestions are welcome,

Regards,

vmware · Oct 8, 2021

Hi YAGA,

Yup, looks like we run into the exact same issue. Did you restore those VMs before, or after the Oct 7th PVE update?

I have only one VM that could boot in PVE 7.0. It was restored from 6.4 backup on 3rd Oct. I suspect the restore script has a bug, corrupting the VM disk image during restore.

My plan is to roll back the restore script (Persumably, package proxmox-backup-file-restore) and then test.

YAGA · Oct 8, 2021

vmware said:
Hi YAGA,

Yup, looks like we run into the exact same issue. Did you restore those VMs before, or after the Oct 7th PVE update?

I have only one VM that could boot in PVE 7.0. It was restored from 6.4 backup on 3rd Oct. I suspect the restore script has a bug, corrupting the VM disk image during restore.

My plan is to roll back the restore script (Persumably, package proxmox-backup-file-restore) and then test.

Hi,

I've several nodes and all of them are affected by the bootloop issue: all existing VM with Cloud-Init didn't boot anymore.

From now, I can create a new VM with Cloud-Init and it works.

2 nodes have been upgraded from PVE 6.x to PVE 7.x and all existing VM with Cloud-Init, originally created with PVE 6.x, no longer work today.
4 nodes have been fresh installed with PVE 7.x and all the VM with Cloud-Init restored from a backup, originally created with PVE 6.x, no longer work today.

The problem probably occurred a few weeks ago during a PVE update and after a node reboot.

On the other hand, I am sure that all the VMs were working perfectly after the migration or the installation of PVE 7.x but it is a later PVE update which now prevents booting.

Regards,

vmware · Oct 9, 2021

I've never used cloud-init, but my VMs also failed to boot.

YAGA, I was wondering what OS your VMs run? Debian or Ubuntu or something else? My only VM that is still booting runs Ubuntu 20.04. The one that I tested and failed runs Debian 10.

Rolling back proxmox-backup-file-restore didn't seem to help. Right now I'm trying to make backups using Proxmox Backup Server, then restore them on the new PVE 7.0 hosts.

YAGA · Oct 9, 2021

All my MVs run Debian 10.

I've no clue to debug since SeaBIOS is not able to boot the VM.

journalctl doesn't provide useful information in my case.

Any advice to debug will be very welcome,

Regards,

spirit · Oct 9, 2021

I don't see "boot" option in your config.

do you have something like : "boot: order=scsi0" ? (it's needed to find boot drive)

YAGA · Oct 9, 2021

spirit said:
I don't see "boot" option in your config.

do you have something like : "boot: order=scsi0" ? (it's needed to find boot drive)

Hi Spirit,

I've manually added a line with "boot: order=scsi0" in the VM config file but it doesn't help.

Even under SeaBIOS if I select the right partition to boot, it doesn't boot.

Since I've several nodes with many VMs on each node, I don't suspect a file corruption and, few weeks ago, all of them were ok.

Regards,

vmware · Oct 9, 2021

So I've got a Proxmox Backup Server running. Used it to back up a VM (Ubuntu 21.04) and restored to the new PVE 7.0 host.

Still got the same SeaBIOS "boot loop"!

YAGA · Oct 9, 2021

I have exactly the same screenshot during the SeaBIOS "boot loop".

I am "happy" not to be alone but surprised that there are only two of us having this problem.

spirit · Oct 9, 2021

@YAGA
" 2 nodes have been upgraded from PVE 6.x to PVE 7.x and all existing VM with Cloud-Init, originally created with PVE 6.x, no longer work today."

So, no backup restore in this case ?

YAGA · Oct 9, 2021

spirit said:
@YAGA
" 2 nodes have been upgraded from PVE 6.x to PVE 7.x and all existing VM with Cloud-Init, originally created with PVE 6.x, no longer work today."

So, no backup restore in this case ?

I've tried to restore a backup but same story it doesn't boot.

vmware · Oct 10, 2021

YAGA said:
I am "happy" not to be alone but surprised that there are only two of us having this problem.

This also baffles me. My PVE 7.0 is brand new fresh install, and almost immediately run into this issue. Surely this would have been picked up by Proxmox's testers?

YAGA · Oct 10, 2021

vmware said:
This also baffles me. My PVE 7.0 is brand new fresh install, and almost immediately run into this issue. Surely this would have been picked up by Proxmox's testers?

I found an old VM backup from July 2021 with PVE 6.4 which was functional because already tested.

VM Details: debian-10.6.1-20201023-openstack-amd64.qcow2 - QEMU VM - created: 2020/12/13 19:05:36

I made a restore of this backup today on the same server with the latest PVE 7.x community updates and it does not work, I still have the boot loop in SeaBIOS.

Therefore, I'm pretty sure something isn't working anymore either related to a change or regression in PVE.

Edit:

This issue seems to be related to VirtIO SCSI, I've tried with options aio=native or aio=io_uring but without success.
If i select LSI 53C895A , the VM can boot but crash during the boot process

It seems very similar to this post https://forum.proxmox.com/threads/a...e-down-due-not-found-disks.92238/#post-405285

spirit · Oct 10, 2021

I don't known if it could be related, do you have tried to change machine version and downgrade it in options ?

YAGA · Oct 11, 2021

spirit said:
I don't known if it could be related, do you have tried to change machine version and downgrade it in options ?

Hi Spirit,

Yes I do, I've tried i440fx latest, 6.0, 5.2, 5.1 without success.

I've also checked with lvdisplay the status of logical volumes for the each VM and there are all available in read/write access.

Unfortunately, I've no other clues at this time...

vmware · Oct 12, 2021

Do backups created in PVE 7.0 restore OK? @YAGA

YAGA · Oct 22, 2021

vmware said:
Do backups created in PVE 7.0 restore OK? @YAGA

No, unfortunately not, stuck in a boot loop.

[SOLVED] PVE 7.0: All VMs with Cloud-init & SeaBIOS fail during boot process (bootloop/disk not found)

Renowned Member

Renowned Member

Member

Renowned Member

Member

Renowned Member

Member

Renowned Member

Distinguished Member

Renowned Member

Member

Attachments

Renowned Member

Distinguished Member

Renowned Member

Member

Renowned Member

Distinguished Member

Renowned Member

Member

Renowned Member