[SOLVED] PVE7 debian10 OpenStack Cloudimage fails booting (but does not in PVE6).

matiaspecchia

Member
Jul 26, 2013
25
0
21
Hello everyone,

I was trying PVE 7 with debian10 OpenStack (kvm) cloud images and I found a problem.
If the VM has serial terminal as Display the system does not boot (I don't understand if after bootloader or during bootloader) and goes in an infinite restart-loop.

For triggering the error
The weird about this is that using the same virtual machine, with the same configuration in a pve6 node (6.4-13) makes the VM booting normally. Generating a backup and importing in the other node is going to be enough.

Since I have the same VM in PVE6 and PVE7 in the same experimental hardware I have been doing some comparisons.

I made some investigation

Serial console device
  • if I change the monitor to VGA the VM is going to boot.
  • If I logged in the VM I add
    Code:
    GRUB_TERMINAL=console
    to /etc/default/grub and the then I do
    Code:
    sudo update-grub
    . I may use serial terminal as display.
Hardware diferences from VM perspective

if I change
Code:
machine: pc-i440fx-5.2
in the VM definition the problem will persists. But there I can't find hardware diferences in the VMs hosted in PVE6 or PVE7. I am using lshw, dmidecode and lscpi.

KVM invocations on host

I compared the KVM invocations form PVE6 and PVE7 and I don't find special changes related with serial port (shell variables for insignificant info as MAC address):
Diff:
 /usr/bin/kvm \  -id ${VM_ID} \
  -name test \
  -no-shutdown \
- -chardev socket,id=qmp,path=/var/run/qemu-server/${VM_ID}.qmp,server,nowait \
+ -chardev socket,id=qmp,path=/var/run/qemu-server/${VM_ID}.qmp,server=on,wait=off \
  -mon chardev=qmp,mode=control \
  -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 \
  -mon chardev=qmp-event,mode=control \
@@ -27,14 +27,14 @@ INIT_NAME="${INIT_NAME}:-iqn.1993-08.org.debian:01:ca181c45bf2a"
  -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f \
  -device vmgenid,guid=${GU_ID} \
  -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 \
- -chardev socket,id=serial0,path=/var/run/qemu-server/${VM_ID}.serial0,server,nowait \
+ -chardev socket,id=serial0,path=/var/run/qemu-server/${VM_ID}.serial0,server=on,wait=off \
  -device isa-serial,chardev=serial0 \
  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 \
  -iscsi initiator-name=${INIT_NAME} \
- -drive file=/dev/zvol/rpool/data/vm-${VM_ID}-cloudinit,if=none,id=drive-ide2,media=cdrom,aio=threads \
+ -drive file=/dev/zvol/rpool/data/vm-${VM_ID}-cloudinit,if=none,id=drive-ide2,media=cdrom,aio=io_uring \
  -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2 \
  -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 \
- -drive file=/dev/zvol/rpool/data/vm-${VM_ID}-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on \
+ -drive file=/dev/zvol/rpool/data/vm-${VM_ID}-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on \
  -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=101 \
  -netdev type=tap,id=net0,ifname=tap${VM_ID}i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on \
  -device virtio-net-pci,mac=${MAC_ADDR},netdev=net0,bus=pci.0,addr=0x12,id=net0

PVE6 Fresh install in the hardware

Additionally, I made a PVE6 fresh install in the hardware which was running pve7, in order to discard any cause related with the hardware. The VM startup was working as expected. So I understand the problem is related with PVE7.

VM setup

Code:
#TEST vm
boot: c
bootdisk: scsi0
cipassword: <REDACTED>
ciuser: debian
ide2: local:100/vm-100-cloudinit.qcow2,media=cdrom,size=4M
ipconfig0: ip=dhcp
memory: 512
name: test
net0: virtio=<REDACTED>,bridge=vmbr0
scsi0: local-zfs:vm-100-disk-0,size=2G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=<REDACTED>
sshkeys: <REDACTED>
vga: serial0
vmgenid: <REDACTED>

Feedback: next steps

Do you consider I may try some particular action?

I understand the problem is related on the way KVM announces the serial console but I don't know how to get more information.
On the other hand the same VM in a pve6 node works normally.
 

matiaspecchia

Member
Jul 26, 2013
25
0
21
here the output of pveversion:

Code:
# pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve)
pve-manager: 7.0-13 (running version: 7.0-13/7aa7e488)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-10
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
openvswitch-switch: 2.15.0+ds1-2
proxmox-backup-client: 2.0.11-1
proxmox-backup-file-restore: 2.0.11-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20210831-1
pve-firewall: 4.2-4
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-16
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1
 

matiaspecchia

Member
Jul 26, 2013
25
0
21
Could you try booting it with VGA on Default and configuring GRUB to use a serial terminal?
See the following Ubuntu documentation on how to do that: https://help.ubuntu.com/community/SerialConsoleHowto
Hello Mira,
Thanks for your response.

Yes, I can, but this does not explain why the same VM setup works fine in PVE6 and does not in PVE7.

I tried comparing the elements I said but this is not enough.
I'd prefer touch nothing in the VM setup migrating from one host to a newer one with pve7.
 

mira

Proxmox Staff Member
Staff member
Aug 1, 2018
1,619
154
68
This might be related to a change in QEMU. Hard to say with all the changes, but PVE 7 uses QEMU 6.0/6.1 while PVE 6 is still on 5.2.

I've tried to reproduce this with the latest Ubuntu image a week ago, but it worked. So this may be an issue with wrong default configuration after all.
 
Last edited:

matiaspecchia

Member
Jul 26, 2013
25
0
21
I've tried to reproduce this with the latest Ubuntu image a week ago, but it worked. So this may be an issue with wrong default configuration after all.
The subject of the Thread is "PVE7 debian10 OpenStack Cloudimage fails booting (but does not in PVE6)."
in the first message I wrote:
For triggering the error
If you want to trigger the error use the official debian10 image for OpenStack.
I left the checksum in order to help anyone to reproduce it.

Thanks,
kind regards,
 

mira

Proxmox Staff Member
Staff member
Aug 1, 2018
1,619
154
68
Well, tried the OpenStack image as well as the genericcloud one.

The genericcloud one contains the required settings to support a serial socket and is designed for use with QEMU.

The OpenStack image wouldn't even finish boot without serial socket configuration. Could it be there's something wrong with that one?


Ubuntu images are the ones that are primarily tested, and the documentation contains an example for an Ubuntu cloud image. Those always work out of the box with a serial socket. If yours doesn't, then use default graphics and NoVNC instead.
 

matiaspecchia

Member
Jul 26, 2013
25
0
21
Well, tried the OpenStack image as well as the genericcloud one.
I don't know what images you mean.

Could you give us the link to these images (and checksums in order to avoid any misunderstanding) ?

In case you want to check what I said you have the link for downloading and the file chechksum.
 
Last edited:

matiaspecchia

Member
Jul 26, 2013
25
0
21
Hello Mira,
Now I understand:
I checked with the images generic and genericcloud of Buster from the link you provided and from directory latest which at this moment refers to october 2021:
- debian-10-generic-amd64-20211011-792.qcow2 (sha512: f3dac13104b4e28eb62c46cbbb1e30fc9c792834500f9101d477c19c258c31ff04850933ee0b3e63236eca38c854447d95a0ab45163c4a3fccf05f9dd95601a5 )
- debian-10-genericcloud-amd64-20211011-792.qcow2 (sha512: a57002b85b472d732adb271ab556bc267c9694f2d6c017724f5775602fe58110cdd9daf2d30b7f92e6e844e26d95ec34848e0146b201e218b83ce7c8a24ac00a ).

As far I understand the information provided by Debian web generates missunderstanding:
while at Debian Offitial Cloud Images in the supported images there is is a link to the place I was using in this place there is the tittle "(Old) Debian Official Cloud Images for OpenStack".

So the images we should use are are from https://cloud.debian.org/images/cloud/${release}

I don't understand what presents qemu in pve6 and pve7 tha makes useless the old cloud images (I shared here what I searched lspci,dmidecode, etc for each case), anyway I consider the thread as solved.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!