[SOLVED] VMs not working anymore after PVE 7 upgrade

[Spinning_rust]

Hello everyone,

after the upgrade to PVE 7 i encountered several issues.
One of them is that all of my VMs will not boot up anymore. The virtual BIOS does not recognize the qemu drives and claims "no bootable device".

Unfortunately, i don't have any backups of these VMs, as my Proxmox Backup Server can not be attached to 2 out of 3 cluster nodes (I'll make a new thread for that). I'll attach the Proxmox system reports as txt files.

Thanks in advance for any advice and tipps.

 

[fabian]

could you provide a sample VM config and the storage.cfg as well? does a newly created VM boot normally?

 

[Spinning_rust]

could you provide a sample VM config and the storage.cfg as well? does a newly created VM boot normally?

Yes, a newly created VM does work and boot properly.
I attached the config of the new vm (new-vm-cfg.txt) and one of a not booting VM (broken-vm-cfg.txt).
The storage config is in the file (middle-storage-cfg.txt).

 

[fabian]

the new one boots properly == the new one boots from disk after you install an OS? or just it boots from the iso in the "cdrom drive"? can you try adding ",aio=threads" to the old VM config's sata0 entry?

 

[Spinning_rust]

the new one boots properly == the new one boots from disk after you install an OS? or just it boots from the iso in the "cdrom drive"? can you try adding ",aio=threads" to the old VM config's sata0 entry?

Good morning,

I installed a new Debian VM yesterday and the install works fine, just like booting said install. So no problem there.

However, adding ",aio=threads" to the VM config did not help.
Attached is the VM config i case i did something wrong, and also a screenshot of the boot screen error i get in the web UI.

 

[fabian]

no that looks okay. does the VM boot if you do a backup and restore it into a new VMID?

 

[Spinning_rust]

no that looks okay. does the VM boot if you do a backup and restore it into a new VMID?

I migrated this VM to the one node that is able to connect to the PBS and running a backup now. it's excruciatingly slow; so maybe this one has issues as well I'll wait until it either finishes or fails.

 

[Rocha Neto]

no that looks okay. does the VM boot if you do a backup and restore it into a new VMID?

Hello,

I have a similar problem. When upgrading to version 7, vm fails to start. And it stays on this screen forever. Tried to restore a backup with another VMID. And the problem persists.

https://prnt.sc/1j0f7ig

VM was fully functional. I tried to make a backup to a new PBS that I uploaded. And backup was not performed due to version issues in QEMU.

ERROR: PBS backups are not supported by the running QEMU version. Please make sure you've installed the latest version and the VM has been restarted.

So I decided to restart. And now the vm doesn't go up anymore. I thought it was something isolated. Tried to launch another vm that was off. And she doesn't go up either.

Any idea?

 

[fabian]

Hello,

I have a similar problem. When upgrading to version 7, vm fails to start. And it stays on this screen forever. Tried to restore a backup with another VMID. And the problem persists.

https://prnt.sc/1j0f7ig

VM was fully functional. I tried to make a backup to a new PBS that I uploaded. And backup was not performed due to version issues in QEMU.

ERROR: PBS backups are not supported by the running QEMU version. Please make sure you've installed the latest version and the VM has been restarted.

So I decided to restart. And now the vm doesn't go up anymore. I thought it was something isolated. Tried to launch another vm that was off. And she doesn't go up either.

Any idea?

please provide details - pveversion -v, storage config, VM config, any messages from the logs around starting..

 

[Rocha Neto]

pveversion -v

proxmox-ve: 7.0-2 (running kernel: 5.4.34-1-pve)
pve-manager: 7.0-10 (running version: 7.0-10/d2f465d3)
pve-kernel-5.11: 7.0-6
pve-kernel-helper: 7.0-6
pve-kernel-5.4: 6.4-5
pve-kernel-5.11.22-3-pve: 5.11.22-6
pve-kernel-5.4.128-1-pve: 5.4.128-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.2.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-5
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-9
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.7-1
proxmox-backup-file-restore: 2.0.7-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-8
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-12
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

storage.cnf

dir: local
path /var/lib/vz
content vztmpl,iso,backup

lvmthin: Storage01
thinpool StorageSSD
vgname SSD
content rootdir,images

dir: Backup2TB
path /mnt/Backup2TB
content backup
maxfiles 1
shared 0

vm config

agent: 1
bootdisk: virtio0
cipassword:
ciuser: root
cores: 1
cpu: host
cpulimit: 1
cpuunits: 1024
ide0: Storage01:vm-904-cloudinit,media=cdrom,size=4M
ide2: none,media=cdrom
ipconfig0: ip=,gw=
kvm: 1
memory: 1024
name: atos
net0: virtio=CA:A0:F6:2E:16:9A,bridge=vmbr0
numa: 0
onboot: 0
ostype: l26
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=30467227-d852-4f78-9353-068709e54f3a
sockets: 1
virtio0: Storage01:vm-904-disk-0,cache=writeback,discard=on,size=50G

logs

Aug 4 09:37:18 mundau pvedaemon[29203]: start VM 904: UPID:mundau:00007213:E2B0B69F:610A89FE:qmstart:904:root@pam:
Aug 4 09:37:18 mundau pvedaemon[26494]: <root@pam> starting task UPID:mundau:00007213:E2B0B69F:610A89FE:qmstart:904:root@pam:
Aug 4 09:37:18 mundau systemd[1]: Started 904.scope.
Aug 4 09:37:18 mundau systemd-udevd[29210]: Using default interface naming scheme 'v247'.
Aug 4 09:37:18 mundau systemd-udevd[29210]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 4 09:37:19 mundau kernel: [38032972.226667] device tap904i0 entered promiscuous mode
Aug 4 09:37:19 mundau kernel: [38032972.244654] vmbr0: port 28(tap904i0) entered blocking state
Aug 4 09:37:19 mundau kernel: [38032972.244657] vmbr0: port 28(tap904i0) entered disabled state
Aug 4 09:37:19 mundau kernel: [38032972.244791] vmbr0: port 28(tap904i0) entered blocking state
Aug 4 09:37:19 mundau kernel: [38032972.244793] vmbr0: port 28(tap904i0) entered forwarding state
Aug 4 09:37:19 mundau pvedaemon[26494]: <root@pam> end task UPID:mundau:00007213:E2B0B69F:610A89FE:qmstart:904:root@pam: OK

more logs

Aug 4 09:39:09 mundau pvestatd[1384]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:09 mundau pvestatd[1384]: VM 107 qmp command failed - VM 107 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:09 mundau pvestatd[1384]: VM 109 qmp command failed - VM 109 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:09 mundau pvestatd[1384]: VM 103 qmp command failed - VM 103 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:09 mundau pvestatd[1384]: VM 120 qmp command failed - VM 120 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:09 mundau pvestatd[1384]: VM 800 qmp command failed - VM 800 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 113 qmp command failed - VM 113 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 119 qmp command failed - VM 119 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 102 qmp command failed - VM 102 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 804 qmp command failed - VM 804 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 121 qmp command failed - VM 121 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 801 qmp command failed - VM 801 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 802 qmp command failed - VM 802 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 106 qmp command failed - VM 106 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 110 qmp command failed - VM 110 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 800 qmp command failed - VM 800 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 120 qmp command failed - VM 120 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 103 qmp command failed - VM 103 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 109 qmp command failed - VM 109 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 107 qmp command failed - VM 107 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 900 qmp command failed - VM 900 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 111 qmp command failed - VM 111 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 116 qmp command failed - VM 116 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 123 qmp command failed - VM 123 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 114 qmp command failed - VM 114 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:19 mundau pvestatd[1384]: VM 803 qmp command failed - VM 803 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 804 qmp command failed - VM 804 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 102 qmp command failed - VM 102 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 113 qmp command failed - VM 113 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 119 qmp command failed - VM 119 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 801 qmp command failed - VM 801 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 121 qmp command failed - VM 121 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 802 qmp command failed - VM 802 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 110 qmp command failed - VM 110 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 106 qmp command failed - VM 106 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 120 qmp command failed - VM 120 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 800 qmp command failed - VM 800 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found
Aug 4 09:39:29 mundau pvestatd[1384]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - The command query-proxmox-support has not been found

 

[fabian]

you are still running the PVE 6 kernel.. did you reboot after the upgrade?

 

[Rocha Neto]

you are still running the PVE 6 kernel.. did you reboot after the upgrade?

I just realized that. Reboot cannot be done after upgrade.

I will perform as soon as possible.

 

[Spinning_rust]

no that looks okay. does the VM boot if you do a backup and restore it into a new VMID?

Good morning again,

I started the backup of the VM towards the PBS yesterday and for some reason, the read speed is at maximum 4MB/s; mostly around a few hundred KB/s. The storage is an external iSCSI attached HP MSA 2040 disk shelf but all other disk intensive tasks like installing a new VM have a very normal speed. After around 16h and only 26% progress I aborted the backup progress.

Do you have any other ideas?

 

[Spinning_rust]

Good morning again,

I started the backup of the VM towards the PBS yesterday and for some reason, the read speed is at maximum 4MB/s; mostly around a few hundred KB/s. The storage is an external iSCSI attached HP MSA 2040 disk shelf but all other disk intensive tasks like installing a new VM have a very normal speed. After around 16h and only 26% progress I aborted the backup progress.

Do you have any other ideas?

Update: after some external storage and nework tweaking i could finally make a backup. Restoring a new vm disk with this backup resulted in another "no bootable Device found" message in the Proxmox bios on VM startup.

 

[fabian]

okay. and if you boot a live-cd / iso and look at the disk (e.g., with gdisk) - does the partition table look correct? does it print any warnings?

the error message you get is because seabios (the VM bios used by qemu) does not find an mbr where it expects one to be..

 

[Spinning_rust]

okay. and if you boot a live-cd / iso and look at the disk (e.g., with gdisk) - does the partition table look correct? does it print any warnings?

the error message you get is because seabios (the VM bios used by qemu) does not find an mbr where it expects one to be..

i live-booted into a debian shell from the netinstall iso and according to fdisk (sgdisk wasnt on board), there are no recognized partitions on this disk

Should i try again with the "proper" debian live image and sgdisk?

 

[fabian]

no. if you shutdown the VM and try the same from the host with the LV /dev/mapper/... does it show the partitions?

 

[Spinning_rust]

no. if you shutdown the VM and try the same from the host with the LV /dev/mapper/... does it show the partitions?

weirdly enough, the one VM i used all the time (VMID 100) does not show up here (see dev-mapper.txt).
But when I tried to list the partitions for VMID 106 (Ubuntu VM with the same symptoms as described throughout this thread, just on another node i also upgraded from 6 --> 7) i get the output from the attached file "dev-mapper-106.txt".

So, same as in the VM itself

 

[fabian]

that is quite strange - it looks like all the pre-existing disks lost their partition tables (and potentially their full content?), but newly allocated volumes work fine. were there any changes on your iscsi storage that coincided with the upgrade? PVE obviously does not touch the volume's content except when asked to copy/delete volumes, and then it definitely doesn't remove the partition table.. could you post the output of 'lvs'?

 

[Spinning_rust]

that is quite strange - it looks like all the pre-existing disks lost their partition tables (and potentially their full content?), but newly allocated volumes work fine. were there any changes on your iscsi storage that coincided with the upgrade? PVE obviously does not touch the volume's content except when asked to copy/delete volumes, and then it definitely doesn't remove the partition table.. could you post the output of 'lvs'?

No, there were not any changes to the storage...
Whats also strange, a friend has the exact same issues with PVE community edition and local storage after a Proxmox 6 --> 7 in place upgrade. So my setup here is is the second one i see with this behavior

root@top:~# lvs
  LV            VG  Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vm-100-disk-0 PLS -wi-------  128.00g                                                   
  vm-101-disk-1 PLS -wi-a-----    8.00g                                                   
  vm-102-disk-0 PLS -wi-------   32.00g                                                   
  vm-103-disk-0 PLS -wi-a-----  128.00g                                                   
  vm-103-disk-1 PLS -wi-a-----    4.00m                                                   
  vm-104-disk-0 PLS -wi-a-----   32.00g                                                   
  vm-105-disk-0 PLS -wi-a-----   16.00g                                                   
  vm-106-disk-0 PLS -wi-ao---- 1000.00g                                                   
  vm-107-disk-0 PLS -wi-ao----   32.00g                                                   
  vm-108-disk-0 PLS -wi-------   64.00g                                                   
  vm-109-disk-0 PLS -wi-a-----   32.00g                                                   
  vm-110-disk-0 PLS -wi-a-----   32.00g                                                   
  vm-110-disk-1 PLS -wi-a-----  128.00g                                                   
  vm-110-disk-2 PLS -wi-a-----   32.00g                                                   
  vm-110-disk-3 PLS -wi-a-----  128.00g                                                   
  vm-110-disk-4 PLS -wi-a-----   32.00g                                                   
  vm-111-disk-0 PLS -wi-------   32.00g                                                   
  vm-111-disk-1 PLS -wi-------   32.00g