VM No bootable device after reboot

[fiona]

Can VM 9000 boot fine? Or does that one also fail? The filesystem is there and so is the boot label, but maybe it doesn't contain anything actually bootable.

 

[L.O.S]

9000 boot fine (it is the original VM) the clone 4052 doesn't

 

[fiona]

root@saori:~# qemu-img compare /dev/zvol/FileZilla/vm-4052-disk-0 /dev/zvol/FileZilla2/vm-9000-disk-0
Warning: Image size mismatch!
Images are identical.

But except for the size mismatch, the images are identical. I.e. man qemu-img says:

By default, images with different size are considered identical if the larger image contains only unallocated and/or zeroed sectors in the area after the end of the other image.

I'm a bit out of ideas honestly. Did you change anything in the configuration after cloning? Does it work if you clone to the same storage as the original?

 

[L.O.S]

no I didn't change anything. and cloning to the original drive makes no difference.
the simple acts of cloning make it not work.

I have the same type of issue restoring from backups, or even a simple reboot of a working VM causing that VM not to boot.

it lools like the boot partitions of the image gets randomly corrupted. but I see no error messages or logs.

 

[nghinv235]

vm not working. no bootable device

 

[emunt6]

Hi!

Maybe ZFS discard/trim problem?
Other option, change the VM disk options: "cache=none,aio=native" and clone again.

qm config
scsi0: FileZilla2:vm-1003-disk-0,size=44544M

Are you sure this is the right syntax?
AFK the "size=44544M" param is valid for image-files, you using block-device directly.

Possible:

scsi0: /dev/zvol/FileZilla2/vm-1003-disk-0,cache=none,aio=native

 

[fiona]

Hi,

qm config
scsi0: FileZilla2:vm-1003-disk-0,size=44544M

Are you sure this is the right syntax?
AFK the "size=44544M" param is valid for image-files, you using block-device directly.

The size parameter in the configuration is informational only and used for all Proxmox VE managed volumes.

Possible:

scsi0: /dev/zvol/FileZilla2/vm-1003-disk-0,cache=none,aio=native

It's better to use a Proxmox VE managed volume rather than pass-through. E.g. you won't be able to live migrate like this.

EDIT: for completeness, using the managed volume will resolve to the actual path for the QEMU commandline, so there really is no advantage for using passthrough here. E.g.:

root@pve8a1 ~ # qm config 117 | grep vm-117-disk-1        
scsi2: zfs:vm-117-disk-1,discard=on,iothread=1,size=1G
root@pve8a1 ~ # qm showcmd 117 --pretty | grep vm-117-disk-1
  -drive 'file=/dev/zvol/zfs/vm-117-disk-1,if=none,id=drive-scsi2,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \

 

[fiona]

Hi,

vm not working. no bootable device

please share the output of qm config <ID> replacing <ID> with the actual ID of your VM and the output of pveversion -v. Make sure the boot device is selected in the Boot Order setting in the Options tab for the VM.

 

[emunt6]

root@pve8a1 ~ # qm config 117 | grep vm-117-disk-1
scsi2: zfs:vm-117-disk-1,discard=on,iothread=1,size=1G
root@pve8a1 ~ # qm showcmd 117 --pretty | grep vm-117-disk-1
-drive 'file=/dev/zvol/zfs/vm-117-disk-1,if=none,id=drive-scsi2,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
[/CODE]

AFK this parameters:

discard=on

aio=io_uring

should not be used on ZFS volume, causes data corruption.

Which PVE version uses this settings by default?

 

[UdoB]

should not be used on ZFS volume, causes data corruption.

References?

io_uring is the default on my systems (PVE 8.1.4, current). And because I have ZFS on NVMe I do enable the "Discard:"-checkbox per virtual harddisk.

Risky? What to do instead?

 

[fiona]

AFK this parameters:
discard=on
aio=io_uring
should not be used on ZFS volume, causes data corruption.

Never heard of that. If it would be a general issue, a lot of people would complain. Can you share any details?

Which PVE version uses this settings by default?

discard=on is not a default, this is just an example I posted.

 

[L.O.S]

still having the issue. so finally decided to reset everything and go for a fresh install.

the backups on the fresh install are all faillig the same way.
also, creating a new cloud init vm on the new system fail on the same way.

I found that creating a vm from a .img was failing. but, if i rename the img to qcow2 the same process works? any idea?

 

[L.O.S]

the new install is even worst than the previous system. resizing a disk vm cause to through mdadm errors and then, revert to the no bootable device issue.

 

[BoY]

I have the same problem. All my new installed VMs can't boot, showing that "not a bootable disk" and "no bootable device"

 

[fiona]

Hi,

I have the same problem. All my new installed VMs can't boot, showing that "not a bootable disk" and "no bootable device"

please share the VM configuration qm config <ID> replacing <ID> with the ID of your VM and output of pveversion -v as well as the output of the following (if it's qcow2, it needs to be different commands!):

fdisk -l /path/to/your/device
lsblk -o NAME,FSTYPE /path/to/your/device
wipefs /path/to/your/device

Use pvesm path storage:vm-XYZ-disk-N using your actual volume ID to get the path.

 

[emunt6]

still having the issue. so finally decided to reset everything and go for a fresh install.

the backups on the fresh install are all faillig the same way.
also, creating a new cloud init vm on the new system fail on the same way.

I found that creating a vm from a .img was failing. but, if i rename the img to qcow2 the same process works? any idea?

Use always:

discard=off
aio=native
cache=none
scsihw=virtio-scsi-pci

Hardcode to every VM's config, Proxmox guys loves to change these values in every release.

 

[BoY]

Hi,

please share the VM configuration qm config <ID> replacing <ID> with the ID of your VM and output of pveversion -v as well as the output of the following (if it's qcow2, it needs to be different commands!):

fdisk -l /path/to/your/device
lsblk -o NAME,FSTYPE /path/to/your/device
wipefs /path/to/your/device

Use pvesm path storage:vm-XYZ-disk-N using your actual volume ID to get the path.

I have solved the problem. I use the importdisk in terminal instead of using the web UI "create a vm" directly. Although I don't understand what's the difference of them, and what happened.

 

[fiona]

@L.O.S it's very likely that you were hit by the following ZFS issue: https://github.com/openzfs/zfs/issues/15904 and the trigger are partition numbers >= 16 in the virtual images.

root@saori:~# lsblk -o NAME,FSTYPE /dev/zvol/FileZilla2/vm-9000-disk-0
NAME     FSTYPE
zd400p16 ext4

Here it can be seen that the device link actually points to the partition zd400p16 rather than zd400.

Bugzilla entry on our side: https://bugzilla.proxmox.com/show_bug.cgi?id=5288