What could cause "No Bootable Media found"

[Faris Raouf]

Several people have posted about problems with VMs not booting with "no Bootable Media found" errors recently. Yes, we're only taking about 2 or 3 users posting about a problem out of many, many thousands, but it is clearly not an isolated incident with only one user being unlucky and encountering some random issue.

Unfortunately this causes me to worry that it might happen to me. Yes, this is silly/stupid, but it is how I am at the moment. If you have no experience of such a mental tunnel vision then you won't understand. Please just take it as being similar to having a broken arm but invisible to the naked eye - it can be fixed but it takes time to heal.

I'll shortly be making backups of the MBR/partition tables of all my VMs and maybe I'll feel a little better afterwards.

But it seems to me that something must be going on that is causing this to happen to more than one user in recent weeks.

So far I've not seen any absolute commonality. There's a hint that it may occur during high disk I/O, or maybe during backup (which is sort of a higher than usual disk I/O, I suppose), but that's about it.

For those who experienced this issue, I'd appreciate it if you could post some information about your setup, such as: PVE version, Local storage or Ceph? LVM-Thin or something else? Generally high disk i/o, high I/O wait, or not? Did you backup very soon before the incident happened? What sort of hardware are you using?

And finally, do you keep your Proxmox system up to date or do you have older packages?

 

[mcgnomer]

This happened to me and is why I'm here. I am going to create a post on it.
I have an R720
ProxMox 5.4-13

Containers

• DNS - centos minimal
• IpChecker - centos minimal
• gameserver(bad use case but it works amazingly well)
• gitserver(again bad use case but was just playing around)

VMs

• Plex
• OpenMediaVault
• Gameserver
• WindowsVM

At the time the issue happened to me I did two things.
1st. I tried to increment the disk size on my OpenMediaVault by 1 gig
2nd. I tried to passthrough my data raid (since my omv was dead) to my GameServer VM

Both of these things have caused these two VM's to have that error.

As to the i/o, wait - i don't think so. I wasn't actively using the NAS and my Wife was watching Ms. Maisel at the time so i know she wasn't watching plex.

 

[pdontcheff]

I'm moving virtual machines from XenServer to Proxmox. With Proxmox version 6.0-1, I successfully transferred 4 virtual machines with "qm importdisk". However, I have upgraded the system to 6.1-3 and now cannot convert VHD drives to qcow2 format. There are no error messages during the conversion, but the new virtual machine does not start. The error message is "Not a bootable disk". It happens with two different virtual machines that start normally with XenServer. There is probably some problem with "qm importdisk" in the new version of Proxmox.

 

[Faris Raouf]

@pdontcheff I'm not sure the issue is the same as we're discussing (disk images that were absolutely fine suddenly losing MBR/partition table and become unbootable with "not a bootable disk") but maybe there's a connection with what you have experienced and I appreciate your post.

I've been thinking about it, and I wonder if we should try to determine what could and what could not cause the MBR/partition table to be damaged or lost.

** I do not pretend to know or understand the low level ways filesystems work and what writes/does not write to the MBR/partition table ** but ...

It seems to me that NOTHING should write to the MBR or partition table during normal operation of a the VM's OS. Surely they are written once and then they only need to be read, unless the partition is resized. I don't know if the fstrim process could cause damage though, and I have that running weekly within all VMs (with no issues thus far). I only use Centos 7 VMs.

Outside the VM, during a Proxmox backup operation, when a snapshot is in progress I can imagine there is some sort of interaction with all bits in the disk image. I do not know if anything else might touch the disk in some way during normal KVM/Qemu/Proxmox operations.

I can say that I very recently resized a VM's partition by 100GB with no issues. I have rebooted both the VM and the entire Node since then with no issues. And I backup weekly, with no issues. I/O Waits are quite low as I use SSDs for local storage/lvm-thin but generally disk I/O is high with lots and lots of reads and writes going on (web/email/database within the VMs).

And clearly 1000s of proxmox users are also having no problems. But some people are. Enough people to be a worry for me.

 

[pdontcheff]

I tested another variant. Probably, when the virtual disk has been expanded, Citrix's XenCenter tool has a problem exporting to the OVF (VHD) format. I exported a virtual machine to an XVA format (which is a TAR archive), uncompressed and converted image to a RAW format using the xva-img tool. Then I imported it into Proxmox VM with 'qm diskimport' and voila - the virtual machine started.

 

[TheSteveO]

I have had this happen multiple times on multiple VMs. I run all my VMs from qcow2 files on an SSD with ext4. I have them weekly backed up to an NFS share (hosted on unraid).

root@pve:~# pveversion
pve-manager/6.3-6/2184247e (running kernel: 5.4.106-1-pve)

The only thing i've found to fix them is to restore a backup from before they were broken. I haven't tried to correct the filesystems on the broken vms yet though.

Edit: They tend to die on reboots of the Proxmox host, as they otherwise tend to just stay running as i ignore them and let them do their thing.

 

[TheSteveO]

Running an ubuntu live CD and attempting recovery of one of the VMs, i see i have a bad superblock on the disk. i ran mke2fs -n /dev/sda and found a number of other superblocks, running e2fsck -b <number> /dev/sda didn't produce any good results, all superblocks listed ended up failing.

The disk looks un-partitioned and i can't get anything to show any partitions.

The only VMs i don't have problems with are those with the hard disks of scsi type, my sata type virtual hard disks are the ones currently running into issues.