VM fail to startup at boot

[leonidas_o]

Running on Proxmox Virtual Environment 7.2-11

proxmox-ve: 7.2-1 (running kernel: 5.15.64-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-13
pve-kernel-helper: 7.2-13
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

I can't remember having changed anything, also compared all VM settings in Hardware, Cloud-Init and Options with another VM but can't find the culprit. Both VMs have the "start at boot" option checked, but the first is failing, the second is booting up perfectly fine.

Promox shows:

failed to stat '/mnt/pve/nfs1/images/100/vm-100-cloudinit.qcow2'
TASK ERROR: disk image '/mnt/pve/nfs1/images/100/vm-100-cloudinit.qcow2' already exists

I regenerated the image (Cloud-Init tab -> Regenerate Image), I removed the cloud init drive and re-added it. Nothing helps. I can simply manually start the VM, it will boot up. But If I restart the whole node, it won't boot automatically while the other VM will.

 

[apoc]

Somehow that sounds like a timing issue to me.

Some ideas:
Have you configured a boot order? Is it set to "any"?
Have you tried to flip the boot order around? Does the issue move to the other VM?

 

[leonidas_o]

I have a boot order, but played around with it already and I also removed it (set it to any) but same behaviour unfortunately.

 

[apoc]

Have you changed the start on boot delay value?
Host > System > Options > Start on boot delay
For me the default works well. Perhaps you need a higher value?!

 

[leonidas_o]

I set that delay just on that failing VM, not on the host like you described, but also same error. That's really weird, it just makes no sense to me. It won't startup in boot, but I can start it manually. All the VM's are created from the same template and have the same settings and all of them are using the same nfs storage.

 

[apoc]

I agree with you. If a manual start works well - This is the reason why I think it might be a timing issue.
What settings have you used? To rule out the timing issue I'd use a very high value (5 min) and see if something changes.
Does syslog show anything?

 

[leonidas_o]

mhhh, I once had only the boot order for both VM's set to 2 but not flipping it (stupid me). I entered the boot order back again, but this time I flipped it and the other VM was failing, so yes definitely a timing issue.
So I reverted it, set the initial failing VM back to boot order 1, the other to 2 and "Host > System > Options > Start on boot delay" to 10 seconds. It works now. Awesome, thank you very much @apoc , you had the right feeling.
Very strange error message for such a timing issue.

Quick question, how does the "Host > System > Options > Start on boot delay" behave? So like, when Proxmox boots, it waits 10 seconds and then starts the first VM, afterwards immediately the second VM etc.
OR
does it wait between each VM start 10 seconds?

 

[apoc]

Awesome, thank you very much @apoc , you had the right feeling.

It feels very good to be right Glad I could help

Quick question, how does the "Host > System > Options > Start on boot delay" behave? So like, when Proxmox boots, it waits 10 seconds and then starts the first VM, afterwards immediately the second VM etc.
OR
does it wait between each VM start 10 seconds?

Good question.

This might answer your question - I think it is the second option:
https://forum.proxmox.com/threads/delay-start-of-a-vm.111212/post-479307

/edit: However if that would be the case - it should not work for you - I am confused

HTH

 

[leonidas_o]

I just tried the following:
- removed "Host > System > Options > Start on boot delay"
- set startup delay on the VM1 itself which has startup order 1 to 12 seconds and left the other VM with startup order 2 as it is.

result was same error. I actually would have expected that this should wait for 12 seconds then start VM1 then VM2.
So I again set "Host > System > Options > Start on boot delay" t0 10 seconds, and it works.
when looking into syslog:

Nov 01 21:29:05 mynode1 pvesh[2126]: Starting VM 100
...
Nov 01 21:29:09 mynode1 pvesh[2126]: Starting VM 101

So it doesn't look like that it is waiting every time 10 seconds between each VM start, which I wouldn't want anyway. Long story short, it works now, I will leave it as it is. One more time, thank you very much, I appreciate your help

 

[datschlatscher]

The boot order delays are a little unintuitive but not complicated in essence:

PVE will wait the length of startall-on-boot-delay before starting any VM. After that it will start the VMs according to their boot order, with each succeeding one being delayed by the timeout value of the VM before it.

So if we assume a configuration like:

• startall-delay: 30 seconds
• VM #1, boot order: 1, delay: 10 seconds
• VM #2, order: 2, delay: 5 seconds
• VM #3, order: 3

Then VM #1 will start after a 30 seconds delay, VM #2 after 10 more seconds and finally VM #3 after 5 more seconds.