[SOLVED] VM doesn't start on startup

[j1a2o]

Getting the following error when the VM tries to autostart after reboot:

kvm: -drive file=/dev/zvol/ssd/vm-101-disk-0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap: Could not open '/dev/zvol/ssd/vm-101-disk-0': Unknown error 512
TASK ERROR: start failed: QEMU exited with code 1

After I log into the box, I can usually start the VM manually (after some time).

pveversion -v shows:

proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-15 (running version: 7.4-15/a5d2a31e)
pve-kernel-5.15: 7.4-4
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.5-1-pve: 5.15.5-1
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-4
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

 

[bbgeek17]

Probably the storage is not ready, add a startup delay in VM options to see if that helps. Then examine your journalctl to see where/when storage comes in, and what, if anything, delays it.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[j1a2o]

zfs storage seems to be finished before the QEMU start error. Here are some relevant log entries:

Boot 83a99796caf44610950d42ff42f9ec13
Jun 27 19:12:15 pve kernel: ZFS: Loaded module v2.1.11-pve1, ZFS pool version 5000, ZFS filesystem version 5

Jun 27 19:12:21 pve systemd[1]: Starting Import ZFS pools by cache file...
Jun 27 19:12:21 pve systemd[1]: Finished Import ZFS pools by cache file.
Jun 27 19:12:21 pve systemd[1]: Reached target ZFS pool import target.
Jun 27 19:12:21 pve systemd[1]: Starting Mount ZFS filesystems...
Jun 27 19:12:21 pve systemd[1]: Starting Wait for ZFS Volume (zvol) links in /dev...
Jun 27 19:12:21 pve zvol_wait[2118]: Testing 1 zvol links
Jun 27 19:12:21 pve zvol_wait[2118]: All zvol links are now present.
Jun 27 19:12:21 pve systemd[1]: Finished Wait for ZFS Volume (zvol) links in /dev.
Jun 27 19:12:21 pve systemd[1]: Reached target ZFS volumes are ready.
Jun 27 19:12:21 pve systemd[1]: Finished Mount ZFS filesystems.

Jun 27 19:12:54 pve pvestatd[3657]: zfs error: cannot open 'aux': no such pool
Jun 27 19:12:58 pve pve-guests[4670]: start failed: QEMU exited with code 1

Jun 27 19:22:31 pve systemd[1]: zfs-share.service: Succeeded.
Jun 27 19:22:31 pve systemd[1]: zfs-zed.service: Succeeded.

What's interesting is that it says "Testing 1 zvol links". But when I run zvol_wait after everything has stabilized, it says 3 instead of 1:

# zvol_wait
Testing 3 zvol links
All zvol links are now present.

 

[j1a2o]

I was able to solve this by regenerating the zpool.cache file on all my pools: https://openzfs.github.io/openzfs-docs/Project and Community/FAQ.html#the-etc-zfs-zpool-cache-file