After update and reboot, VM will no longer start

ddt3 · Sunday at 16:39

Today I updated my 3 servers and rebooted. Everything seems to work fine except for one VM: my printserver
It no longer works because the VM (111) running cups will not startup:
TASK ERROR: timeout: no zvol device link for 'vm-111-cloudinit' found after 300 sec.
Other VMs with cloud-init data on the same zpool (OneTB) appear to work fine.

Would really appreciate some help to fix this!

Bash:

root@pve:~# pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.8.12-10-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10
proxmox-kernel-6.8: 6.8.12-10
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.1-1
proxmox-backup-file-restore: 3.4.1-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.3.10
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.3
pve-firewall: 5.1.1
pve-firmware: 3.15-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.2
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2

Bash:

root@pve:~# zfs list
NAME                                     USED  AVAIL  REFER  MOUNTPOINT
FastData                                 181G   268G    96K  /FastData
FastData/vm-100-disk-0                  20.6G   272G  17.2G  -
FastData/vm-130-disk-0                  8.13G   276G   931M  -
FastData/vm-132-disk-0                   152G   282G   139G  -
OneTB                                    844G  55.2G   104K  /OneTB
OneTB/base-5000-disk-0                  3.82G  57.4G  1.55G  -
OneTB/base-6000-disk-0                  2.68G  57.2G   658M  -
OneTB/subvol-104-disk-0                 1.06G  7.23G   793M  /OneTB/subvol-104-disk-0
OneTB/subvol-107-disk-0                 2.83G  5.31G  2.69G  /OneTB/subvol-107-disk-0
OneTB/subvol-107-disk-1                  152K  8.00G    96K  /OneTB/subvol-107-disk-1
OneTB/vm-110-cloudinit                     6M  55.2G    76K  -
OneTB/vm-110-disk-0                      132G  55.2G   132G  -
OneTB/vm-111-cloudinit                     6M  55.2G    72K  -
OneTB/vm-111-disk-0                     53.8G  83.0G  14.1G  -
OneTB/vm-111-state-PaperCutVoorUpgrade  4.56G  58.7G  1.07G  -
OneTB/vm-111-state-Werkt                4.56G  59.1G   620M  -
OneTB/vm-111-state-WorkingVersion       4.56G  58.9G   903M  -
OneTB/vm-113-disk-0                     8.62G  63.3G   550M  -
OneTB/vm-116-cloudinit                     6M  55.2G    76K  -
OneTB/vm-116-disk-0                     31.9G  55.2G  32.6G  -
OneTB/vm-117-disk-0                     8.62G  61.8G  2.01G  -
OneTB/vm-119-disk-0                     10.2G  61.3G  4.06G  -
OneTB/vm-120-cloudinit                     6M  55.2G    76K  -
OneTB/vm-120-disk-0                     79.6G  79.4G  55.3G  -
OneTB/vm-121-disk-0                     8.62G  63.8G  34.7M  -
OneTB/vm-123-disk-0                     8.62G  63.8G    56K  -
OneTB/vm-123-disk-1                     8.62G  63.1G   738M  -
OneTB/vm-130-disk-0                      104G   158G  1.39G  -
OneTB/vm-130-disk-1                      161G   153G  63.3G  -
OneTB/vm-131-cloudinit                     6M  55.2G    72K  -
OneTB/vm-131-disk-0                     44.7G  95.9G  3.72G  -
OneTB/vm-131-state-New                  4.56G  59.5G   289M  -
OneTB/vm-132-cloudinit                     6M  55.2G    72K  -
OneTB/vm-132-disk-0                      155G   138G  72.6G  -
OneTB/vm-5000-cloudinit                    6M  55.2G    76K  -
OneTB/vm-6000-cloudinit                    6M  55.2G    72K  -
Safe-Data                               1.41T  2.10T    96K  /Safe-Data
Safe-Data/vm-100-disk-0                 1.01T  2.14T   988G  -
Safe-Data/vm-120-disk-0                  413G  2.28T   234G  -

Bash:

cat /etc/pve/qemu-server/111.conf
## Print server
### cups.familie-dokter.net.
agent: enabled=1
boot: c
bootdisk: scsi0
cipassword: *******
ciuser: localuser
cores: 2
ide2: OneTB:vm-111-cloudinit,media=cdrom
ipconfig0: ip=dhcp
memory: 2048
meta: creation-qemu=6.2.0,ctime=1656684757
name: cups
net0: virtio=bc:24:11:48:92:5f,bridge=vmbr0
onboot: 1
parent: PaperCutVoorUpgrade
scsi0: OneTB:vm-111-disk-0,size=32972M
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=127a142d-60cb-4945-9312-531e02175898
sshkeys: *****
startup: order=100
usb0: host=04a9:176b,usb3=1
vga: serial0
vmgenid: 034aa725-defe-4e59-84b7-d0ad86bb5655

[PaperCutVoorUpgrade]
#Upgrading papercut
agent: enabled=1
boot: c
bootdisk: scsi0
cipassword: ******
ciuser: localuser
cores: 2
ide2: OneTB:vm-111-cloudinit,media=cdrom
ipconfig0: ip=dhcp
memory: 2048
meta: creation-qemu=6.2.0,ctime=1656684757
name: cups
net0: virtio=bc:24:11:48:92:5f,bridge=vmbr0
onboot: 1
parent: WorkingVersion
runningcpu: kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-i440fx-9.0+pve0
scsi0: OneTB:vm-111-disk-0,size=32972M
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=127a142d-60cb-4945-9312-531e02175898
snaptime: 1744020002
sshkeys: *******
startup: order=100
usb0: host=04a9:176b,usb3=1
vga: serial0
vmgenid: 034aa725-defe-4e59-84b7-d0ad86bb5655
vmstate: OneTB:vm-111-state-PaperCutVoorUpgrade

[Werkt]
agent: enabled=1
boot: c
bootdisk: scsi0
cipassword: *******
ciuser: localuser
cores: 2
ide2: OneTB:vm-111-cloudinit,media=cdrom
ipconfig0: ip=dhcp
memory: 2048
meta: creation-qemu=6.2.0,ctime=1656684757
name: cups
net0: virtio=bc:24:11:48:92:5f,bridge=vmbr0
runningcpu: kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-i440fx-9.0+pve0
scsi0: OneTB:vm-111-disk-0,size=32972M
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=127a142d-60cb-4945-9312-531e02175898
snaptime: 1721980199
sshkeys: *******
usb0: host=1-4,usb3=1
vga: serial0
vmgenid: beeaab41-f2dc-439b-aa02-b3cfdaacb28a
vmstate: OneTB:vm-111-state-Werkt

[WorkingVersion]
#Voor prutsen met PaperCut
agent: enabled=1
boot: c
bootdisk: scsi0
cipassword: *******
ciuser: localuser
cores: 2
ide2: OneTB:vm-111-cloudinit,media=cdrom
ipconfig0: ip=dhcp
memory: 2048
meta: creation-qemu=6.2.0,ctime=1656684757
name: cups
net0: virtio=bc:24:11:48:92:5f,bridge=vmbr0
parent: Werkt
runningcpu: kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-i440fx-9.0+pve0
scsi0: OneTB:vm-111-disk-0,size=32972M
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=127a142d-60cb-4945-9312-531e02175898
snaptime: 1731139154
sshkeys: ****
usb0: host=04a9:176b,usb3=1
vga: serial0
vmgenid: beeaab41-f2dc-439b-aa02-b3cfdaacb28a
vmstate: OneTB:vm-111-state-WorkingVersion

Bash:

root@pve:~# zpool status OneTB
  pool: OneTB
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 01:55:10 with 0 errors on Sun Apr 13 02:19:17 2025
config:

    NAME                               STATE     READ WRITE CKSUM
    OneTB                              ONLINE       0     0     0
     ata-ST1000DM010-2EP102_W9ARLWP5  ONLINE       0     0     0

errors: No known data errors

reinob · Sunday at 18:06

I'm not familiar with cloudinit, but from reading the manual this jumped to my mind:

Code:

# qm cloudinit update <vmid>

This should be harmless (I think) and should (hopefully) update/fix whatever has happened with the link to the cloudinit drive.

I hope, if you test this, that it fixes it rather than breaking it further!

ddt3 · Sunday at 19:06

reinob said:
I'm not familiar with cloudinit, but from reading the manual this jumped to my mind:

Code:

# qm cloudinit update <vmid>

This should be harmless (I think) and should (hopefully) update/fix whatever has happened with the link to the cloudinit drive.

I hope, if you test this, that it fixes it rather than breaking it further!

Thanks for the reply, but I already tried that from the web interface (and now also from the command line), it yields the "same" error:
timeout: no zvol device link for 'vm-111-cloudinit' found after 10 sec.

reinob · Sunday at 19:20

OK, I googled around a bit and it seems this problem is very old (and apparently not solved).

In https://www.reddit.com/r/Proxmox/comments/lodxes/vm_not_starting_after_reboot_no_zvol_device_link/
this was solved by starting/enabling

Code:

# systemctl enable --now zfs-load-key.service

(which runs "zfs load-key -a"), but that looks like only relevant with encryption.

There's also https://forum.proxmox.com/threads/t...-700-disk-0-found-after-300-sec-found.106242/
It would be interesting to see the output of

Code:

# zvol_wait

also exporting the pool with

Code:

# zpool export POOLNAME

Reading your first message again, maybe that zpool status message telling you to run "zpool upgrade" may be relevant?

Good luck

ddt3 · Monday at 13:54

Thank you @reinob your answer solved it (see below)

I have indeed not done a zpool upgrade, for a number of reasons:

Sofar I have not upgrade any of the pools, the message has been there for quite some time
Every description of zpool upgrade mentioned that there all also some risks
The release notes of proxmox do not state a necessity to upgrade
Other cloud-init images on the same pool have no issues, it is just this one

So basically, I am scared to break somerhing

I om not using ZFS for system files (e.g. I am not booting from ZFS) that should make it easier...
But still: could the update solve any issues?

Bash:

root@pve:~# zvol_wait
Testing 33 zvol links
Still waiting on 1 zvol links ...
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
OneTB/vm-111-cloudinit
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
OneTB/vm-111-cloudinit
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
OneTB/vm-111-cloudinit
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
OneTB/vm-111-cloudinit
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
OneTB/vm-111-cloudinit
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
OneTB/vm-111-cloudinit
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
OneTB/vm-111-cloudinit
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
OneTB/vm-111-cloudinit
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
OneTB/vm-111-cloudinit
^C

So zvol_wait behaves as expected: it keeps waiting for OneTB/vm-111-cloudinit
Just performed another zpool scrub OneTB no errors found:

Code:

root@pve:~# zpool status OneTB
  pool: OneTB
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 01:48:17 with 0 errors on Mon Apr 28 11:45:23 2025
config:

    NAME                               STATE     READ WRITE CKSUM
    OneTB                              ONLINE       0     0     0
     ata-ST1000DM010-2EP102_W9ARLWP5  ONLINE       0     0     0

errors: No known data errors

reinob said:
also exporting the pool with

Code:

# zpool export POOLNAME

So I shutdown all the vms that were using this pool and performed a zpool export that did the trick!

Search

Search

After update and reboot, VM will no longer start

ddt3

Member

reinob

New Member

ddt3

Member

reinob

New Member

ddt3

Member

We value your privacy