[SOLVED] TASK ERROR: timeout: no zvol device link for 'vm-700-disk-0' found after 300 sec found.

P

proximoxi

Active Member
Aug 24, 2018
18
0
41
38
edit: sorry, wrong forum :( can some mod move this to proxmox sever?

hi there,

after reboot from Host, 1 VM can't be started anymore.

TASK ERROR: timeout: no zvol device link for 'vm-700-disk-0' found after 300 sec found.

Code: 
# zfs list
NAME                        USED  AVAIL     REFER  MOUNTPOINT
vms_lnx                     179G   720G      104K  /vms_lnx
vms_lnx/vm-700-disk-0      28.7G   720G     28.7G  -

Symlink is missing?

Code: 
# ls -al /dev/zvol/vms_lnx/
lrwxrwxrwx 1 root root   11 Mar  9 21:50 vm-500-disk-0 -> ../../zd352
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-500-disk-0-part1 -> ../../zd352p1
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-500-disk-0-part2 -> ../../zd352p2
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-500-disk-0-part5 -> ../../zd352p5
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-500-disk-0-part6 -> ../../zd352p6
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-500-disk-0-part7 -> ../../zd352p7
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-500-disk-0-part8 -> ../../zd352p8
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-700-disk-0-part1 -> ../../zd256p1
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-700-disk-0-part5 -> ../../zd256p5
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-700-disk-0-part6 -> ../../zd256p6
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-700-disk-0-part8 -> ../../zd256p8
lrwxrwxrwx 1 root root   11 Mar  9 21:50 vm-800-disk-0 -> ../../zd288
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-800-disk-0-part1 -> ../../zd288p1
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-800-disk-0-part2 -> ../../zd288p2
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-800-disk-0-part5 -> ../../zd288p5
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-800-disk-0-part6 -> ../../zd288p6
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-800-disk-0-part7 -> ../../zd288p7
lrwxrwxrwx 1 root root   13 Mar  9 21:50 vm-800-disk-0-part8 -> ../../zd288p8

Code: 
# lsblk
zd256                230:256  0    62G  0 disk
├─zd256p1            230:257  0   6.1G  0 part
├─zd256p2            230:258  0     1K  0 part
├─zd256p5            230:261  0  32.4G  0 part
├─zd256p6            230:262  0     2G  0 part
├─zd256p7            230:263  0   483M  0 part
└─zd256p8            230:264  0    21G  0 part

Syslog

Code: 
Mar  9 21:51:57 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:52:03 proxmox pvedaemon[4989]: <root@pam> starting task UPID:proxmox:00005EA2:0000323B:62291373:qmstart:700:root@pam:
Mar  9 21:52:03 proxmox pvedaemon[24226]: start VM 700: UPID:proxmox:00005EA2:0000323B:62291373:qmstart:700:root@pam:
Mar  9 21:52:27 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:52:57 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:53:28 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:53:58 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:54:28 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:54:58 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:55:28 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:55:58 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:56:28 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:56:58 proxmox zvol_wait[3976]: vms_lnx/vm-700-disk-0
Mar  9 21:57:02 proxmox pvedaemon[24226]: timeout: no zvol device link for 'vm-700-disk-0' found after 300 sec found.
Mar  9 21:57:02 proxmox pvedaemon[4989]: <root@pam> end task UPID:proxmox:00005EA2:0000323B:62291373:qmstart:700:root@pam: timeout: no zvol device link for 'vm-700-disk-0' found after 300 sec found.

can you help me?
thanks a lot
 
Last edited:
please post the output of `pveversion -v` (and make sure you've installed the latest updates!
 
  • Like
Reactions: proximoxi
Code: 
# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-5-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-12
pve-kernel-5.13: 7.1-8
pve-kernel-5.11: 7.0-10
pve-kernel-5.13.19-5-pve: 5.13.19-13
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-1-pve: 5.11.22-2
ceph-fuse: 15.2.13-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-7
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

all updates installed.

thank you
 
Last edited:
Hmm - could you please share the outputs of:
* `journalctl -b` (log since boot -probably best attach it as plain-text file here)
* `zpool status`
* `zvol_wait`

thanks!
 
  • Like
Reactions: proximoxi
Code: 
# zpool status
  pool: vms_lnx
 state: ONLINE
  scan: scrub repaired 0B in 00:24:43 with 0 errors on Sun Feb 13 00:48:44 2022
config:

        NAME                                           STATE     READ WRITE CKSUM
        vms_lnx                                        ONLINE       0     0     0
          ata-Samsung_SSD_860_EVO_1TB_S3Z9NB0M205085Y  ONLINE       0     0     0

errors: No known data errors

Code: 
# zvol_wait
Testing 24 zvol links
Still waiting on 1 zvol links ...
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
vms_lnx/vm-700-disk-0
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
vms_lnx/vm-700-disk-0
Still waiting on 1 zvol links ...
No progress since last loop.
Checking if any zvols were deleted.
Remaining zvols:
vms_lnx/vm-700-disk-0
^C

thank you
 
Hm - not sure how the complete situation is based on the outputs (it seems you've not pasted the complete outputs?)

* but could you try exporting the pool and reimporting it (of course will only work if you don't have other guests running there)
* does this happen every time you reboot?
* could you share the output of `zfs get all vms_lnx/vm-700-disk-0` ?

Do you have any particular non-default settings/configuration on that system/the pool/the VM?
 
  • Like
Reactions: proximoxi
Hi Stoiko

i only removed some privacy mail and users... and some monitoring lines which have nothing todo with this

After exporting the pool i can start the vm again. i rebooted also again and its working now...

Code: 
# zvol_wait
Testing 24 zvol links
All zvol links are now present.

Thank you much !
 
proximoxi said:
After exporting the pool i can start the vm again. i rebooted also again and its working now...
Click to expand...
glad it's working!

Currently I think that it's an issue with zfs (the fix is commited, but not yet packaged) - I assume the issue won't happen again once zfs-2.1.3 has been released:
https://lists.proxmox.com/pipermail/pve-devel/2022-March/052045.html

EDIT: thinking a bit more about it - the fix for a similar issue was already in 2.1.2 - so I'm not 100% sure that the issue will be gone

should you run into this again (after install zfsutils-linux and a pve-kernel containing zfs 2.1.3) - please notify us here (again with journal and outputs of the commands)! - Thanks!
 
Last edited:
  • Like
Reactions: proximoxi
Stoiko Ivanov said:
* but could you try exporting the pool and reimporting it (of course will only work if you don't have other guests running there)
Click to expand...
Thanks you. I've had the same issue. I've just exported the pool without importing. Just one command zpool export POOLNAME and it started.
 
I know this thread is older but somebody referred me here. I had a working VM it was even working but I couldn’t back it up to a PBS like I normally could.
I would run a zfs list -t all and the volume was missing. This is all while the VM was running. Then I shutdown the VM and reboot the node, now the VM won’t boot.

Code: 
Task viewer: VM/CT 201 - Backup

INFO: starting new backup job: vzdump 201 --notes-template '{{guestname}}' --storage csd-txs-01 --node pve03 --mode snapshot --remove 0
INFO: Starting Backup of VM 201 (qemu)
INFO: Backup started at 2023-10-12 10:20:50
INFO: status = running
INFO: VM Name: CSD-PBS-002
INFO: include disk 'scsi0' 'local-zfs:vm-201-disk-0' 100G
ERROR: Backup of VM 201 failed - timeout: no zvol device link for 'vm-201-disk-0' found after 300 sec found.
INFO: Failed at 2023-10-12 10:25:49
INFO: Backup job finished with errors
TASK ERROR: job errors
 
Pls I have a basic question to ask. When I download the os-image I wish to use (Amazon Linux 2 .qcow2), where do I move the image to so it gets attached for the vm to boot from.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back