Missing virtual disk after upgrade

jggundin · Apr 21, 2021

Hello

After fully upgrading a node to 6.3 and latest kernel, 5.4.106-1 using ZFS pool for storage, some VMs are not booting due to a missing symlink from /dev/zvol/<poolname>/vm-<id>-disk-0 to /dev/zd<other id>.

There are more /dev/zd* devices than zvol vm disks.

How can I know what zd device corresponds to a virtual machine disk ID lacking the symlink?

I think I could create manually the link or make a dd in order to regenerate the VM or the ZFS pool.

There seems to be some kind of problem with ZFS because zpool scrub <poolname> seems to stay stalled and no scrub progress is shown in zpool list -v
Also, just after booting there are some kernel warnings like " INFO: task zpool:2083 blocked for more than 483 seconds."
Physical access to the device is working properly as I have had a dd of the whole device.

Thank you!
Regards
Javier

rauxon · Apr 21, 2021

That does sound like there is a zfs issue - can you run the following commands and show the output?

Code:

zpool status
zfs list
ls -la /dev/<poolname>/
ps -ef | grep zfs
dmesg

With that info, someone here may be able to spot's what going on. I know I've had to reboot a couple of times to get ZFS to actually do what I want it to. The 'blocked for more than 483 seconds' is indicative of something locking.

jggundin · May 3, 2021

I reinstalled with proxmox 6.3, keeping the ZFS pools and restored the VM cfg files trying to avoid a slow backup restoration.
The problem is persisting. Slow IO after some uptime hours, with the same dmesg message, but less frequently.

Here are the output of the proposed commands:

Code:

root@plumbus:~# zpool status
  pool: fastpool_plumbus
state: ONLINE
config:

        NAME                                               STATE     READ WRITE CKSUM
        fastpool_plumbus                                   ONLINE       0     0     0
          nvme-eui.0000000000000000000cca0c02c08c80-part2  ONLINE       0     0     0

errors: No known data errors

  pool: rpool_plumbus
state: ONLINE
  scan: scrub repaired 0B in 00:21:56 with 0 errors on Sun Apr 11 00:46:00 2021
config:

        NAME                                            STATE     READ WRITE CKSUM
        rpool_plumbus                                   ONLINE       0     0     0
          wwn-0x600304802473f0002376b35a067cb61c-part2  ONLINE       0     0     0

errors: No known data errors

Code:

root@plumbus:~# zfs list
NAME                               USED  AVAIL     REFER  MOUNTPOINT
fastpool_plumbus                  1.49T   281G       96K  /fastpool_plumbus
fastpool_plumbus/base-127-disk-0  7.66G   281G     7.66G  -
fastpool_plumbus/base-404-disk-0  71.0G   281G     71.0G  -
fastpool_plumbus/base-505-disk-0  38.1G   281G     38.1G  -
fastpool_plumbus/vm-102-disk-0    52.1M   281G     38.1G  -
fastpool_plumbus/vm-1118-disk-0   1.37G   281G     1.37G  -
fastpool_plumbus/vm-1118-disk-1   76.4G   281G     75.6G  -
fastpool_plumbus/vm-1118-disk-2     56K   281G       56K  -
fastpool_plumbus/vm-1118-disk-3     56K   281G       56K  -
fastpool_plumbus/vm-1202-disk-0   39.3G   281G     80.0G  -
fastpool_plumbus/vm-1203-disk-0   23.2G   281G     74.3G  -
fastpool_plumbus/vm-1204-disk-0   83.9G   281G     96.8G  -
fastpool_plumbus/vm-1205-disk-0   24.7G   281G     74.2G  -
fastpool_plumbus/vm-1206-disk-0   22.2G   281G     74.0G  -
fastpool_plumbus/vm-1207-disk-0   28.4G   281G     76.4G  -
fastpool_plumbus/vm-1208-disk-0   18.7G   281G     73.1G  -
fastpool_plumbus/vm-1209-disk-0   53.5G   281G     86.1G  -
fastpool_plumbus/vm-121-disk-0    11.9G   281G     16.7G  -
fastpool_plumbus/vm-1211-disk-0   20.1G   281G     72.6G  -
fastpool_plumbus/vm-122-disk-0    7.58G   281G     12.2G  -
fastpool_plumbus/vm-123-disk-0    4.80G   281G     11.2G  -
fastpool_plumbus/vm-124-disk-0    1.28G   281G     8.28G  -
fastpool_plumbus/vm-131-disk-0    41.7G   281G     41.3G  -
fastpool_plumbus/vm-131-disk-1     425M   281G      424M  -
fastpool_plumbus/vm-132-disk-0    23.7G   281G     29.4G  -
fastpool_plumbus/vm-133-disk-0    41.0G   281G     46.4G  -
fastpool_plumbus/vm-134-disk-0    20.4G   281G     26.0G  -
fastpool_plumbus/vm-135-disk-0    32.9G   281G     38.5G  -
fastpool_plumbus/vm-136-disk-0    14.5G   281G     20.2G  -
fastpool_plumbus/vm-137-disk-0    20.8G   281G     26.6G  -
fastpool_plumbus/vm-138-disk-0    11.0G   281G     16.7G  -
fastpool_plumbus/vm-139-disk-0    22.7G   281G     28.5G  -
fastpool_plumbus/vm-140-disk-0    7.63G   281G     14.1G  -
fastpool_plumbus/vm-141-disk-0    17.3G   281G     23.2G  -
fastpool_plumbus/vm-154-disk-0    35.4G   281G     35.2G  -
fastpool_plumbus/vm-210-disk-0     201G   281G      194G  -
fastpool_plumbus/vm-227-disk-0     321G   281G      320G  -
fastpool_plumbus/vm-505-disk-1    7.20G   281G     7.20G  -
fastpool_plumbus/vm-701-disk-0    67.1G   281G     64.8G  -
rpool_plumbus                     2.11T  3.44T       96K  /rpool_plumbus
rpool_plumbus/base-101-disk-0      203G  3.60T     37.9G  -
rpool_plumbus/base-101-disk-1     40.2G  3.47T     7.20G  -
rpool_plumbus/base-1102-disk-0     309G  3.59T      152G  -
rpool_plumbus/base-127-disk-2      164G  3.59T     7.66G  -
rpool_plumbus/vm-101-disk-2       33.0G  3.47T     7.20G  -
rpool_plumbus/vm-101-disk-3        165G  3.57T     37.9G  -
rpool_plumbus/vm-102-disk-0        216K  3.44T     7.20G  -
rpool_plumbus/vm-102-disk-1        127M  3.44T     37.9G  -
rpool_plumbus/vm-103-disk-0        124M  3.44T     37.9G  -
rpool_plumbus/vm-103-disk-1        216K  3.44T     7.20G  -
rpool_plumbus/vm-104-disk-0        141M  3.44T     37.9G  -
rpool_plumbus/vm-104-disk-1        224K  3.44T     7.20G  -
rpool_plumbus/vm-105-disk-0        145M  3.44T     37.9G  -
rpool_plumbus/vm-105-disk-1        216K  3.44T     7.20G  -
rpool_plumbus/vm-106-disk-0          8K  3.44T     37.9G  -
rpool_plumbus/vm-106-disk-1          8K  3.44T     7.20G  -
rpool_plumbus/vm-107-disk-0          8K  3.44T     37.9G  -
rpool_plumbus/vm-107-disk-1          8K  3.44T     7.20G  -
rpool_plumbus/vm-108-disk-0          8K  3.44T     37.9G  -
rpool_plumbus/vm-108-disk-1          8K  3.44T     7.20G  -
rpool_plumbus/vm-109-disk-0          8K  3.44T     7.20G  -
rpool_plumbus/vm-109-disk-1          8K  3.44T     37.9G  -
rpool_plumbus/vm-110-disk-0          8K  3.44T     7.20G  -
rpool_plumbus/vm-110-disk-1          8K  3.44T     37.9G  -
rpool_plumbus/vm-111-disk-0          8K  3.44T     37.9G  -
rpool_plumbus/vm-111-disk-1          8K  3.44T     7.20G  -
rpool_plumbus/vm-112-disk-0        200G  3.60T     35.4G  -
rpool_plumbus/vm-112-disk-1       33.8G  3.47T      815M  -
rpool_plumbus/vm-113-disk-0        208G  3.60T     42.8G  -
rpool_plumbus/vm-113-disk-1       33.7G  3.47T      763M  -
rpool_plumbus/vm-114-disk-0          8K  3.44T     7.20G  -
rpool_plumbus/vm-114-disk-1          8K  3.44T     37.9G  -
rpool_plumbus/vm-127-disk-0         56K  3.44T       56K  -
rpool_plumbus/vm-127-disk-1         56K  3.44T       56K  -
rpool_plumbus/vm-127-disk-3       33.0G  3.47T       56K  -
rpool_plumbus/vm-132-disk-0       66.0G  3.51T     3.01M  -
rpool_plumbus/vm-133-disk-0       66.0G  3.51T     3.01M  -
rpool_plumbus/vm-134-disk-0       66.0G  3.51T     3.02M  -
rpool_plumbus/vm-135-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-136-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-137-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-138-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-139-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-140-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-141-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-225-disk-0        177G  3.57T     42.9G  -
rpool_plumbus/vm-229-disk-0        103G  3.53T     8.05G  -
rpool_plumbus/vm-310-disk-0       3.25G  3.44T      152G  -
rpool_plumbus/vm-311-disk-0       4.85G  3.44T      152G  -

Code:

root@plumbus:~# ls -la /dev/fastpool_plumbus/
total 0
drwxr-xr-x  2 root root  3660 Apr 30 20:09 .
drwxr-xr-x 25 root root 11380 Apr 30 20:12 ..
lrwxrwxrwx  1 root root     8 Apr 30 20:09 base-127-disk-0 -> ../zd160
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part1 -> ../zd160p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part2 -> ../zd160p2
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part3 -> ../zd160p3
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part4 -> ../zd160p4
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part5 -> ../zd160p5
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part6 -> ../zd160p6
lrwxrwxrwx  1 root root     8 Apr 30 20:09 base-404-disk-0 -> ../zd576
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-404-disk-0-part1 -> ../zd576p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-404-disk-0-part2 -> ../zd576p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 base-505-disk-0 -> ../zd208
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part1 -> ../zd208p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part2 -> ../zd208p2
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part3 -> ../zd208p3
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part4 -> ../zd208p4
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part5 -> ../zd208p5
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part6 -> ../zd208p6
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-102-disk-0 -> ../zd304
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part1 -> ../zd304p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part2 -> ../zd304p2
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part3 -> ../zd304p3
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part4 -> ../zd304p4
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part5 -> ../zd304p5
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part6 -> ../zd304p6
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1118-disk-0 -> ../zd592
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1118-disk-1 -> ../zd128
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part1 -> ../zd128p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part2 -> ../zd128p2
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part3 -> ../zd128p3
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part4 -> ../zd128p4
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part5 -> ../zd128p5
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part6 -> ../zd128p6
lrwxrwxrwx  1 root root     7 Apr 30 20:09 vm-1118-disk-2 -> ../zd32
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1118-disk-3 -> ../zd400
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1202-disk-0 -> ../zd512
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1202-disk-0-part1 -> ../zd512p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1202-disk-0-part2 -> ../zd512p2
lrwxrwxrwx  1 root root     7 Apr 30 20:09 vm-1203-disk-0 -> ../zd48
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-1203-disk-0-part1 -> ../zd48p1
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-1203-disk-0-part2 -> ../zd48p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1204-disk-0 -> ../zd416
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1204-disk-0-part1 -> ../zd416p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1204-disk-0-part2 -> ../zd416p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1205-disk-0 -> ../zd288
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1205-disk-0-part1 -> ../zd288p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1205-disk-0-part2 -> ../zd288p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1206-disk-0 -> ../zd336
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1206-disk-0-part1 -> ../zd336p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1206-disk-0-part2 -> ../zd336p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1207-disk-0 -> ../zd144
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1207-disk-0-part1 -> ../zd144p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1207-disk-0-part2 -> ../zd144p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1208-disk-0 -> ../zd272
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1208-disk-0-part1 -> ../zd272p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1208-disk-0-part2 -> ../zd272p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1209-disk-0 -> ../zd432
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1209-disk-0-part1 -> ../zd432p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1209-disk-0-part2 -> ../zd432p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1211-disk-0 -> ../zd112
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1211-disk-0-part1 -> ../zd112p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1211-disk-0-part2 -> ../zd112p2
lrwxrwxrwx  1 root root     7 Apr 30 20:09 vm-121-disk-0 -> ../zd96
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part1 -> ../zd96p1
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part2 -> ../zd96p2
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part3 -> ../zd96p3
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part4 -> ../zd96p4
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part5 -> ../zd96p5
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part6 -> ../zd96p6
...


root@plumbus:~# ls -la /dev/rpool_plumbus/
total 0
drwxr-xr-x  2 root root  2640 Apr 30 20:12 .
drwxr-xr-x 25 root root 11380 Apr 30 20:12 ..
lrwxrwxrwx  1 root root     8 Apr 30 20:12 base-101-disk-0 -> ../zd800
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part1 -> ../zd800p1
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part2 -> ../zd800p2
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part3 -> ../zd800p3
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part4 -> ../zd800p4
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part5 -> ../zd800p5
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part6 -> ../zd800p6
lrwxrwxrwx  1 root root     9 Apr 30 20:12 base-1102-disk-0 -> ../zd1104
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part1 -> ../zd1104p1
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part2 -> ../zd1104p2
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part3 -> ../zd1104p3
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part4 -> ../zd1104p4
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part5 -> ../zd1104p5
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part6 -> ../zd1104p6
lrwxrwxrwx  1 root root     8 Apr 30 20:12 base-127-disk-2 -> ../zd640
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part1 -> ../zd640p1
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part2 -> ../zd640p2
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part3 -> ../zd640p3
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part4 -> ../zd640p4
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part5 -> ../zd640p5
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part6 -> ../zd640p6
lrwxrwxrwx  1 root root     9 Apr 30 20:12 vm-101-disk-2 -> ../zd1008
lrwxrwxrwx  1 root root     8 Apr 30 20:12 vm-101-disk-3 -> ../zd688
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part1 -> ../zd688p1
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part2 -> ../zd688p2
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part3 -> ../zd688p3
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part4 -> ../zd688p4
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part5 -> ../zd688p5
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part6 -> ../zd688p6
lrwxrwxrwx  1 root root     9 Apr 30 20:12 vm-102-disk-0 -> ../zd1072
lrwxrwxrwx  1 root root     8 Apr 30 20:12 vm-102-disk-1 -> ../zd752
...

Nothing on ps -ef|grep zfs

root@plumbus:~# dmesg
https://pastebin.com/ydEA8gCW

Wondering if there is something wrong related with this thread:
https://forum.proxmox.com/threads/w...-pool-itself-when-upgrading-to-zfs-2-0.88413/

Search

Search

Missing virtual disk after upgrade

jggundin

Active Member

rauxon

Member

jggundin

Active Member

We value your privacy