Missing virtual disk after upgrade

J

jggundin

Active Member
Aug 7, 2019
6
0
41
43
Hello

After fully upgrading a node to 6.3 and latest kernel, 5.4.106-1 using ZFS pool for storage, some VMs are not booting due to a missing symlink from /dev/zvol/<poolname>/vm-<id>-disk-0 to /dev/zd<other id>.

There are more /dev/zd* devices than zvol vm disks.

How can I know what zd device corresponds to a virtual machine disk ID lacking the symlink?

I think I could create manually the link or make a dd in order to regenerate the VM or the ZFS pool.

There seems to be some kind of problem with ZFS because zpool scrub <poolname> seems to stay stalled and no scrub progress is shown in zpool list -v
Also, just after booting there are some kernel warnings like " INFO: task zpool:2083 blocked for more than 483 seconds."
Physical access to the device is working properly as I have had a dd of the whole device.

Thank you!
Regards
Javier
 
That does sound like there is a zfs issue - can you run the following commands and show the output?
Code: 
zpool status
zfs list
ls -la /dev/<poolname>/
ps -ef | grep zfs
dmesg
With that info, someone here may be able to spot's what going on. I know I've had to reboot a couple of times to get ZFS to actually do what I want it to. The 'blocked for more than 483 seconds' is indicative of something locking.
 
I reinstalled with proxmox 6.3, keeping the ZFS pools and restored the VM cfg files trying to avoid a slow backup restoration.
The problem is persisting. Slow IO after some uptime hours, with the same dmesg message, but less frequently.

Here are the output of the proposed commands:
Code: 
root@plumbus:~# zpool status
  pool: fastpool_plumbus
state: ONLINE
config:

        NAME                                               STATE     READ WRITE CKSUM
        fastpool_plumbus                                   ONLINE       0     0     0
          nvme-eui.0000000000000000000cca0c02c08c80-part2  ONLINE       0     0     0

errors: No known data errors

  pool: rpool_plumbus
state: ONLINE
  scan: scrub repaired 0B in 00:21:56 with 0 errors on Sun Apr 11 00:46:00 2021
config:

        NAME                                            STATE     READ WRITE CKSUM
        rpool_plumbus                                   ONLINE       0     0     0
          wwn-0x600304802473f0002376b35a067cb61c-part2  ONLINE       0     0     0

errors: No known data errors

Code: 
root@plumbus:~# zfs list
NAME                               USED  AVAIL     REFER  MOUNTPOINT
fastpool_plumbus                  1.49T   281G       96K  /fastpool_plumbus
fastpool_plumbus/base-127-disk-0  7.66G   281G     7.66G  -
fastpool_plumbus/base-404-disk-0  71.0G   281G     71.0G  -
fastpool_plumbus/base-505-disk-0  38.1G   281G     38.1G  -
fastpool_plumbus/vm-102-disk-0    52.1M   281G     38.1G  -
fastpool_plumbus/vm-1118-disk-0   1.37G   281G     1.37G  -
fastpool_plumbus/vm-1118-disk-1   76.4G   281G     75.6G  -
fastpool_plumbus/vm-1118-disk-2     56K   281G       56K  -
fastpool_plumbus/vm-1118-disk-3     56K   281G       56K  -
fastpool_plumbus/vm-1202-disk-0   39.3G   281G     80.0G  -
fastpool_plumbus/vm-1203-disk-0   23.2G   281G     74.3G  -
fastpool_plumbus/vm-1204-disk-0   83.9G   281G     96.8G  -
fastpool_plumbus/vm-1205-disk-0   24.7G   281G     74.2G  -
fastpool_plumbus/vm-1206-disk-0   22.2G   281G     74.0G  -
fastpool_plumbus/vm-1207-disk-0   28.4G   281G     76.4G  -
fastpool_plumbus/vm-1208-disk-0   18.7G   281G     73.1G  -
fastpool_plumbus/vm-1209-disk-0   53.5G   281G     86.1G  -
fastpool_plumbus/vm-121-disk-0    11.9G   281G     16.7G  -
fastpool_plumbus/vm-1211-disk-0   20.1G   281G     72.6G  -
fastpool_plumbus/vm-122-disk-0    7.58G   281G     12.2G  -
fastpool_plumbus/vm-123-disk-0    4.80G   281G     11.2G  -
fastpool_plumbus/vm-124-disk-0    1.28G   281G     8.28G  -
fastpool_plumbus/vm-131-disk-0    41.7G   281G     41.3G  -
fastpool_plumbus/vm-131-disk-1     425M   281G      424M  -
fastpool_plumbus/vm-132-disk-0    23.7G   281G     29.4G  -
fastpool_plumbus/vm-133-disk-0    41.0G   281G     46.4G  -
fastpool_plumbus/vm-134-disk-0    20.4G   281G     26.0G  -
fastpool_plumbus/vm-135-disk-0    32.9G   281G     38.5G  -
fastpool_plumbus/vm-136-disk-0    14.5G   281G     20.2G  -
fastpool_plumbus/vm-137-disk-0    20.8G   281G     26.6G  -
fastpool_plumbus/vm-138-disk-0    11.0G   281G     16.7G  -
fastpool_plumbus/vm-139-disk-0    22.7G   281G     28.5G  -
fastpool_plumbus/vm-140-disk-0    7.63G   281G     14.1G  -
fastpool_plumbus/vm-141-disk-0    17.3G   281G     23.2G  -
fastpool_plumbus/vm-154-disk-0    35.4G   281G     35.2G  -
fastpool_plumbus/vm-210-disk-0     201G   281G      194G  -
fastpool_plumbus/vm-227-disk-0     321G   281G      320G  -
fastpool_plumbus/vm-505-disk-1    7.20G   281G     7.20G  -
fastpool_plumbus/vm-701-disk-0    67.1G   281G     64.8G  -
rpool_plumbus                     2.11T  3.44T       96K  /rpool_plumbus
rpool_plumbus/base-101-disk-0      203G  3.60T     37.9G  -
rpool_plumbus/base-101-disk-1     40.2G  3.47T     7.20G  -
rpool_plumbus/base-1102-disk-0     309G  3.59T      152G  -
rpool_plumbus/base-127-disk-2      164G  3.59T     7.66G  -
rpool_plumbus/vm-101-disk-2       33.0G  3.47T     7.20G  -
rpool_plumbus/vm-101-disk-3        165G  3.57T     37.9G  -
rpool_plumbus/vm-102-disk-0        216K  3.44T     7.20G  -
rpool_plumbus/vm-102-disk-1        127M  3.44T     37.9G  -
rpool_plumbus/vm-103-disk-0        124M  3.44T     37.9G  -
rpool_plumbus/vm-103-disk-1        216K  3.44T     7.20G  -
rpool_plumbus/vm-104-disk-0        141M  3.44T     37.9G  -
rpool_plumbus/vm-104-disk-1        224K  3.44T     7.20G  -
rpool_plumbus/vm-105-disk-0        145M  3.44T     37.9G  -
rpool_plumbus/vm-105-disk-1        216K  3.44T     7.20G  -
rpool_plumbus/vm-106-disk-0          8K  3.44T     37.9G  -
rpool_plumbus/vm-106-disk-1          8K  3.44T     7.20G  -
rpool_plumbus/vm-107-disk-0          8K  3.44T     37.9G  -
rpool_plumbus/vm-107-disk-1          8K  3.44T     7.20G  -
rpool_plumbus/vm-108-disk-0          8K  3.44T     37.9G  -
rpool_plumbus/vm-108-disk-1          8K  3.44T     7.20G  -
rpool_plumbus/vm-109-disk-0          8K  3.44T     7.20G  -
rpool_plumbus/vm-109-disk-1          8K  3.44T     37.9G  -
rpool_plumbus/vm-110-disk-0          8K  3.44T     7.20G  -
rpool_plumbus/vm-110-disk-1          8K  3.44T     37.9G  -
rpool_plumbus/vm-111-disk-0          8K  3.44T     37.9G  -
rpool_plumbus/vm-111-disk-1          8K  3.44T     7.20G  -
rpool_plumbus/vm-112-disk-0        200G  3.60T     35.4G  -
rpool_plumbus/vm-112-disk-1       33.8G  3.47T      815M  -
rpool_plumbus/vm-113-disk-0        208G  3.60T     42.8G  -
rpool_plumbus/vm-113-disk-1       33.7G  3.47T      763M  -
rpool_plumbus/vm-114-disk-0          8K  3.44T     7.20G  -
rpool_plumbus/vm-114-disk-1          8K  3.44T     37.9G  -
rpool_plumbus/vm-127-disk-0         56K  3.44T       56K  -
rpool_plumbus/vm-127-disk-1         56K  3.44T       56K  -
rpool_plumbus/vm-127-disk-3       33.0G  3.47T       56K  -
rpool_plumbus/vm-132-disk-0       66.0G  3.51T     3.01M  -
rpool_plumbus/vm-133-disk-0       66.0G  3.51T     3.01M  -
rpool_plumbus/vm-134-disk-0       66.0G  3.51T     3.02M  -
rpool_plumbus/vm-135-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-136-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-137-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-138-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-139-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-140-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-141-disk-0       33.0G  3.47T     2.64M  -
rpool_plumbus/vm-225-disk-0        177G  3.57T     42.9G  -
rpool_plumbus/vm-229-disk-0        103G  3.53T     8.05G  -
rpool_plumbus/vm-310-disk-0       3.25G  3.44T      152G  -
rpool_plumbus/vm-311-disk-0       4.85G  3.44T      152G  -

Code: 
root@plumbus:~# ls -la /dev/fastpool_plumbus/
total 0
drwxr-xr-x  2 root root  3660 Apr 30 20:09 .
drwxr-xr-x 25 root root 11380 Apr 30 20:12 ..
lrwxrwxrwx  1 root root     8 Apr 30 20:09 base-127-disk-0 -> ../zd160
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part1 -> ../zd160p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part2 -> ../zd160p2
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part3 -> ../zd160p3
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part4 -> ../zd160p4
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part5 -> ../zd160p5
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-127-disk-0-part6 -> ../zd160p6
lrwxrwxrwx  1 root root     8 Apr 30 20:09 base-404-disk-0 -> ../zd576
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-404-disk-0-part1 -> ../zd576p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-404-disk-0-part2 -> ../zd576p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 base-505-disk-0 -> ../zd208
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part1 -> ../zd208p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part2 -> ../zd208p2
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part3 -> ../zd208p3
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part4 -> ../zd208p4
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part5 -> ../zd208p5
lrwxrwxrwx  1 root root    10 Apr 30 20:09 base-505-disk-0-part6 -> ../zd208p6
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-102-disk-0 -> ../zd304
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part1 -> ../zd304p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part2 -> ../zd304p2
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part3 -> ../zd304p3
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part4 -> ../zd304p4
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part5 -> ../zd304p5
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-102-disk-0-part6 -> ../zd304p6
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1118-disk-0 -> ../zd592
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1118-disk-1 -> ../zd128
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part1 -> ../zd128p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part2 -> ../zd128p2
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part3 -> ../zd128p3
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part4 -> ../zd128p4
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part5 -> ../zd128p5
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1118-disk-1-part6 -> ../zd128p6
lrwxrwxrwx  1 root root     7 Apr 30 20:09 vm-1118-disk-2 -> ../zd32
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1118-disk-3 -> ../zd400
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1202-disk-0 -> ../zd512
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1202-disk-0-part1 -> ../zd512p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1202-disk-0-part2 -> ../zd512p2
lrwxrwxrwx  1 root root     7 Apr 30 20:09 vm-1203-disk-0 -> ../zd48
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-1203-disk-0-part1 -> ../zd48p1
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-1203-disk-0-part2 -> ../zd48p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1204-disk-0 -> ../zd416
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1204-disk-0-part1 -> ../zd416p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1204-disk-0-part2 -> ../zd416p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1205-disk-0 -> ../zd288
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1205-disk-0-part1 -> ../zd288p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1205-disk-0-part2 -> ../zd288p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1206-disk-0 -> ../zd336
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1206-disk-0-part1 -> ../zd336p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1206-disk-0-part2 -> ../zd336p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1207-disk-0 -> ../zd144
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1207-disk-0-part1 -> ../zd144p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1207-disk-0-part2 -> ../zd144p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1208-disk-0 -> ../zd272
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1208-disk-0-part1 -> ../zd272p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1208-disk-0-part2 -> ../zd272p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1209-disk-0 -> ../zd432
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1209-disk-0-part1 -> ../zd432p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1209-disk-0-part2 -> ../zd432p2
lrwxrwxrwx  1 root root     8 Apr 30 20:09 vm-1211-disk-0 -> ../zd112
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1211-disk-0-part1 -> ../zd112p1
lrwxrwxrwx  1 root root    10 Apr 30 20:09 vm-1211-disk-0-part2 -> ../zd112p2
lrwxrwxrwx  1 root root     7 Apr 30 20:09 vm-121-disk-0 -> ../zd96
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part1 -> ../zd96p1
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part2 -> ../zd96p2
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part3 -> ../zd96p3
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part4 -> ../zd96p4
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part5 -> ../zd96p5
lrwxrwxrwx  1 root root     9 Apr 30 20:09 vm-121-disk-0-part6 -> ../zd96p6
...


root@plumbus:~# ls -la /dev/rpool_plumbus/
total 0
drwxr-xr-x  2 root root  2640 Apr 30 20:12 .
drwxr-xr-x 25 root root 11380 Apr 30 20:12 ..
lrwxrwxrwx  1 root root     8 Apr 30 20:12 base-101-disk-0 -> ../zd800
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part1 -> ../zd800p1
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part2 -> ../zd800p2
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part3 -> ../zd800p3
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part4 -> ../zd800p4
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part5 -> ../zd800p5
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-101-disk-0-part6 -> ../zd800p6
lrwxrwxrwx  1 root root     9 Apr 30 20:12 base-1102-disk-0 -> ../zd1104
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part1 -> ../zd1104p1
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part2 -> ../zd1104p2
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part3 -> ../zd1104p3
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part4 -> ../zd1104p4
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part5 -> ../zd1104p5
lrwxrwxrwx  1 root root    11 Apr 30 20:12 base-1102-disk-0-part6 -> ../zd1104p6
lrwxrwxrwx  1 root root     8 Apr 30 20:12 base-127-disk-2 -> ../zd640
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part1 -> ../zd640p1
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part2 -> ../zd640p2
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part3 -> ../zd640p3
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part4 -> ../zd640p4
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part5 -> ../zd640p5
lrwxrwxrwx  1 root root    10 Apr 30 20:12 base-127-disk-2-part6 -> ../zd640p6
lrwxrwxrwx  1 root root     9 Apr 30 20:12 vm-101-disk-2 -> ../zd1008
lrwxrwxrwx  1 root root     8 Apr 30 20:12 vm-101-disk-3 -> ../zd688
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part1 -> ../zd688p1
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part2 -> ../zd688p2
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part3 -> ../zd688p3
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part4 -> ../zd688p4
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part5 -> ../zd688p5
lrwxrwxrwx  1 root root    10 Apr 30 20:12 vm-101-disk-3-part6 -> ../zd688p6
lrwxrwxrwx  1 root root     9 Apr 30 20:12 vm-102-disk-0 -> ../zd1072
lrwxrwxrwx  1 root root     8 Apr 30 20:12 vm-102-disk-1 -> ../zd752
...

Nothing on ps -ef|grep zfs


root@plumbus:~# dmesg
https://pastebin.com/ydEA8gCW

Wondering if there is something wrong related with this thread:
https://forum.proxmox.com/threads/w...-pool-itself-when-upgrading-to-zfs-2-0.88413/
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom