ZFS suspended on fresh v8 install after reboot, on v7.4 it works

de_Ramon

New Member
Nov 22, 2021
7
1
3
48
Hi,

as already postet in Fresh 8.0.3 install renames disk names when ZFS is used and crashes on reboot - weird I did a fresh install with the 7.4-1 iso and all works as expected.


The 7.4-1 installation did also rename all the disk added to the zfs pool and hid the SAS-Shelf...

Before creation of ZFS-Pool an reboot the Shelf is visible:
Code:
# ls /sys/class/enclosure/1\:0\:24\:0/
0  1  10  11  12  13  14  15  16  17  18  19  2  20  21  22  23  3  4  5  6  7  8  9  components  device  id  power  subsystem  uevent

Code:
# lsblk -d -o VENDOR,MODEL,SERIAL,HCTL,SIZE,PHY-SEC,LOG-SEC,NAME | sed -e "`ls -1d /sys/class/enclosure/*/*/device/block/*|sed "s+.*enclosure/\(.*\)/device/block/\(.*\)+s-\2\\$-\2 \1-+"`"
VENDOR   MODEL                 SERIAL                           HCTL         SIZE PHY-SEC LOG-SEC NAME
LSI      RAID 5/6 SAS 6G       00b98e2f23aff7c827a0ba0001570003 0:2:0:0      1.4T    4096     512 sda
LSI      RAID 5/6 SAS 6G       0035534f2805f8c827a0ba0001570003 0:2:1:0    278.9G    4096     512 sdb
NETAPP   X477_SMEGX04TA07      S1Z2ZM13                         1:0:0:0      3.6T     512     512 sdc 1:0:24:0/0
NETAPP   X477_SMEGX04TA07      S1Z2ZLQ0                         1:0:1:0      3.6T     512     512 sdd 1:0:24:0/1
NETAPP   X477_SMEGX04TA07      S1Z2ZFGP                         1:0:2:0      3.6T     512     512 sde 1:0:24:0/2
NETAPP   X477_SMEGX04TA07      S1Z2ZGJ4                         1:0:3:0      3.6T     512     512 sdf 1:0:24:0/3
NETAPP   X477_SMEGX04TA07      S1Z2ZFGY                         1:0:4:0      3.6T     512     512 sdg 1:0:24:0/4
NETAPP   X477_SMEGX04TA07      S1Z2ZLTS                         1:0:5:0      3.6T     512     512 sdh 1:0:24:0/5
NETAPP   X477_SMEGX04TA07      S1Z2ZFML                         1:0:6:0      3.6T     512     512 sdi 1:0:24:0/6
NETAPP   X477_SMEGX04TA07      S1Z2ZLYY                         1:0:7:0      3.6T     512     512 sdj 1:0:24:0/7
NETAPP   X477_SMEGX04TA07      S1Z2ZNHN                         1:0:8:0      3.6T     512     512 sdk 1:0:24:0/8
NETAPP   X477_SMEGX04TA07      S1Z2ZFME                         1:0:9:0      3.6T     512     512 sdl 1:0:24:0/9
NETAPP   X477_SMEGX04TA07      S1Z2ZF7V                         1:0:10:0     3.6T     512     512 sdm 1:0:24:0/10
NETAPP   X477_SMEGX04TA07      S1Z2ZLG2                         1:0:11:0     3.6T     512     512 sdn 1:0:24:0/11
NETAPP   X477_SMEGX04TA07      S1Z2YGVY                         1:0:12:0     3.6T     512     512 sdo 1:0:24:0/12
NETAPP   X477_SMEGX04TA07      S1Z2ZFRJ                         1:0:13:0     3.6T     512     512 sdp 1:0:24:0/13
HL-DT-ST HL-DT-ST DVDRAM GT80N B3GRNC1041458                    3:0:0:0     1024M     512     512 sr0
KVM      vmDisk-CD             212052060601                     10:0:0:0    1024M     512     512 sr1
NETAPP   X477_SMEGX04TA07      S1Z2ZM0Z                         1:0:14:0     3.6T     512     512 sdq 1:0:24:0/14
NETAPP   X477_SMEGX04TA07      S1Z2ZFDH                         1:0:15:0     3.6T     512     512 sdr 1:0:24:0/15
NETAPP   X477_SMEGX04TA07      S1Z2YHCZ                         1:0:16:0     3.6T     512     512 sds 1:0:24:0/16
NETAPP   X477_SMEGX04TA07      S1Z2YH4S                         1:0:17:0     3.6T     512     512 sdt 1:0:24:0/17
NETAPP   X477_SMEGX04TA07      S1Z2ZF6B                         1:0:18:0     3.6T     512     512 sdu 1:0:24:0/18
NETAPP   X477_SMEGX04TA07      S1Z2ZH23                         1:0:19:0     3.6T     512     512 sdv 1:0:24:0/19
NETAPP   X477_SMEGX04TA07      S1Z2YFVA                         1:0:20:0     3.6T     512     512 sdw 1:0:24:0/20
NETAPP   X477_SMEGX04TA07      S1Z2YGVX                         1:0:21:0     3.6T     512     512 sdx 1:0:24:0/21
NETAPP   X477_SMEGX04TA07      S1Z2YGZ5                         1:0:22:0     3.6T     512     512 sdy 1:0:24:0/22
NETAPP   X477_SMEGX04TA07      S1Z2YHB5                         1:0:23:0     3.6T     512     512 sdz 1:0:24:0/23
KVM      vmDisk                212052060601                     11:0:0:0       0B     512     512 sdaa

After reboot the disk /dev/sdb - /dev/sdz will become to /dev/sdab - /dev/sday

Code:
lsblk -d -o VENDOR,MODEL,SERIAL,HCTL,SIZE,PHY-SEC,LOG-SEC,NAME | sed -e "`ls -1d /sys/class/enclosure/*/*/device/block/*|sed "s+.*enclosure/\(.*\)/device/block/\(.*\)+s-\2\\$-\2 \1-+"`" | grep -i netapp
ls: cannot access '/sys/class/enclosure/*/*/device/block/*': No such file or directory
NETAPP   X477_SMEGX04TA07      S1Z2ZM13                         1:0:26:0     3.6T     512     512 sdab
NETAPP   X477_SMEGX04TA07      S1Z2ZLQ0                         1:0:27:0     3.6T     512     512 sdac
NETAPP   X477_SMEGX04TA07      S1Z2ZFGP                         1:0:28:0     3.6T     512     512 sdad
NETAPP   X477_SMEGX04TA07      S1Z2ZGJ4                         1:0:29:0     3.6T     512     512 sdae
NETAPP   X477_SMEGX04TA07      S1Z2ZFGY                         1:0:30:0     3.6T     512     512 sdaf
NETAPP   X477_SMEGX04TA07      S1Z2ZLTS                         1:0:31:0     3.6T     512     512 sdag
NETAPP   X477_SMEGX04TA07      S1Z2ZFML                         1:0:32:0     3.6T     512     512 sdah
NETAPP   X477_SMEGX04TA07      S1Z2ZLYY                         1:0:33:0     3.6T     512     512 sdai
NETAPP   X477_SMEGX04TA07      S1Z2ZNHN                         1:0:34:0     3.6T     512     512 sdaj
NETAPP   X477_SMEGX04TA07      S1Z2ZFME                         1:0:35:0     3.6T     512     512 sdak
NETAPP   X477_SMEGX04TA07      S1Z2ZF7V                         1:0:36:0     3.6T     512     512 sdal
NETAPP   X477_SMEGX04TA07      S1Z2ZLG2                         1:0:37:0     3.6T     512     512 sdam
NETAPP   X477_SMEGX04TA07      S1Z2YGVY                         1:0:38:0     3.6T     512     512 sdan
NETAPP   X477_SMEGX04TA07      S1Z2ZFRJ                         1:0:39:0     3.6T     512     512 sdao
NETAPP   X477_SMEGX04TA07      S1Z2ZM0Z                         1:0:40:0     3.6T     512     512 sdap
NETAPP   X477_SMEGX04TA07      S1Z2ZFDH                         1:0:41:0     3.6T     512     512 sdaq
NETAPP   X477_SMEGX04TA07      S1Z2YHCZ                         1:0:42:0     3.6T     512     512 sdar
NETAPP   X477_SMEGX04TA07      S1Z2YH4S                         1:0:43:0     3.6T     512     512 sdas
NETAPP   X477_SMEGX04TA07      S1Z2ZF6B                         1:0:44:0     3.6T     512     512 sdat
NETAPP   X477_SMEGX04TA07      S1Z2ZH23                         1:0:45:0     3.6T     512     512 sdau
NETAPP   X477_SMEGX04TA07      S1Z2YFVA                         1:0:46:0     3.6T     512     512 sdav
NETAPP   X477_SMEGX04TA07      S1Z2YGVX                         1:0:47:0     3.6T     512     512 sdaw
NETAPP   X477_SMEGX04TA07      S1Z2YGZ5                         1:0:48:0     3.6T     512     512 sdax
NETAPP   X477_SMEGX04TA07      S1Z2YHB5                         1:0:49:0     3.6T     512     512 sday

and the shelf is "gone":
Code:
# ls /sys/class/enclosure/

This behavior is therefore identical to that of version 8.0, but here the ZFS pool is not correctly mounted at boot time. I suspect that the mount happens during the time when the disks are renamed / reimported.

v8 Status after first reboot:
Code:
# zpool status
  pool: pve-pool-01
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
config:

        NAME                        STATE     READ WRITE CKSUM
        pve-pool-01                 ONLINE       0     0     0
          raidz2-0                  ONLINE       0    24     0
            scsi-35000c50099acb2b3  ONLINE       3     8     0
            scsi-35000c50099acd897  ONLINE       3     8     0
            scsi-35000c50099b03c9f  ONLINE       3     6     0
            scsi-35000c50099afd167  ONLINE       3     6     0
            scsi-35000c50099b06867  ONLINE       3     4     0
            scsi-35000c50099acc6db  ONLINE       3     4     0
            scsi-35000c50099b03067  ONLINE       3     2     0
            scsi-35000c50099acb9fb  ONLINE       0     0     0
            scsi-35000c50099ac1a63  ONLINE       0     0     0
            scsi-35000c50099b030cb  ONLINE       3     2     0
            scsi-35000c50099b05013  ONLINE       3     2     0
            scsi-35000c50099acf193  ONLINE       3     4     0
            scsi-35000c50099ad1ed7  ONLINE       3     6     0
            scsi-35000c50099b0060b  ONLINE       3     6     0
            scsi-35000c50099acb317  ONLINE       3    10     0
            scsi-35000c50099b03fe3  ONLINE       3    10     0
            scsi-35000c50099acf2bf  ONLINE       3    12     0
            scsi-35000c50099ad183f  ONLINE       3     8     0
            scsi-35000c50099b054ef  ONLINE       3     6     0
            scsi-35000c50099afa22b  ONLINE       3     4     0
            scsi-35000c50099ae007b  ONLINE       3     2     0
            scsi-35000c50099ad1ea7  ONLINE       3     4     0
            scsi-35000c50099ad15cb  ONLINE       3     6     0
            scsi-35000c50099acf82f  ONLINE       3     6     0
errors: List of errors unavailable: pool I/O is currently suspended

Any ideas?

See also https://forum.proxmox.com/threads/f...s-is-used-and-crashes-on-reboot-weird.130951/

Thanks for your help!

ramon
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!