ZFS pool is enabled but not active

M

mert98

New Member
Apr 9, 2022
9
1
3
i have a Proxmox node running on version 7.1-7 with 4 SSD and 2 HDD storage configured with ZFS. Everything was working perfectly. Until yesterday, i rebooted node (whis was not my first reboot after setup), and now one of the SSD storages named 'zfs_ssd_1' is not activated. The GUI shows that it is enabled but not activated. When I run zpool status, it does not show 'zfs_ssd_1'. However, when I run
blkid, it shows 'zfs_ssd_1' on '/dev/sdb1'. When I run lsblk, it confirms that '/dev/sdb1' is mounted.

I have checked similar forum posts (https://forum.proxmox.com/threads/zfs-i-o-error.33670/) and ran a smartctl test, which completed with no errors. However, I was unable to clear errors with zpool clear zfs_ssd_1, which gave the error 'cannot open 'zfs_ssd_1': no such pool'. I believe the problem is with ZFS since other system commands can detect the storage with no problem.

Are there any other solutions or methods I can try to resolve this issue?

Screenshot 2023-04-28 at 12.50.03 PM.png

zpool status

Code: 
zpool status
  pool: zfs_hdd_1
 state: ONLINE
  scan: scrub repaired 0B in 14:02:51 with 0 errors on Sun Apr  9 14:26:52 2023
config:

        NAME                                STATE     READ WRITE CKSUM
        zfs_hdd_1                           ONLINE       0     0     0
          ata-ST16000NE000-2WX103_ZR50J8HZ  ONLINE       0     0     0

errors: No known data errors

  pool: zfs_hdd_2
 state: ONLINE
  scan: scrub repaired 0B in 17:14:09 with 0 errors on Sun Apr  9 17:38:12 2023
config:

        NAME                                 STATE     READ WRITE CKSUM
        zfs_hdd_2                            ONLINE       0     0     0
          ata-ST16000NM001G-2KK103_ZL2GJ7GS  ONLINE       0     0     0

errors: No known data errors

  pool: zfs_nvme_1
 state: ONLINE
  scan: scrub repaired 0B in 00:39:49 with 0 errors on Sun Apr  9 01:03:54 2023
config:

        NAME                               STATE     READ WRITE CKSUM
        zfs_nvme_1                         ONLINE       0     0     0
          nvme-eui.6479a73f202002a7-part1  ONLINE       0     0     0

errors: No known data errors

  pool: zfs_ssd_2
 state: ONLINE
  scan: scrub repaired 0B in 02:32:11 with 0 errors on Sun Apr  9 02:56:18 2023
config:

        NAME                                           STATE     READ WRITE CKSUM
        zfs_ssd_2                                      ONLINE       0     0     0
          ata-Samsung_SSD_870_QVO_4TB_S5STNG0NC01902Z  ONLINE       0     0     0

errors: No known data errors

  pool: zfs_ssd_3
 state: ONLINE
  scan: scrub repaired 0B in 01:32:30 with 0 errors on Sun Apr  9 01:56:38 2023
config:

        NAME                                           STATE     READ WRITE CKSUM
        zfs_ssd_3                                      ONLINE       0     0     0
          ata-Samsung_SSD_870_QVO_4TB_S5STNG0NC01843T  ONLINE       0     0     0

blkid

Code: 
blkid
/dev/nvme1n1p1: LABEL="zfs_nvme_1" UUID="11974131303405226487" UUID_SUB="3233527513562740301" BLOCK_SIZE="4096" TYPE="zfs_member" PARTLABEL="zfs-a3fa31dd8248b3be" PARTUUID="c8f3520b-492e-ae42-9297-ca4a43033d3c"
/dev/nvme0n1p2: UUID="3CEC-091F" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="529e330a-9598-4e94-98f0-29610d14c440"
/dev/nvme0n1p3: UUID="1g3mOx-PfHQ-psa2-GfP9-ydUg-2i1R-YFQJIr" TYPE="LVM2_member" PARTUUID="d1354f69-a08c-4bd8-a7df-41195ce19c3c"
/dev/mapper/pve-swap: UUID="3458894b-613d-4ff6-9d50-fb2494342ba0" TYPE="swap"
/dev/mapper/pve-root: UUID="2d740743-96f8-4058-8d05-1f681e942e6f" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sda1: UUID="d701877a-a835-40c0-8564-1449a579d609" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="af544811-8def-4582-be32-35ddb8df4abe"
/dev/sdb1: LABEL="zfs_ssd_1" UUID="13532520614518614830" UUID_SUB="15735742699506282199" BLOCK_SIZE="4096" TYPE="zfs_member" PARTLABEL="zfs-f14f52327cf1b1e2" PARTUUID="7f9c12b3-d668-3446-abef-bb31896e125e"
/dev/sdf1: LABEL="zfs_ssd_3" UUID="4438819123320962145" UUID_SUB="17266877199441579046" BLOCK_SIZE="4096" TYPE="zfs_member" PARTLABEL="zfs-a8b753fc7c283da9" PARTUUID="a4beaccf-2e15-f04f-b353-697aae991b89"
/dev/sdc1: LABEL="zfs_hdd_1" UUID="14137249408840784235" UUID_SUB="16108780857322988310" BLOCK_SIZE="4096" TYPE="zfs_member" PARTLABEL="zfs-f602d675bc0d0c59" PARTUUID="dc8db521-f2f0-1948-aa1b-fade319c6dc5"
/dev/sdd1: LABEL="zfs_hdd_2" UUID="3064075094854740590" UUID_SUB="8331361168905152812" BLOCK_SIZE="4096" TYPE="zfs_member" PARTLABEL="zfs-06a7e839d48182ca" PARTUUID="e2a6369b-f039-7e49-84c1-d557292034d2"
/dev/sde1: LABEL="zfs_ssd_2" UUID="7460628251768455693" UUID_SUB="5418112381992914531" BLOCK_SIZE="4096" TYPE="zfs_member" PARTLABEL="zfs-54ed532b4af9a5d4" PARTUUID="6535f9df-a239-0841-a9be-d26dc2443608"
/dev/zd48p2: UUID="d7149358-5b59-480b-ac69-3302698d3c15" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="90d459d2-94eb-412a-a46c-aaa708e21b17"
/dev/zd0: LABEL="storage" UUID="9921686d-58ac-40c6-9d0c-b9fdcc859253" BLOCK_SIZE="4096" TYPE="ext4"
/dev/zd16p1: UUID="87baa5b5-7ee5-4947-83a6-28e029af9847" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="storage" PARTUUID="982e4c47-d1af-422e-8897-9b901e414196"
/dev/zd80p2: UUID="f16de96f-610b-4abc-97d0-8e745521a37b" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="0e57feb7-8f41-43e8-a343-c10ee0b82497"
/dev/zd112: UUID="2566b6bf-a4c9-4435-a98f-c7a5b387a42d" BLOCK_SIZE="4096" TYPE="ext4"
/dev/zd128p2: UUID="ff9fd2fc-5e17-49bb-8acb-f1dab0f25fef" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="9c8f318a-ed6b-48b9-ba54-e91099d50841"
/dev/zd96p2: UUID="ababb4e9-46f0-4084-af4f-f703da8f6764" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="c79d76bc-b8b7-493e-a022-f3cdb12aaff9"
/dev/zd144p2: UUID="2a6f8edf-c9ab-4f46-a736-697c241341a2" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="44895e25-7535-4c11-9ce4-93dc9215d5d8"
/dev/zd144p3: UUID="lB0Qr6-hBvK-CNFT-8lG7-xEKw-mcZ0-LVxOIS" TYPE="LVM2_member" PARTUUID="649b890d-65b7-43e2-9c23-508ff783f088"
/dev/zd160p2: UUID="d5a3ed9c-a3fe-437d-b1b3-3c7b1f77945f" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="32aa0079-3748-4a0b-9513-6e2f845c8023"
/dev/zd32: LABEL="storage" UUID="27ba33f9-9c65-4342-9aa2-877766d1dbb2" BLOCK_SIZE="4096" TYPE="ext4"
/dev/zd64: LABEL="cold_storage" UUID="7b2da8cf-9024-403a-93f1-1d28026f567c" BLOCK_SIZE="4096" TYPE="ext4"
/dev/zd176: UUID="2c3c1929-f9d1-41d3-af4a-82b4bf6bd3e5" BLOCK_SIZE="4096" TYPE="ext4"
/dev/zd192: UUID="046a408f-2919-4a09-a28c-19abe52e17b0" BLOCK_SIZE="4096" TYPE="ext4"
/dev/zd224: UUID="666d2953-1acd-497e-a462-5d05aa91d121" BLOCK_SIZE="4096" TYPE="ext4"
/dev/zd208: LABEL="storage" UUID="9921686d-58ac-40c6-9d0c-b9fdcc859253" BLOCK_SIZE="4096" TYPE="ext4"
/dev/zd240: LABEL="storage" UUID="9921686d-58ac-40c6-9d0c-b9fdcc859253" BLOCK_SIZE="4096" TYPE="ext4"
/dev/nvme1n1p9: PARTUUID="c9710ed6-cc8b-3e4e-b39c-e6c49338b2fc"
/dev/nvme0n1p1: PARTUUID="d0edf39e-b685-434f-b671-d40319f36ec4"
/dev/sdb9: PARTUUID="51d17a08-4106-4f40-977e-95ce92023f2d"
/dev/sdf9: PARTUUID="7a85b0e9-52af-e741-bdf5-58455575deee"
/dev/sdc9: PARTUUID="5b3e3d8d-8478-4344-a5c6-7cc913f1f641"
/dev/sdd9: PARTUUID="5532a9a5-6a82-ae48-91bf-a07b23ea18e9"
/dev/sde9: PARTUUID="05ebb001-1a4d-cd44-a321-2e21083ebefb"
/dev/zd48p1: PARTUUID="51d05779-7b3b-44d8-a1ee-4e7af3c02fef"
/dev/zd80p1: PARTUUID="6263d773-fa78-450e-bb09-6334961e287f"
/dev/zd128p1: PARTUUID="c4a0a16f-eab5-4562-859f-d40ea49d8ef0"
/dev/zd96p1: PARTUUID="b3918558-cde6-42e3-9e11-037927955d64"
/dev/zd144p1: PARTUUID="babfee28-4550-431a-beaf-f1187992b4b4"
/dev/zd160p1: PARTUUID="28fc7126-c7f5-4fa1-bfc9-e58336da6643"

lsblk

Code: 
lsblk
NAME                 MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                    8:0    0   3.6T  0 disk
└─sda1                 8:1    0   3.6T  0 part /mnt/pve/tempssd
sdb                    8:16   0   3.6T  0 disk
├─sdb1                 8:17   0   3.6T  0 part
└─sdb9                 8:25   0     8M  0 part
sdc                    8:32   0  14.6T  0 disk
├─sdc1                 8:33   0  14.6T  0 part
└─sdc9                 8:41   0     8M  0 part
sdd                    8:48   0  14.6T  0 disk
├─sdd1                 8:49   0  14.6T  0 part
└─sdd9                 8:57   0     8M  0 part
sde                    8:64   0   3.6T  0 disk
├─sde1                 8:65   0   3.6T  0 part
└─sde9                 8:73   0     8M  0 part
sdf                    8:80   0   3.6T  0 disk
├─sdf1                 8:81   0   3.6T  0 part
└─sdf9                 8:89   0     8M  0 part
zd0                  230:0    0   3.4T  0 disk
zd16                 230:16   0     2T  0 disk
└─zd16p1             230:17   0     2T  0 part
zd32                 230:32   0   7.8T  0 disk
zd48                 230:48   0   300G  0 disk
├─zd48p1             230:49   0     1M  0 part
└─zd48p2             230:50   0   300G  0 part
zd64                 230:64   0   5.9T  0 disk
zd80                 230:80   0   1.2T  0 disk
├─zd80p1             230:81   0     1M  0 part
└─zd80p2             230:82   0   1.2T  0 part
zd96                 230:96   0   300G  0 disk
├─zd96p1             230:97   0     1M  0 part
└─zd96p2             230:98   0   300G  0 part
zd112                230:112  0  1000G  0 disk
zd128                230:128  0   250G  0 disk
├─zd128p1            230:129  0     1M  0 part
└─zd128p2            230:130  0   250G  0 part
zd144                230:144  0   250G  0 disk
├─zd144p1            230:145  0     1M  0 part
├─zd144p2            230:146  0     1G  0 part
└─zd144p3            230:147  0   249G  0 part
zd160                230:160  0   300G  0 disk
├─zd160p1            230:161  0     1M  0 part
└─zd160p2            230:162  0   300G  0 part
zd176                230:176  0   1.5T  0 disk
zd192                230:192  0  1000G  0 disk
zd208                230:208  0   3.4T  0 disk
zd224                230:224  0  1000G  0 disk
zd240                230:240  0   5.4T  0 disk
nvme1n1              259:0    0   3.6T  0 disk
├─nvme1n1p1          259:1    0   3.6T  0 part
└─nvme1n1p9          259:2    0     8M  0 part
nvme0n1              259:3    0 476.9G  0 disk
├─nvme0n1p1          259:4    0  1007K  0 part
├─nvme0n1p2          259:5    0   512M  0 part /boot/efi
└─nvme0n1p3          259:6    0 476.4G  0 part
  ├─pve-swap         253:0    0     8G  0 lvm  [SWAP]
  ├─pve-root         253:1    0    96G  0 lvm  /
  ├─pve-data_tmeta   253:2    0   3.6G  0 lvm
  │ └─pve-data-tpool 253:4    0 349.3G  0 lvm
  │   └─pve-data     253:5    0 349.3G  1 lvm
  └─pve-data_tdata   253:3    0 349.3G  0 lvm
    └─pve-data-tpool 253:4    0 349.3G  0 lvm
      └─pve-data     253:5    0 349.3G  1 lvm


after i ran systemctl test

Code: 
root@deepmia-cluster2:~# smartctl --all /dev/sdb1
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.19-2-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 870 QVO 4TB
Serial Number:    S5STNG0NC01892H
LU WWN Device Id: 5 002538 f70c096d6
Firmware Version: SVQ01B6Q
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Apr 29 12:52:37 2023 +03
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (  25) The self-test routine was aborted by
                                        the host.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x53) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 320) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   096   096   000    Old_age   Always       -       19232
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       138
177 Wear_Leveling_Count     0x0013   097   097   000    Pre-fail  Always       -       28
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   069   047   000    Old_age   Always       -       31
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       118
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       127290026072

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Aborted by host               90%     19170         -
# 2  Extended offline    Completed without error       00%     19167         -
# 3  Extended offline    Aborted by host               90%     19162         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
  256        0    65535  Read_scanning was never started
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
Last edited:
Hello again so far i tried zpool import which shows zfs_ssd_1 in ONLINE state and no erros however when i enter zpool import zfs_ssd_1, i get I/O error.

Code: 
root@deepmia-cluster2:~# zpool import
   pool: zfs_ssd_1
     id: 13532520614518614830
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        zfs_ssd_1                                      ONLINE
          ata-Samsung_SSD_870_QVO_4TB_S5STNG0NC01892H  ONLINE

Code: 
root@deepmia-cluster2:~# zpool import zfs_ssd_1
cannot import 'zfs_ssd_1': I/O error
        Destroy and re-create the pool from
        a backup source.

i have also tried zpool import -f and zpool import -F options but get the same error. I know i have option zpool import -XF but dont want to use it since its risky. Are there any other methods i can try ?
 
Last edited:
Don't use consumer QLC SSD like your Samsung QVO with ZFS. They won't offer the durability and write performance ZFS needs.
mert98 said:
after i ran systemctl test
Click to expand...
The last successfuly smarttest you showed was 60 hours ago. Was it done after the I/O errors occured or before?

mert98 said:
i have also tried zpool import -f and zpool import -F options but get the same error. I know i have option zpool import -XF but dont want to use it since its risky. Are there any other methods i can try ?
Click to expand...
I would check the cabling. This can cause IO errors too. Might also not hurt to check if the SSD got a newer firmware available.
 
Dunuin said:
Don't use consumer QLC SSD like your Samsung QVO with ZFS. They won't offer the durability and write performance ZFS needs.

The last successfuly smarttest you showed was 60 hours ago. Was it done after the I/O errors occured or before?


I would check the cabling. This can cause IO errors too. Might also not hurt to check if the SSD got a newer firmware available.
Click to expand...
Hello; Smartctl test conducted after I/O error. I have checked the cabling and everthing seems in order. I have also tried zpool import -XF zfs_ssd_1 but did not work.
 
Last edited:
Is there any way to start VMs attached to zfs_ssd_1. One of them uses it as a extra storage, Can i boot vm without deleting volume ?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom