could not activate storage '[POOL NAME]', zfs error: cannot import '[POOL NAME]': no such pool available (500)

[jezman]

Hello! I'm new to the world of Proxmox and have been having a great time. I had created a ZFS pool of 8 HDDs and a few VMs (TrueNAS, Home Assistant, Windows 10) ). Everything was working well.

Then, this afternoon, my ZFS storage pool has a grey question mark next to it, and I get the error in the subject line of this thread. I didn't do anything (at least to my knowledge) to trigger this. I have tried googling for hours, but no luck.

The interesting thing is that Plex (which is running in my TrueNAS VM) still works fine with all my content. Network file sharing in windows also works fine with my files available,

When I run "zpool status" in the shell the storage pool doesn't show up (but my SSD does). I have also tried to import the zpool and it's not found.

Does anyone have any suggestions? They would be really appreciated.

Thanks!

 

[raistlinkell]

I've also just discovered my proxmox node 2 has suffered the same "Error: could not activate storage '<StoreName>' zfs error: cannot import <StoreName>: No such pool available.

I've run a smartctl on the drive and it passed no problems. All 3 nodes host an internal identical 6Tb SATA NAS drive.




i ssh'd to the proxmox node and executed "journalctl -xe" command. Below is the output.

t@proxmoxbox2:~# journalctl -xe
Nov 16 08:48:03 proxmoxbox2 kernel: ata3.00: error: { UNC }
Nov 16 08:48:03 proxmoxbox2 kernel: ata3.00: configured for UDMA/133
Nov 16 08:48:03 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=4s
Nov 16 08:48:03 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#13 Sense Key : Medium Error [current]
Nov 16 08:48:03 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#13 Add. Sense: Unrecovered read error - auto reallocate failed
Nov 16 08:48:03 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#13 CDB: Read(16) 88 00 00 00 00 02 ba a0 ac 20 00 00 00 e0 00 00
Nov 16 08:48:03 proxmoxbox2 kernel: blk_update_request: I/O error, dev sdc, sector 11721026592 op 0x0:(READ) flags 0x0 phys_seg 28 prio class 0
Nov 16 08:48:03 proxmoxbox2 kernel: ata3: EH complete
Nov 16 08:48:07 proxmoxbox2 kernel: ata3.00: exception Emask 0x0 SAct 0x8000003c SErr 0x0 action 0x0
Nov 16 08:48:07 proxmoxbox2 kernel: ata3.00: irq_stat 0x40000008
Nov 16 08:48:07 proxmoxbox2 kernel: ata3.00: failed command: READ FPDMA QUEUED
Nov 16 08:48:07 proxmoxbox2 kernel: ata3.00: cmd 60/e0:18:20:0a:00/00:00:00:00:00/40 tag 3 ncq dma 114688 in
                                             res 41/40:00:20:0a:00/00:00:00:00:00/00 Emask 0x409 (media error) <F>
Nov 16 08:48:07 proxmoxbox2 kernel: ata3.00: status: { DRDY ERR }
Nov 16 08:48:07 proxmoxbox2 kernel: ata3.00: error: { UNC }
Nov 16 08:48:07 proxmoxbox2 kernel: ata3.00: configured for UDMA/133
Nov 16 08:48:07 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=8s
Nov 16 08:48:07 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#3 Sense Key : Medium Error [current]
Nov 16 08:48:07 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#3 Add. Sense: Unrecovered read error - auto reallocate failed
Nov 16 08:48:07 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#3 CDB: Read(16) 88 00 00 00 00 00 00 00 0a 20 00 00 00 e0 00 00
Nov 16 08:48:07 proxmoxbox2 kernel: blk_update_request: I/O error, dev sdc, sector 2592 op 0x0:(READ) flags 0x0 phys_seg 27 prio class 0
Nov 16 08:48:07 proxmoxbox2 kernel: ata3: EH complete
Nov 16 08:48:07 proxmoxbox2 pvestatd[3549]: zfs error: cannot open 'Int6TbStore': no such pool
Nov 16 08:48:07 proxmoxbox2 pvestatd[3549]: could not activate storage 'Int6TbStore', command 'zpool import -d /dev/disk/by-id/ -o 'cachefile=none' Int6TbStore' failed: got timeout
Nov 16 08:48:07 proxmoxbox2 pvestatd[3549]: status update time (20.394 seconds)
Nov 16 08:48:07 proxmoxbox2 pvestatd[3549]: zfs error: cannot open 'Int6TbStore': no such pool
Nov 16 08:48:11 proxmoxbox2 kernel: ata3.00: exception Emask 0x0 SAct 0x6000008d SErr 0x0 action 0x0
Nov 16 08:48:11 proxmoxbox2 kernel: ata3.00: irq_stat 0x40000008
Nov 16 08:48:11 proxmoxbox2 kernel: ata3.00: failed command: READ FPDMA QUEUED
Nov 16 08:48:11 proxmoxbox2 kernel: ata3.00: cmd 60/e0:00:20:08:00/00:00:00:00:00/40 tag 0 ncq dma 114688 in
                                             res 41/40:00:20:08:00/00:00:00:00:00/00 Emask 0x409 (media error) <F>
Nov 16 08:48:11 proxmoxbox2 kernel: ata3.00: status: { DRDY ERR }
Nov 16 08:48:11 proxmoxbox2 kernel: ata3.00: error: { UNC }
Nov 16 08:48:11 proxmoxbox2 kernel: ata3.00: configured for UDMA/133
Nov 16 08:48:11 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
Nov 16 08:48:11 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
Nov 16 08:48:11 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error - auto reallocate failed
Nov 16 08:48:11 proxmoxbox2 kernel: sd 2:0:0:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 08 20 00 00 00 e0 00 00
Nov 16 08:48:11 proxmoxbox2 kernel: blk_update_request: I/O error, dev sdc, sector 2080 op 0x0:(READ) flags 0x0 phys_seg 28 prio class 0
Nov 16 08:48:11 proxmoxbox2 kernel: ata3: EH complete

Can this be recovered or is it a Proxmox node rebuild?

 

[raistlinkell]

well a bit more investigating of the drive in question.

I executed a smartctl -a /dev/sdc on the drive and the below was reported

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   001   001   051    Pre-fail  Always   FAILING_NOW 40612
  3 Spin_Up_Time            0x0027   223   202   021    Pre-fail  Always       -       7841
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       184
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   053   053   000    Old_age   Always       -       34813
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       181
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       146
193 Load_Cycle_Count        0x0032   193   193   000    Old_age   Always       -       21888
194 Temperature_Celsius     0x0022   112   107   000    Old_age   Always       -       40
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       4
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

Looks like the HDD is failing... :-(

 

[akhileshb]

I have the same error and my drives arent failing. They show as zfspool too. Did anyone get this resolved?