[solved] zfs-import-scan failed

K

kriznik

Member
Sep 29, 2023
34
2
8
Hello,
I'm digging in the logs for quite a bit, but don't know what to do next.
I've seen during boot up of my system error with failing scan service

so i've checked the status of it after

Code: 
root@ragnar:~# systemctl status zfs-import-scan.service                                                                                          
× zfs-import-scan.service - Import ZFS pools by device scanning                                                                                  
     Loaded: loaded (/lib/systemd/system/zfs-import-scan.service; enabled; preset: disabled)                                                      
     Active: failed (Result: exit-code) since Mon 2024-01-29 11:37:21 CET; 1h 0min ago                                                            
       Docs: man:zpool(8)                                                                                                                        
    Process: 1265 ExecStart=/sbin/zpool import -aN -d /dev/disk/by-id -o cachefile=none $ZPOOL_IMPORT_OPTS (code=exited, status=1/FAILURE)        
   Main PID: 1265 (code=exited, status=1/FAILURE)                                                                                                
        CPU: 187ms                                                                                                                                
                                                                                                                                                 
Jan 29 11:37:19 ragnar systemd[1]: Starting zfs-import-scan.service - Import ZFS pools by device scanning...                                      
Jan 29 11:37:21 ragnar zpool[1265]: cannot import 'DATA': pool was previously in use from another system.                                        
Jan 29 11:37:21 ragnar zpool[1265]: Last accessed by NAS (hostid=5c83bc37) at Thu Jan  1 01:00:00 1970                                            
Jan 29 11:37:21 ragnar zpool[1265]: The pool can be imported, use 'zpool import -f' to import the pool.                                          
Jan 29 11:37:21 ragnar systemd[1]: zfs-import-scan.service: Main process exited, code=exited, status=1/FAILURE                                    
Jan 29 11:37:21 ragnar systemd[1]: zfs-import-scan.service: Failed with result 'exit-code'.                                                      
Jan 29 11:37:21 ragnar systemd[1]: Failed to start zfs-import-scan.service - Import ZFS pools by device scanning.


Code: 
root@ragnar:~# zpool list                                                                                                                           
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT                                                               
ZFS   36.4T  11.7T  24.7T        -         -     0%    32%  1.00x    ONLINE  -                                                                     
root@ragnar:~#

Thing is there is no pool named DATA and never was. Maybe it's leftover on some disk somewhere, but this instalation of the Proxmox has never have DATA zfs pool defined.
Can you please help me how to get rid of this error and start zfs-import-scan service successfully?

Thank you! :)
 
Last edited:
"zpool import" should give you a list of pools that are not imported, but found. maybe that gives you a clue ;)
 
tried that earlier, but no luck

Code: 
root@ragnar:~# zpool import -f                                                                                                                     
no pools available to import                                                                                                                       
root@ragnar:~# zpool import -f DATA                                                                                                                 
cannot import 'DATA': no such pool available
 
please don't add arguments to the commands I ask you to run, especially not dangerous ones like "-f" ! maybe the problematic disk disappears later on during the boot, or is used in a way that makes "zpool import" not scan it.. did you at some point have a VM or other system called "NAS"?
 
don't be worried, am not as noobish, import i've run as well before

Code: 
root@ragnar:~# zpool import                                                                                                                        
no pools available to import                                                                                                                      
root@ragnar:~#

no, NAS is nfs mounted storage in the datacenter and no VM's are currently using it as any kind of storage except one of the backup jobs
 
Last edited:
well there is a pool label on some disk at boot time for the "DATA" pool that was last mounted by a system with hostname "NAS" ;) I cannot tell you where it is or what it was. the message will go away if you find the source and clear the label.
 
I do understand meaning of it, I just dont understand how to find it and clear it, as that disk has been used fresh new with new proxmox instalation. data pool for nas, has been on the disks which are now part of the ZFS pool I'm using now. So I guess that's leftover somewhere there.

Any ideas where to look at or tool to use?

update:
it looks like, during wiping out disks in proxmox, the old ZFS metadata has not been cleared correctly, as it shows this

Code: 
root@ragnar:~# zdb -l /dev/sda                                                                                                                    
failed to unpack label 0                                                                                                                          
failed to unpack label 1                                                                                                                          
failed to unpack label 2                                                                                                                          
failed to unpack label 3                                                                                                                          
root@ragnar:~# zdb -l /dev/sdb                                                                                                                    
failed to unpack label 0                                                                                                                          
failed to unpack label 1                                                                                                                          
failed to unpack label 2                                                                                                                          
failed to unpack label 3                                                                                                                          
root@ragnar:~# zdb -l /dev/sde                                                                                                                    
failed to unpack label 0                                                                                                                          
failed to unpack label 1                                                                                                                          
------------------------------------                                                                                                              
LABEL 2 (Bad label cksum)                                                                                                                          
------------------------------------                                                                                                              
    version: 5000                                                                                                                                  
    name: 'DATA'                                                                                                                                  
    state: 0                                                                                                                                      
    txg: 4                                                                                                                                        
    pool_guid: 6399757860077896370                                                                                                                
    errata: 0                                                                                                                                      
    hostid: 1552137271                                                                                                                            
    hostname: 'NAS'                                                                                                                                
    top_guid: 4239667779330243254                                                                                                                  
    guid: 726153779968111526                                                                                                                      
    vdev_children: 4                                                                                                                              
    vdev_tree:                                                                                                                                    
        type: 'mirror'                                                                                                                            
        id: 1                                                                                                                                      
        guid: 4239667779330243254                                                                                                                  
        metaslab_array: 140                                                                                                                        
        metaslab_shift: 34                                                                                                                        
        ashift: 12                                                                                                                                
        asize: 4998826557440                                                                                                                      
        is_log: 0                                                                                                                                  
        create_txg: 4                                                                                                                              
        children[0]:                                                                                                                              
            type: 'disk'                                                                                                                          
            id: 0                                                                                                                                  
            guid: 17334839784067705788                                                                                                            
            path: '/dev/disk/by-partuuid/9a11fdc7-2d7a-499c-8218-2b2d3883aa66'                                                                    
            whole_disk: 0                                                                                                                          
            create_txg: 4                                                                                                                          
        children[1]:                                                                                                                              
            type: 'disk'                                                                                                                          
            id: 1                                                                                                                                  
            guid: 726153779968111526                                                                                                              
            path: '/dev/disk/by-partuuid/71920111-9bb5-4a6b-9a20-ec0fcb032e79'                                                                    
            whole_disk: 0                                                                                                                          
            create_txg: 4                                                                                                                          
    features_for_read:                                                                                                                            
        com.delphix:hole_birth                                                                                                                    
        com.delphix:embedded_data                                                                                                                  
        com.klarasystems:vdev_zaps_v2                                                                                                              
    labels = 2 3                                                                                                                                  
root@ragnar:~# zdb -l /dev/sdf                                                                                                                    
failed to unpack label 0                                                                                                                          
failed to unpack label 1                                                                                                                          
------------------------------------                                                                                                              
LABEL 2 (Bad label cksum)                                                                                                                          
------------------------------------                                                                                                              
    version: 5000                                                                                                                                  
    name: 'DATA'                                                                                                                                  
    state: 0                                                                                                                                      
    txg: 4                                                                                                                                        
    pool_guid: 6399757860077896370                                                                                                                
    errata: 0                                                                                                                                      
    hostid: 1552137271                                                                                                                            
    hostname: 'NAS'                                                                                                                                
    top_guid: 15706609563549042126                                                                                                                
    guid: 1484003445448850713                                                                                                                      
    vdev_children: 4                                                                                                                              
    vdev_tree:                                                                                                                                    
        type: 'mirror'                                                                                                                            
        id: 2                                                                                                                                      
        guid: 15706609563549042126                                                                                                                
        metaslab_array: 135                                                                                                                        
        metaslab_shift: 34                                                                                                                        
        ashift: 12                                                                                                                                
        asize: 4998826557440                                                                                                                      
        is_log: 0                                                                                                                                  
        create_txg: 4                                                                                                                              
        children[0]:                                                                                                                              
            type: 'disk'                                                                                                                          
            id: 0                                                                                                                                  
            guid: 1484003445448850713                                                                                                              
            path: '/dev/disk/by-partuuid/00f575f4-17a0-4fcd-8076-b0d5f9c0bd35'                                                                    
            whole_disk: 0                                                                                                                          
            create_txg: 4                                                                                                                          
        children[1]:                                                                                                                              
            type: 'disk'                                                                                                                          
            id: 1                                                                                                                                  
            guid: 5384620560390686383                                                                                                              
            path: '/dev/disk/by-partuuid/1bca37bd-fe90-43db-b82d-8372f5f808ec'                                                                    
            whole_disk: 0                                                                                                                          
            create_txg: 4                                                                                                                          
    features_for_read:                                                                                                                            
        com.delphix:hole_birth                                                                                                                    
        com.delphix:embedded_data                                                                                                                  
        com.klarasystems:vdev_zaps_v2                                                                                                              
    labels = 2 3                                                                                                                                  
                                                                                                                      
root@ragnar:~#

what would be proper way of clearing it without destroing current ZFS pool?
 
Last edited:
are these two disks now part of the ZFS pool? or what is on them?
 
it's 8 disks in one pool, so yes, data are on them.

Code: 
root@ragnar:~# zpool status ZFS                                                                                                                     
  pool: ZFS                                                                                                                                         
 state: ONLINE                                                                                                                                     
status: Some supported and requested features are not enabled on the pool.                                                                         
        The pool can still be used, but some features are unavailable.                                                                             
action: Enable all features using 'zpool upgrade'. Once this is done,                                                                               
        the pool may no longer be accessible by software that does not support                                                                     
        the features. See zpool-features(7) for details.                                                                                           
  scan: resilvered 12.7G in 00:45:20 with 0 errors on Mon Jan 15 13:04:44 2024                                                                     
config:                                                                                                                                             
                                                                                                                                                    
        NAME                        STATE     READ WRITE CKSUM                                                                                     
        ZFS                         ONLINE       0     0     0                                                                                     
          raidz2-0                  ONLINE       0     0     0                                                                                     
            scsi-35000c500e0e509bc  ONLINE       0     0     0                                                                                     
            scsi-35000c500ece2637a  ONLINE       0     0     0                                                                                     
            scsi-35000c500ecdffc07  ONLINE       0     0     0                                                                                     
            scsi-35000c500d585d15f  ONLINE       0     0     0                                                                                     
            scsi-35000c500ecd2de3c  ONLINE       0     0     0                                                                                     
            scsi-35000c500ed3bddd0  ONLINE       0     0     0                                                                                     
            scsi-35000c500d588b8f8  ONLINE       0     0     0                                                                                     
            scsi-35000c500ed41df5e  ONLINE       0     0     0                                                                                     
                                                                                                                                                    
errors: No known data errors                                                                                                                       
root@ragnar:~#

what little worries me, that wipe out disks is maybe not enough when creating new zfs pool in prox, hmm
 
the "Wipe Disk" button actually wipes all partitions and the disk itself (using wipefs) and also overwrites the first 200M of the disk with zeroes. wipefs seems to not completely wipe the label, labels 0 and 1 are covered by the 200M, but the other two copies are not.. I am not sure how you get out of this situation other than finding and overwriting the "wrong" labels manually (or removing, wiping the full disk, and re-adding it)
 
well I can pull one by one, wipe it in different machine and then put them back which shall persist all the data. Question is if it will persist metadata again as well.

iterestingly I can see strange displacement of metadata (nota: disks are all the same and has been wiped out by proxmox UI "Wipe Disk" and then ZFS was created from UI)

Code: 
root@ragnar:~# zdb -l /dev/sdc                                                                                                                    
failed to unpack label 0                                                                                                                          
failed to unpack label 1                                                                                                                          
------------------------------------                                                                                                              
LABEL 2 (Bad label cksum)                                                                                                                          
------------------------------------                                                                                                              
    version: 5000                                                                                                                                  
    name: 'DATA'                                                                                                                                  
    state: 0                                                                                                                                      
    txg: 4                                                                                                                                        
    pool_guid: 6399757860077896370                                                                                                                
    errata: 0                                                                                                                                      
    hostid: 1552137271                                                                                                                            
    hostname: 'NAS'                                                                                                                                
    top_guid: 5987978239907774942                                                                                                                  
    guid: 5837332462316516772                                                                                                                      
    vdev_children: 4                                                                                                                              
    vdev_tree:                                                                                                                                    
        type: 'mirror'                                                                                                                            
        id: 3                                                                                                                                      
        guid: 5987978239907774942                                                                                                                  
        metaslab_array: 128                                                                                                                        
        metaslab_shift: 34                                                                                                                        
        ashift: 12                                                                                                                                
        asize: 4998826557440                                                                                                                      
        is_log: 0                                                                                                                                  
        create_txg: 4                                                                                                                              
        children[0]:                                                                                                                              
            type: 'disk'                                                                                                                          
            id: 0                                                                                                                                  
            guid: 4102303746080812214                                                                                                              
            path: '/dev/disk/by-partuuid/8b2cac09-a7cd-4532-b28a-368ac91f79c9'                                                                    
            whole_disk: 0                                                                                                                          
            create_txg: 4                                                                                                                          
        children[1]:                                                                                                                              
            type: 'disk'                                                                                                                          
            id: 1                                                                                                                                  
            guid: 5837332462316516772                                                                                                              
            path: '/dev/disk/by-partuuid/53a889a2-a80b-4cd4-a6d2-ea89025cdbde'                                                                    
            whole_disk: 0                                                                                                                          
            create_txg: 4                                                                                                                          
    features_for_read:                                                                                                                            
        com.delphix:hole_birth                                                                                                                    
        com.delphix:embedded_data                                                                                                                  
        com.klarasystems:vdev_zaps_v2                                                                                                              
    labels = 2 3                                                                                                                                  
root@ragnar:~#

then there is this partition

(next post because of lenght)

and then there is this partition

Code: 
root@ragnar:~# zdb -l /dev/sdc9                                                                                                                    
failed to unpack label 0                                                                                                                          
failed to unpack label 1                                                                                                                          
------------------------------------                                                                                                              
LABEL 2 (Bad label cksum)                                                                                                                          
------------------------------------                                                                                                              
    version: 5000                                                                                                                                  
    name: 'DATA'                                                                                                                                  
    state: 0                                                                                                                                      
    txg: 4                                                                                                                                        
    pool_guid: 6399757860077896370                                                                                                                
    errata: 0                                                                                                                                      
    hostid: 1552137271                                                                                                                            
    hostname: 'NAS'                                                                                                                                
    top_guid: 5987978239907774942                                                                                                                  
    guid: 5837332462316516772                                                                                                                      
    vdev_children: 4                                                                                                                              
    vdev_tree:                                                                                                                                    
        type: 'mirror'                                                                                                                            
        id: 3                                                                                                                                      
        guid: 5987978239907774942                                                                                                                  
        metaslab_array: 128                                                                                                                        
        metaslab_shift: 34                                                                                                                        
        ashift: 12                                                                                                                                
        asize: 4998826557440                                                                                                                      
        is_log: 0                                                                                                                                  
        create_txg: 4                                                                                                                              
        children[0]:                                                                                                                              
            type: 'disk'                                                                                                                          
            id: 0                                                                                                                                  
            guid: 4102303746080812214                                                                                                              
            path: '/dev/disk/by-partuuid/8b2cac09-a7cd-4532-b28a-368ac91f79c9'                                                                    
            whole_disk: 0                                                                                                                          
            create_txg: 4                                                                                                                          
        children[1]:                                                                                                                              
            type: 'disk'                                                                                                                          
            id: 1                                                                                                                                  
            guid: 5837332462316516772                                                                                                              
            path: '/dev/disk/by-partuuid/53a889a2-a80b-4cd4-a6d2-ea89025cdbde'                                                                    
            whole_disk: 0                                                                                                                          
            create_txg: 4                                                                                                                          
    features_for_read:                                                                                                                            
        com.delphix:hole_birth                                                                                                                    
        com.delphix:embedded_data                                                                                                                  
        com.klarasystems:vdev_zaps_v2                                                                                                              
    labels = 2 3                                                                                                                                  
root@ragnar:~#


where proxmox ZFS metadata should be?
as I can't see them anywhere else than c1 d1 e1 and so on... if i'll clear metedata on sdc and remove partition sdc9 and vice versa sdd sdd9 and so on. Might that work?

Looks to me, that partition are mixed together and UI was not able to properly wipe it out (or it's not designed to do it ... dunno)
 
Code: 
root@ragnar:~# zdb -l /dev/sdc1                                                                                                                     
------------------------------------                                                                                                               
LABEL 0                                                                                                                                             
------------------------------------                                                                                                               
    version: 5000                                                                                                                                   
    name: 'ZFS'                                                                                                                                     
    state: 0                                                                                                                                       
    txg: 1035288                                                                                                                                   
    pool_guid: 1954055311015332810                                                                                                                 
    errata: 0                                                                                                                                       
    hostid: 2556146544                                                                                                                             
    hostname: 'ragnar'                                                                                                                             
    top_guid: 154746484111137249                                                                                                                   
    guid: 15632002743534650992                                                                                                                     
    vdev_children: 1                                                                                                                               
    vdev_tree:                                                                                                                                     
        type: 'raidz'                                                                                                                               
        id: 0                                                                                                                                       
        guid: 154746484111137249                                                                                                                   
        nparity: 2                                                                                                                                 
        metaslab_array: 256                                                                                                                         
        metaslab_shift: 34                                                                                                                         
        ashift: 12                                                                                                                                 
        asize: 40007733608448                                                                                                                       
        is_log: 0                                                                                                                                   
        create_txg: 4                                                                                                                               
        children[0]:                                                                                                                               
            type: 'disk'                                                                                                                           
            id: 0                                                                                                                                   
            guid: 15632002743534650992                                                                                                             
            path: '/dev/disk/by-id/scsi-35000c500e0e509bc-part1'                                                                                   
            devid: 'scsi-35000c500e0e509bc-part1'                                                                                                   
            phys_path: 'pci-0000:03:00.0-scsi-0:0:2:0'                                                                                             
            whole_disk: 1                                                                                                                           
            DTL: 62990                                                                                                                             
            create_txg: 4                                                                                                                           
        children[1]:                                                                                                                               
            type: 'disk'                                                                                                                           
            id: 1                                                                                                                                   
            guid: 675886798905823949                                                                                                               
            path: '/dev/disk/by-id/scsi-35000c500ece2637a-part1'                                                                                   
            devid: 'scsi-35000c500ece2637a-part1'                                                                                                   
            phys_path: 'pci-0000:03:00.0-scsi-0:0:3:0'                                                                                             
            whole_disk: 1                                                                                                                           
            DTL: 62989                                                                                                                             
            create_txg: 4                                                                                                                           
        children[2]:                                                                                                                               
            type: 'disk'                                                                                                                           
            id: 2                                                                                                                                   
            guid: 3584718836626728974                                                                                                               
            path: '/dev/disk/by-id/scsi-35000c500ecdffc07-part1'                                                                                   
            devid: 'scsi-35000c500ecdffc07-part1'                                                                                                   
            phys_path: 'pci-0000:03:00.0-scsi-0:0:4:0'                                                                                             
            whole_disk: 1                                                                                                                           
            DTL: 62988                                                                                                                             
            create_txg: 4 
       children[3]:                                                                                                                               
            type: 'disk'                                                                                                                           
            id: 3                                                                                                                                   
            guid: 7154023817923393384                                                                                                               
            path: '/dev/disk/by-id/scsi-35000c500d585d15f-part1'                                                                                   
            devid: 'scsi-35000c500d585d15f-part1'                                                                                                   
            phys_path: 'pci-0000:03:00.0-scsi-0:0:5:0'                                                                                             
            whole_disk: 1                                                                                                                           
            DTL: 62987                                                                                                                             
            create_txg: 4                                                                                                                           
        children[4]:                                                                                                                               
            type: 'disk'                                                                                                                           
            id: 4                                                                                                                                   
            guid: 10023342656730422498                                                                                                             
            path: '/dev/disk/by-id/scsi-35000c500ecd2de3c-part1'                                                                                   
            devid: 'scsi-35000c500ecd2de3c-part1'                                                                                                   
            phys_path: 'pci-0000:03:00.0-scsi-0:0:6:0'                                                                                             
            whole_disk: 1                                                                                                                           
            DTL: 62986                                                                                                                             
            create_txg: 4                                                                                                                           
        children[5]:                                                                                                                               
            type: 'disk'                                                                                                                           
            id: 5                                                                                                                                   
            guid: 15423533848781696029                                                                                                             
            path: '/dev/disk/by-id/scsi-35000c500ed3bddd0-part1'                                                                                   
            devid: 'scsi-35000c500ed3bddd0-part1'                                                                                                   
            phys_path: 'pci-0000:03:00.0-scsi-0:0:7:0'                                                                                             
            whole_disk: 1                                                                                                                           
            DTL: 62985                                                                                                                             
            create_txg: 4                                                                                                                           
        children[6]:                                                                                                                               
            type: 'disk'                                                                                                                           
            id: 6                                                                                                                                   
            guid: 15582682632238580063                                                                                                             
            path: '/dev/disk/by-id/scsi-35000c500d588b8f8-part1'                                                                                   
            devid: 'scsi-35000c500d588b8f8-part1'                                                                                                   
            phys_path: 'pci-0000:03:00.0-scsi-0:0:8:0'                                                                                             
            whole_disk: 1                                                                                                                           
            DTL: 62984                                                                                                                             
            create_txg: 4                                                                                                                           
        children[7]:                                                                                                                               
            type: 'disk'                                                                                                                           
            id: 7                                                                                                                                   
            guid: 6382130521869932302                                                                                                               
            path: '/dev/disk/by-id/scsi-35000c500ed41df5e-part1'                                                                                   
            devid: 'scsi-35000c500ed41df5e-part1'                                                                                                   
            phys_path: 'pci-0000:03:00.0-scsi-0:0:9:0'                                                                                             
            whole_disk: 1                                                                                                                           
            DTL: 62983                                                                                                                             
            create_txg: 4                                                                                                                           
    features_for_read:                                                                                                                             
        com.delphix:hole_birth                                                                                                                     
        com.delphix:embedded_data                                                                                                                   
    labels = 0 1 2 3
 
alright, so removing residual information from all sdx9 partitions did the trick, and no rebuild or restore data from backup was needed :)

Code: 
zpool labelclear -f /dev/sdc9

I don't know if I could prevent this somehow during ZFS pool creation, as UI is pretty spartan ... but anyway, if somebody will experience something similar, check zfs metadata on all involved disks.

Now zfs-import-scan is without failures
 
Last edited:
  • Like
Reactions: waltar and fabian
You must log in or register to reply here.
Top Bottom