[SOLVED] Previously used zfs drives are not available

[Curt Hall]

luckily noone is using any of the VM's right now (thanks again to COVID) so reboot it is!

 

[Curt Hall]

Rebooting now, none is using the VM's due to COVID and our center being off limits to public, so I cna make changes during day (4:15pm my time right now). We'll see in about 10 minutes...

 

[Curt Hall]

Still nothing, wow this is crazy.

 

[H4R0]

Your drives are multipath formatted and therefore protected..

list them and remove them.

multipath -l
multipath -f <id>

After that reformat your drive with gpt and setup zfs on that partition, not on the whole drive.

You are messing around with the system i hope you have backups.

 

[Curt Hall]

i have all my current VM's on a iscsi target using a HP Nimble. the "local" drives in question have nothing on them. I am configuring them as my backup actually. all VM's are thin client kiosk machines and clones of each other which are spread out over 4 other Proxmox boxes., only difference is IP address and domain name.

 

[Curt Hall]

Is the "ID" the first part starting with 35000...?

root@proxmox3:~# multipath -l
35000c5009f224cdf dm-3 SEAGATE,ST1000NX0453
size=932G features='1 retain_attached_hw_handler' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
`- 0:0:11:0 sdf 8:80 active undef running
35000c500b8ba5d47 dm-4 SEAGATE,ST1000NX0453
size=932G features='1 retain_attached_hw_handler' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
`- 0:0:6:0 sda 8:0 active undef running
2f516411d15cedb126c9ce9008e5e60a7 dm-12 Nimble,Server
size=5.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
|- 3:0:0:0 sdj 8:144 active undef running
|- 1:0:0:0 sdh 8:112 active undef running
|- 2:0:0:0 sdi 8:128 active undef running
`- 4:0:0:0 sdk 8:160 active undef running
35000c500964a380b dm-2 SEAGATE,ST1000NX0453
size=932G features='1 retain_attached_hw_handler' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
`- 0:0:10:0 sde 8:64 active undef running
35000c500964840eb dm-6 SEAGATE,ST1000NX0453
size=932G features='1 retain_attached_hw_handler' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
`- 0:0:8:0 sdc 8:32 active undef running
35000c500964830a7 dm-5 SEAGATE,ST1000NX0453
size=932G features='1 retain_attached_hw_handler' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
`- 0:0:7:0 sdb 8:16 active undef running
35000c500b8ba6ba3 dm-7 SEAGATE,ST1000NX0453
size=932G features='1 retain_attached_hw_handler' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
`- 0:0:9:0 sdd 8:48 active undef running

 

[Curt Hall]

i got it, it was the multipath all along, sheez.

 

[LnxBil]

i got it, it was the multipath all along, sheez.

Sorry, hadn't time to read the forums all day. That was my first thought and therefore I asked for the lsblk. Please always use blacklist device for local storage. The name can be determined e.g. by lsscsi:

blacklist {
    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
    devnode "^hd[a-z][[0-9]*]"
    devnode "^cciss!c[0-9]d[0-9]*"

    # root@machine ~ > lsscsi
    # [0:2:0:0]    disk    FTS      PRAID CP400i     4.66  /dev/sda
    # [3:0:0:0]    disk    FUJITSU  ETERNUS_DXL      1080  /dev/sdb
    # [12:0:0:0]   disk    FUJITSU  ETERNUS_DXL      1080  /dev/sdc
    # [13:0:0:0]   disk    BUFFALO  HD-ABCDE         0108  /dev/sdd

    device {
         vendor "FTS"
         product "PRAID.*"
    }
    device {
         vendor "BUFFALO"
    }
}

 

[Curt Hall]

Sorry, hadn't time to read the forums all day. That was my first thought and therefore I asked for the lsblk. Please always use blacklist device for local storage. The name can be determined e.g. by lsscsi:

blacklist {
    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
    devnode "^hd[a-z][[0-9]*]"
    devnode "^cciss!c[0-9]d[0-9]*"

    # root@machine ~ > lsscsi
    # [0:2:0:0]    disk    FTS      PRAID CP400i     4.66  /dev/sda
    # [3:0:0:0]    disk    FUJITSU  ETERNUS_DXL      1080  /dev/sdb
    # [12:0:0:0]   disk    FUJITSU  ETERNUS_DXL      1080  /dev/sdc
    # [13:0:0:0]   disk    BUFFALO  HD-ABCDE         0108  /dev/sdd

    device {
         vendor "FTS"
         product "PRAID.*"
    }
    device {
         vendor "BUFFALO"
    }
}

I get this error:
root@proxmox3:~# lsscsi
-bash: lsscsi: command not found

 

[LnxBil]

I get this error:
root@proxmox3:~# lsscsi
-bash: lsscsi: command not found

Yes, install it:

apt install lsscsi

 

[Curt Hall]

To be clear they are working now that the multipath was disabled for those 6 HDD's, but here's the output:

root@proxmox3:~# lsscsi
[0:0:6:0] disk SEAGATE ST1000NX0453 N0A3 /dev/sda
[0:0:7:0] disk SEAGATE ST1000NX0453 N003 /dev/sdb
[0:0:8:0] disk SEAGATE ST1000NX0453 N003 /dev/sdc
[0:0:9:0] disk SEAGATE ST1000NX0453 N0A3 /dev/sdd
[0:0:10:0] disk SEAGATE ST1000NX0453 N003 /dev/sde
[0:0:11:0] disk SEAGATE ST1000NX0453 N003 /dev/sdf
[0:2:0:0] disk LSI UCSC-RAID2008M8i 2.13 /dev/sdg
[1:0:0:0] disk Nimble Server 1.0 /dev/sdh
[2:0:0:0] disk Nimble Server 1.0 /dev/sdi
[3:0:0:0] disk Nimble Server 1.0 /dev/sdj
[4:0:0:0] disk Nimble Server 1.0 /dev/sdk

 

[edgares]

You shouldn't need to wipe the disks using dd - just clean out the MBR & GPT tables, all of the backup GPT copies and any LVM data.

"sgdisk --zap-all <device>" should do it.

After you've done that - or if you used the "dd" wipe - you have to get the system to re-trigger the device info for the disk. If its hot-swappable then pulling it out for a bit and re-plugging it will do. Reboot the host will do it too. Surely there is a less intrusive way but I don't know it.

ps - if you do the DD as shown above then you are doing writing byte-at-a-time IO to the disk. Buffered, of course, but still a lot slower than you really want. It will go quite a bit faster if you use block writes.

"dd ibs=81920 obs=81920 if=/dev/zero of=<device> status=progress"

This version will do block writes 20 disk sectors at a time (assuming 4k blocks - 160 sectors at time for 512 byte disks). This is LOTS faster - and adding the progress status helps you watch it go.

This suggestion fixed my issue. Thanks!

 

[bramol]

You shouldn't need to wipe the disks using dd - just clean out the MBR & GPT tables, all of the backup GPT copies and any LVM data.

"sgdisk --zap-all <device>" should do it.

After you've done that - or if you used the "dd" wipe - you have to get the system to re-trigger the device info for the disk. If its hot-swappable then pulling it out for a bit and re-plugging it will do. Reboot the host will do it too. Surely there is a less intrusive way but I don't know it.

ps - if you do the DD as shown above then you are doing writing byte-at-a-time IO to the disk. Buffered, of course, but still a lot slower than you really want. It will go quite a bit faster if you use block writes.

"dd ibs=81920 obs=81920 if=/dev/zero of=<device> status=progress"

This version will do block writes 20 disk sectors at a time (assuming 4k blocks - 160 sectors at time for 512 byte disks). This is LOTS faster - and adding the progress status helps you watch it go.

If
sgdisk --zap-all <device>
Didn't work, try
wipefs --all --force <device>

 

[newbprox]

I did not mean delete partition.
I ment go into gdisk(cli) look at disk structure.
It should list 3 option after message that partiotn table of type found.
Choose to create clean gpt.

This should remove anything on the disk and create an new clean gpt table.

You may need to remove and reinsert the disk after, but most of the time it is ready to use as is.
So step one: in host cli, gdisk /dev/sdX
Step 2 display partition table
Step 3 option to create new gpt.
Step 4 w to write all to disk and exit.
Done...

Beautiful. Fixed this for me. Thank you!