[SOLVED] Help with replacing failed drive, zpool replace not working

[surfing_IT]

Hello everyone,
My ZFS data store had a failed drive this weekend and I'm having trouble replacing it. I'll provide screenshots to better document.

About:
ext4 ssd for boot
10x 8TB in raidz2 + ssd for cache & log
/dev/sdc is the drive being replaced (it had bad sectors, meh)

Steps:
I followed this guide, https://dannyda.com/2020/05/16/how-...al-disk-from-proxmox-pve-for-zfs-pool-easily/
But zpool replace tank /dev/sdc results in the message: cannot replace /dev/sdc with /dev/sdc: no such device in pool
This is where I am stuck, and replacing a drive should not be this hard!

The new /replacement drive is in the same slot (/dev/sdc), always shows as unavilable via zpool status -v and in the ZFS gui.
PVE will add GPT and ZFS automatically in the Disks gui. and shows the partitions added correctly, but is not added to the pool nor resilvering.

Thanks!!

Please see attached screen shots.

 

[LnxBil]

This is where I am stuck, and replacing a drive should not be this hard!

It's not, the output of zpool status shows the correct names of your pool, so you have to use them to replace the disk. And please use code tags and copy&paste text.

 

[surfing_IT]

zpool status was included in the screenshots. see caspar_-_Proxmox_Virtual_Environment.jpg above

 

[surfing_IT]

Thank you. I think I got it now. I ran:

zpool replace tank /dev/disk/by-id/ata-ST8000DM004-2CX188_ZCT1J3AZ-part1 /dev/disk/by-id/ata-ST8000VN004-2M2101_WKD1PMS8-part1

And now zpool status -v says 'replacing-2' and 'please wait for resilver to complete'

 

[surfing_IT]

SOLVED?
To replace a drive:

1. Find failed drive (write down SN of drive, [if no hotswap: shutdown host and physically replace drive, start host])
2. Wipe partitions on new drive via cfdisk
3. In shell, run:

• zpool status -v (to get drive id’s)
• zpool replace -f pool-name /dev/disk/by-id/"old-disk-info" /dev/disk/by-id/“new-disk-info"
• zpool status -v (to confirm resilvering in process)

 

[LnxBil]

SOLVED?
To replace a drive:

1. Find failed drive (write down SN of drive, [if no hotswap: shutdown host and physically replace drive, start host])
2. Wipe partitions on new drive via cfdisk
3. In shell, run:

• zpool status -v (to get drive id’s)
• zpool replace -f pool-name /dev/disk/by-id/"old-disk-info" /dev/disk/by-id/“new-disk-info"
• zpool status -v (to confirm resilvering in process)

That is how it is described in the reference docs, yes. The device name depends on how your zpool is created and how your names are.

 

[Andrea Carnevali]

Hi.

When the disk is replaced, are the partition tables automatically copied as well?
I did a test I could not boot the server from the replaced disk correctly.
The resilver didn't give me any errors.

Hello
Andrew.

 

[Dunuin]

No, if you boot from that pool you need to partition the new disk first by useself, copy over the bootloader and then just use the 3rd partition of that disk with "zpool replace". See the paragraph "Changing a failed bootable device" here for PVE 6.4 or higher.

 

[Andrea Carnevali]

Hi Dunuin.
I replaced a failed disk in a raid 1 zfs pool.
This was the old situation :

root@proxmox1:~# root@proxmox1:~# zpool status -v SSD_4_5_2TB
  pool: SSD_4_5_2TB
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub repaired 0B in 00:35:58 with 0 errors on Sun Oct 10 00:59:59 2021
config:

        NAME                                             STATE     READ WRITE CKSUM
        SSD_4_5_2TB                                      DEGRADED     0     0     0
          mirror-0                                       DEGRADED     0     0     0
            ata-Samsung_SSD_860_PRO_2TB_S42KNX0R701089J  ONLINE       0     0     0
            ata-Samsung_SSD_860_PRO_2TB_S42KNX0R701206P  DEGRADED     0     0    16  too many errors

errors: No known data errors

I have successfully copied the partitions with the command:

sgdisk /dev/sde -R /dev/sdf
sgdisk -G /dev/sdf

Then I ran this command:

zpool replace SSD_4_5_2TB /dev/disk/by-id/ata-Samsung_SSD_860_PRO_2TB_S42KNX0R701206P-part1 /dev/disk/by-id/ata-Samsung_SSD_860_PRO_2TB_S42KNX0R401526V-part1

But now if I do the zfs status I find myself in this situation.

root@proxmox1:~# zpool status
  pool: SSD_4_5_2TB
 state: ONLINE
  scan: resilvered 893G in 00:36:22 with 0 errors on Mon Jan  3 19:02:01 2022
config:

        NAME                                                   STATE     READ WR                                             ITE CKSUM
        SSD_4_5_2TB                                            ONLINE       0                                                  0     0
          mirror-0                                             ONLINE       0                                                  0     0
            ata-Samsung_SSD_860_PRO_2TB_S42KNX0R701089J        ONLINE       0                                                  0     0
            ata-Samsung_SSD_860_PRO_2TB_S42KNX0R401526V-part1  ONLINE       0                                                  0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 01:59:13 with 0 errors on Sun Dec 12 02:23:14 2021
config:

        NAME                                             STATE     READ WRITE CK                                             SUM
        rpool                                            ONLINE       0     0                                                  0
          mirror-0                                       ONLINE       0     0                                                  0
            ata-TOSHIBA_MG04ACA200NY_9887Y0KNF7EE-part3  ONLINE       0     0                                                  0
            ata-TOSHIBA_MG04ACA200NY_98DYK2W2F7EE-part3  ONLINE       0     0                                                  0

errors: No known data errors

Do you think I made a mistake in the replacement procedure?

 

[Dunuin]

You only need to do that if you boot from that pool. But you got another "rpool" where it looks like you are booting from. So a normal "zpool replace" would be fine for your "SSD_4_5_2TB" pool but in case a disk of the "rpool" will fail you would need to do the additional stuff like copying partitions and bootloader and so on.

 

[Andrea Carnevali]

Thanks Dunuin.
I thought I made a mistake.
the zpool status command gives me the second disk as part 1.
This is right ?

Thanks so much.

 

[Dunuin]

Better would be without a partition so both disks use the same...but should work.

 

[Andrea Carnevali]

Thanks.