Replacing zfs drive results in loosing boot partition?

[Brian Read]

I have a backup server on 3 x 2Tb drives.

One drive died on me so I bought a new one (4Tb actually) and replaced it. It has been re-silvered sucessfully.

However it looks as though I have lost the ability to boot from that drive:

root@pbs1:~# lsblk -o +FSTYPE
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT FSTYPE
sda      8:0    0  3.6T  0 disk          
├─sda1   8:1    0  3.6T  0 part          
└─sda9   8:9    0    8M  0 part          
sdb      8:16   0  1.8T  0 disk          
├─sdb1   8:17   0 1007K  0 part          
├─sdb2   8:18   0  512M  0 part            vfat
└─sdb3   8:19   0  1.8T  0 part            zfs_member
sdc      8:32   0  1.8T  0 disk          
├─sdc1   8:33   0 1007K  0 part          
├─sdc2   8:34   0  512M  0 part            vfat
└─sdc3   8:35   0  1.8T  0 part            zfs_member
root@pbs1:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
WARN: /dev/disk/by-uuid/5CDE-445F does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
5CDF-202A is configured with: uefi (versions: 5.11.22-4-pve, 5.4.140-1-pve)
5CE0-768B is configured with: uefi (versions: 5.11.22-4-pve, 5.4.140-1-pve)
root@pbs1:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool  5.45T  2.80T  2.65T        -         -     5%    51%  1.00x    ONLINE  -
root@pbs1:~# zpool status
  pool: rpool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: resilvered 965G in 02:33:10 with 0 errors on Sat Sep 18 17:31:35 2021
config:

    NAME                                     STATE     READ WRITE CKSUM
    rpool                                    ONLINE       0     0     0
      raidz1-0                               ONLINE       0     0     0
        sda                                  ONLINE       0     0     0
        ata-TOSHIBA_HDWD120_X6CKVLDAS-part3  ONLINE       0     0     0
        ata-TOSHIBA_HDWD120_X6CK383AS-part3  ONLINE       0     0     0

errors: No known data errors
root@pbs1:~#

How do I arrange for the system to be bootable from all the drives?

 

[Stoiko Ivanov]

How do I arrange for the system to be bootable from all the drives?

If I see this correctly you replaced the disk by /dev/sda?
(please use /dev/disk/by-id links for zpool operations - they are stable and make far fewer problems in the future (also it's easier to identify which disk has problems)

else as far as I can see you attached the complete disk `zpool attach rpool /dev/sda`?
since this is a boot disk - you should have copied the partition table from one of the healthy disks, and attached only partition 3 to the pool

this is documented in the reference documentation (section 'Changing a failed boot device') :
https://pbs.proxmox.com/docs/sysadmin.html

Afaik it should be possible to detach /dev/sda again - but make sure to read up on the relevant man-pages and docs!

I hope this helps!

 

[Brian Read]

Many thanks for your useful comments.

However neither the "zpool remove" nor the "zpool detach" work to take the drive out of the Raid.

I am told thatr the rem ove does not apply to this type of Raid and that the detach is not applicable.

 

[Brian Read]

aha - found that "offline" works.

 

[Brian Read]

Thanks for the help.

root@pbs1:~# zpool status
  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Sep 21 12:12:46 2021
    2.55T scanned at 506M/s, 1.67T issued at 332M/s, 2.80T total
    572G resilvered, 59.63% done, 00:59:38 to go
config:

    NAME                                                  STATE     READ WRITE CKSUM
    rpool                                                 DEGRADED     0     0     0
      raidz1-0                                            DEGRADED     0     0     0
        replacing-0                                       DEGRADED     0     0     0
          sda                                             OFFLINE      0     0     0
          ata-WDC_WD40EZAZ-00SF3B0_WD-WX42D4104CVR-part3  ONLINE       0     0     0  (resilvering)
        ata-TOSHIBA_HDWD120_X6CKVLDAS-part3               ONLINE       0     0     0
        ata-TOSHIBA_HDWD120_X6CK383AS-part3               ONLINE       0     0     0

errors: No known data errors
root@pbs1:~#