Urgent for production level replacement failed drive

[ieronymous]

Hi

I already read many posts about the title's procedure and had a guide written, but when it comes real world production environment, anxiety takes place and many things could go wrong. So I need your approval, before driving 65Kms to work on Saturday to replace a failed (ready to fail) disk.

What I had written down 2 years ago was
(The failed disk is a member of a raid 10 and not a bootable drive, created for VMs, so it is crucial to have that particular pool healthy)

1.Check failed drive's info with

From gui (Node->Disks) found the serial and also that it has 2 partitions (sdc1 and sdc9)
5000cca07....d38 (serial) (Ok so if I unplug them all I ll find the failed one to be replaced)
scsi-35000cca07....d38 /dev/sdc


ls -alh /dev/disk/by-id/

scsi-35000cca07....d38 -> ../../sdc
scsi-35000cca07....d38-part1 -> ../../sdc1
scsi-35000cca07....d38-part9 -> ../../sdc9

wwn-0x5000cca07....d38 -> ../../sdc
wwn-0x5000cca07....d38-part1 -> ../../sdc1
wwn-0x5000cca07....d38-part9 -> ../../sdc9

Since there are 2 references for the same disk, do I care about scsi-35000cca07....d38 or wwn-0x5000cca07....d38?


blkid
/dev/sdc1: LABEL="HHproxVM" UUID="1464967671323....067" UUID_SUB="9827066056999849589" BLOCK_SIZE="512" TYPE="zfs_member" PARTLABEL="zfs-97d5289003e3eac8" PARTUUID="e0e83999-e8e8-2744-a08c-dfe90e88f766"
/dev/sdc9: PARTUUID="1403d293-4a58-5f47-ab3a-63a4c6559949"
Is there anything I want from here?
Why sdc doesn thave a UUID and only it's partitions do?
Is UUID different than GUID and how to find GUID if preferred?


zpool status
State : Degraded
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.

scsi-35000cca07....d38 FAULTED 0 34 (Write) 0 too many errors (The one to be replaced)


2.Set the failed disk offline
zpool offline HHVM scsi-35000cca07....d38
Does it need -f option to force it?
Does the failed hdd needs to be mentioned otherwise, like UUid for example found from one of the above commands?
Do I need afterwards to remove the disk as a well with zpool remove HHVM scsi-35000cca07....d38 ?


3.Physically change the hard drives (online if you have hot swap or shutdown first if you haven't)

Now 4a. as the next step or 4b or no step 4 since the failed drive is not a bootable one??
4a.create an empty GPT Partition Table on the new hdd with parted:
Do I need to create a GPT partition? Also the failed disk as well as all the other participating in raid 10 have 2 partitions each, not one.
Do I need to replicate the partition table to the new disk since it is not a bootable one?

parted /dev/new-disk
(parted)# print
(parted)# mklabel GPT
(parted)# Yes
(parted)# q

or

4b.Copy the partition table from a mirror member to the new one
sgdisk /dev/disk/by-id/mirror_member_drive -R /dev/disk/by-id/new_drive or????
sgdisk /dev/disk/by-id/new_drive -R /dev/disk/by-id/mirror_member_drive

lsblk (check afterwards if both disks have the same amount of partitions. sda/sda1,sda2,sda3 and sdb/sdb1,sdb2,sdb3)

sgdisk -G /dev/disk/by-id/new_drive
Used for generalizing the new drive's id, because it copied the id of the current member with the above command?


5.Replace the old disk with the new one
zpool replace -f pool_name old_drive new_drive so
zpool replace -f HHVM scsi-35000cca07....d38 new_disk (not in the office yet to check the name of the new one)
Do I use replace option with the disk or a partition of a disk, since all members have two partitions?
So do I need to issue two replace commands one for partition 1 and another for partition 9?


6.Monitoring status
zpool status -v


Please confirm or add extra steps needed in order to have it as well documented as possible.

 

[ieronymous]

New edit :
I tried to somehow replicate my work's environment and did install prox on a test server and created a mirror storage for Vms
called vmpool upon x2 1Tb disks. Afterwards I off-lined one of the two disks simulating the error condition of the disk, shut down the server, replaced the disk and did the following.

1.write down info about the failed drive

blkid
/dev/sdb1: LABEL="vmpool" UUID="16466954564705881042" UUID_SUB="11993938700724391598" BLOCK_SIZE="4096" TYPE="zfs_member" PARTLABEL="zfs-6a8166f17f550b4a" PARTUUID="240325bf-7f52-e444-a2f6-86731f012696"
/dev/sdb9: PARTUUID="44dc74dd-8d66-884f-9755-70a28586a5aa"
Didn t use anywhere that UUID info It was about the partitions after all.

ls -alh /dev/disk/by-id/
wwn-0x50014ee20ebe22d0 -> ../../sdb
wwn-0x50014ee20ebe22d0-part1 -> ../../sdb1
wwn-0x50014ee20ebe22d0-part9 -> ../../sdb9
ata-WDC_WD1003FZEX-00K3CA0_WD-WCC6Y1ZU54UU -> ../../sdb
ata-WDC_WD1003FZEX-00K3CA0_WD-WCC6Y1ZU54UU-part1 -> ../../sdb1
ata-WDC_WD1003FZEX-00K3CA0_WD-WCC6Y1ZU54UU-part9 -> ../../sdb9
Felt right to keep the ata-WDC_WD1003FZEX-00K3CA0_WD-WCC6Y1ZU54UU naming scheme instead of wwn-0x50014ee20ebe22d0 one

2.zpool offline vmpool ata-WDC_WD1003FZEX-00K3CA0_WD-WCC6Y1ZU54UU (the aforementioned failed disk)
Didn t use remove or detach command for that disk (didn t find any info about that being mandatory)

3.Physically replace the failed drive with a new one

4.Re run ls -alh /dev/disk/by-id/ to find id for the new disk
ata-WDC_WD1003FZEX-00K3CA0_WD-WCC6Y6YERD3K old drive
ata-WDC_WD1003FZEX-00K3CA0_WD-WCC6Y1ZU54UU new drive

5.GO to gui node->disks and wipe disk. Just that, dont initialize it with a gpt partition.
Replace procedure will do it automatically. Just the new or used drive to be unallocated - one single partition.

6.Issue the command to replace the old drive with the new one
zpool replace -f pool_name old_drive new_drive
zpool replace -f vmpool ata-WDC_WD1003FZEX-00K3CA0_WD-WCC6Y6YERD3K ata-WDC_WD1003FZEX-00K3CA0_WD-WCC6Y1ZU54UU

7.Check the reselvering process with
zpool status -v vmpool

8.After resilvering process run a scrub to the pool as well
zpool scrub vmpool

9.If you want to be extra cautious run a long smart test

So didn t use at all step 4a / 4b of the first post and the partitions were made automatically. A big one and another one zfs reserves of 8mb size.

Do you have anything to add / remove from the steps at least for none bootable drives?
I would appreciate if you focus your answers to my first post which has the most questions.

Thank you