Replacing hard disks in a mirror setup

K

koolandrew

Active Member
Jul 7, 2021
35
3
28
25
i am stuck trying to replace a hard drive in two servers where they failed. There are only two drives, so it is setup as a simple mirror. I tried to rename etc. and got nowhere.

I tried to wipe the new disk and got the error below.

Is there any way to add these disks without reinstalling proxmox.

Please advise.

1684160270577.png
 
Can you post the output of the following commands?

Code: 
zpool status -v
fdisk -l
ls -alh /dev/disk/by-id
 
Thanks for the quick reply.

root@ktpve32:~# zpool status -v
pool: rpool
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
scan: scrub repaired 0B in 00:00:25 with 0 errors on Sun Jul 10 00:24:26 2022
config:

NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
16938304517973068656 UNAVAIL 0 0 0 was /dev/disk/by-id/scsi-36d4ae520a6226f00293f990107118dba-part3
scsi-36d4ae520a6226f00293f990c07bd03b6-part3 ONLINE 0 0 0

errors: No known data errors

*****************************************

root@ktpve32:~# fdisk -l
Disk /dev/sda: 1.45 TiB, 1599741100032 bytes, 3124494336 sectors
Disk model: PERC H700
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: B9915AF7-B0C9-41DC-B702-2458768806F1

Device Start End Sectors Size Type
/dev/sda1 34 2047 2014 1007K BIOS boot
/dev/sda2 2048 1050623 1048576 512M EFI System
/dev/sda3 1050624 3124494302 3123443679 1.5T Solaris /usr & Apple ZFS


Disk /dev/zd0: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes


Disk /dev/zd16: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes


Disk /dev/zd32: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes
Disklabel type: dos
Disk identifier: 0x630f8a63

Device Boot Start End Sectors Size Id Type
/dev/zd32p1 * 2048 8388607 8386560 4G 83 Linux
/dev/zd32p2 8390654 41940991 33550338 16G 5 Extended
/dev/zd32p5 8390656 41940991 33550336 16G 82 Linux swap / Solaris

Partition 2 does not start on physical sector boundary.


Disk /dev/zd48: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes


Disk /dev/zd64: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes

*****************************************
root@ktpve32:~# ls -alh /dev/disk/by-id
total 0
drwxr-xr-x 2 root root 200 May 12 15:25 .
drwxr-xr-x 7 root root 140 May 12 15:25 ..
lrwxrwxrwx 1 root root 9 May 12 15:25 scsi-36d4ae520a6226f00293f990c07bd03b6 -> ../../sda
lrwxrwxrwx 1 root root 10 May 12 15:25 scsi-36d4ae520a6226f00293f990c07bd03b6-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 May 15 10:10 scsi-36d4ae520a6226f00293f990c07bd03b6-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 May 12 15:25 scsi-36d4ae520a6226f00293f990c07bd03b6-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 May 12 15:25 wwn-0x6d4ae520a6226f00293f990c07bd03b6 -> ../../sda
lrwxrwxrwx 1 root root 10 May 12 15:25 wwn-0x6d4ae520a6226f00293f990c07bd03b6-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 May 15 10:10 wwn-0x6d4ae520a6226f00293f990c07bd03b6-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 May 12 15:25 wwn-0x6d4ae520a6226f00293f990c07bd03b6-part3 -> ../../sda3
 
i followed the instructions above, but i get stuck again.

zpool replace rpool 16938304517973068656
cannot open '16938304517973068656': no such device in /dev
must be a full path or shorthand device name
 
Seems like you are using a RAID Controller (PERC H700). Are you sure that the new disk is properly attached, and the RAID Controller is properly configured? Because there is currently only one disk showing up in fdisk (sda). It seems like the RAID Controller is bunching your attached disks together and exposing them as one single disk to the system (which is strongly discouraged when using ZFS btw).

The reason why you are not able to wipe /dev/sda is because that is your system disk (and therefore currently in use), so I would refrain from trying that...
 
Last edited:
Thanks, i never would have thought of that. I dont understand how this would have happened but i guess some how when we rebooted the machines the bios enabled the raid. I dont get it.

I will check out the raid settings and hopefully that will resolve it.

thanks
 
It turns out that i reinstalled the drives, and they are now being seen. When i issue those commands i get different results as per below.

The disks are there but it is not seeing them for some reason.

zpool status -v
pool: rpool
state: ONLINE
scan: scrub repaired 0B in 00:00:25 with 0 errors on Sun Jul 10 00:24:26 2022
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
scsi-36d4ae520a6226f00293f990c07bd03b6-part3 ONLINE 0 0 0

errors: No known data errors
root@ktpve32:~# fdisk -l
Disk /dev/sda: 1.45 TiB, 1599741100032 bytes, 3124494336 sectors
Disk model: PERC H700
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: B9915AF7-B0C9-41DC-B702-2458768806F1

Device Start End Sectors Size Type
/dev/sda1 34 2047 2014 1007K BIOS boot
/dev/sda2 2048 1050623 1048576 512M EFI System
/dev/sda3 1050624 3124494302 3123443679 1.5T Solaris /usr & Apple ZFS


Disk /dev/zd0: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes


Disk /dev/zd16: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes


Disk /dev/zd32: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes
Disklabel type: dos
Disk identifier: 0x630f8a63

Device Boot Start End Sectors Size Id Type
/dev/zd32p1 * 2048 8388607 8386560 4G 83 Linux
/dev/zd32p2 8390654 41940991 33550338 16G 5 Extended
/dev/zd32p5 8390656 41940991 33550336 16G 82 Linux swap / Solaris

Partition 2 does not start on physical sector boundary.


Disk /dev/zd48: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes


Disk /dev/zd64: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes
root@ktpve32:~# ls -alh /dev/disk/by-id
total 0
drwxr-xr-x 2 root root 200 May 17 08:41 .
drwxr-xr-x 7 root root 140 May 17 08:41 ..
lrwxrwxrwx 1 root root 9 May 17 08:42 scsi-36d4ae520a6226f00293f990c07bd03b6 -> ../../sda
lrwxrwxrwx 1 root root 10 May 17 08:42 scsi-36d4ae520a6226f00293f990c07bd03b6-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 May 17 08:42 scsi-36d4ae520a6226f00293f990c07bd03b6-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 May 17 08:42 scsi-36d4ae520a6226f00293f990c07bd03b6-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 May 17 08:42 wwn-0x6d4ae520a6226f00293f990c07bd03b6 -> ../../sda
lrwxrwxrwx 1 root root 10 May 17 08:42 wwn-0x6d4ae520a6226f00293f990c07bd03b6-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 May 17 08:42 wwn-0x6d4ae520a6226f00293f990c07bd03b6-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 May 17 08:42 wwn-0x6d4ae520a6226f00293f990c07bd03b6-part3 -> ../../sda3
 
There still seems to be only one hard disk available (/dev/sda) - are you sure that the setup on the RAID controller is fine now?
 
Hi, the raid controller wasn't the issue, it was the virtualization. Now the VD has been created and new device added to zpool/rpool on both machines.

I hope that this will be an easier task going forward if other drives fail.
 
You must log in or register to reply here.
Top Bottom