First drive in raidz1 about to fail, non-hot-swap

[swimboy]

I have a 5 drive raidz1 array (sdb,sdc,sdd,sde,sdf) and the smart status on sdb indicates that it's about to fail. My server does not have hot-swap bays so I need to shut down the server to pull out the failing disk.

I thought I had installed grub on sdc correctly to assist in this situation, but when I shut down the server, removed sdb, and rebooted, grub gives me the error "couldn't find a necessary member device of multi-device filesystem"

For the time being, I've replaced sdb and rebooted my server successfully. The raid is not currently degraded, but I don't know how much longer this drive will last. What do I do to get the system to boot with sdb removed so I can get a replacement and resilver?

BTW: the sda drive is SSD with ZIL and L2ARC, no boot partition, so I'm pretty sure that it's trying to use grub from sdc.

 

[swimboy]

Also, another data point that may be pertinent: when I set up Proxmox, I found that zfs was referring to my volumes by name (sdb2, sdc2, etc.) and not by vdevs. I posted on the forum and was basically told "if it works, don't worry about it." Is this biting me in the butt now?

 

[fabian]

please post the output of "pveversion -v", "zpool status rpool" and "lsblk /dev/sd*"

 

[swimboy]

root@goat:~# pveversion -v
proxmox-ve: 4.4-84 (running kernel: 4.4.44-1-pve)
pve-manager: 4.4-13 (running version: 4.4-13/7ea56165)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.44-1-pve: 4.4.44-84
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-109
pve-firmware: 1.1-10
libpve-common-perl: 4.0-94
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-1
pve-docs: 4.4-3
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-96
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80

root@goat:~# zpool status rpool
pool: rpool
state: ONLINE
scan: resilvered 500K in 0h12m with 0 errors on Sun Jun 11 00:55:42 2017
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
sdc2 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
sde2 ONLINE 0 0 0
sdf2 ONLINE 0 0 0
logs
sda1 ONLINE 0 0 0
cache
sda2 ONLINE 0 0 0

errors: No known data errors

root@goat:~# lsblk /dev/sd*
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 111.8G 0 disk
├─sda1 8:1 0 16G 0 part
└─sda2 8:2 0 95.8G 0 part
sda1 8:1 0 16G 0 part
sda2 8:2 0 95.8G 0 part
sdb 8:16 0 3.7T 0 disk
├─sdb1 8:17 0 1007K 0 part
├─sdb2 8:18 0 3.7T 0 part
└─sdb9 8:25 0 8M 0 part
sdb1 8:17 0 1007K 0 part
sdb2 8:18 0 3.7T 0 part
sdb9 8:25 0 8M 0 part
sdc 8:32 0 3.7T 0 disk
├─sdc1 8:33 0 1007K 0 part
├─sdc2 8:34 0 3.7T 0 part
└─sdc9 8:41 0 8M 0 part
sdc1 8:33 0 1007K 0 part
sdc2 8:34 0 3.7T 0 part
sdc9 8:41 0 8M 0 part
sdd 8:48 0 3.7T 0 disk
├─sdd1 8:49 0 1007K 0 part
├─sdd2 8:50 0 3.7T 0 part
└─sdd9 8:57 0 8M 0 part
sdd1 8:49 0 1007K 0 part
sdd2 8:50 0 3.7T 0 part
sdd9 8:57 0 8M 0 part
sde 8:64 0 3.7T 0 disk
├─sde1 8:65 0 1007K 0 part
├─sde2 8:66 0 3.7T 0 part
└─sde9 8:73 0 8M 0 part
sde1 8:65 0 1007K 0 part
sde2 8:66 0 3.7T 0 part
sde9 8:73 0 8M 0 part
sdf 8:80 0 3.7T 0 disk
├─sdf1 8:81 0 1007K 0 part
├─sdf2 8:82 0 3.7T 0 part
└─sdf9 8:89 0 8M 0 part
sdf1 8:81 0 1007K 0 part
sdf2 8:82 0 3.7T 0 part
sdf9 8:89 0 8M 0 part

 

[fortechitsolutions]

Hi, just to mention, just because the system does not have nice 'hot swap easy remove sleds' does not inherently mean the system can't support hot-plugging disks, ie,
- rip open case while it is powered up
- un-cable the bad drive
- remove it physically, assuming screws are not inaccessible while system is powered and assembled more or less
- attach new replacement drive
- happy days, proceed

most motherboards since a long time now have supported this without a great deal of fuss. Possibly you will get a new SDX assigned to the disk on the port since it is a 'new device' (ie, old disk removed is SDA, new disk added is SDE - until the next reboot, possibly)

potentially you are better off for a very brief power cycle, but presumably that is not desirable here; which is why you are posting ?

anyhow. Just because it isn't a pretty disk sled layout, does not mean it isn't hot swap. Just means it is more work to get the drive out, usually. (plus you run some risk of "oops, dropped a screw on the motherboard, oops, shorted something out, oopsie".

ie, be careful

If you want to play-practice, use a throw-away box first (ie, old desktop as long as it is <5 years old, SATA cabled disks) and test this out with a simple linux live CD to observe how things are logged in dmesg as you unplug and replug disks while system is up. Loads of fun!


Tim

 

[swimboy]

I'm really not comfortable with this solution. I have to return the existing drive under warranty to get the replacement, and if the system goes down before the new drive is resilvered, I'm back to square one. I feel I really need to get this machine booting with sdb removed.

 

[Ashley]

Have they given you an idea of how long the warranty return may take? Even once you fix the issue that's a long time running with no redundancy..you normally want to get rebuilding the minute you loose a disk.

 

[fabian]

Have they given you an idea of how long the warranty return may take? Even once you fix the issue that's a long time running with no redundancy..you normally want to get rebuilding the minute you loose a disk.

this! since the disk has not failed (yet), I would even recommend adding the new replacement disk first, then removing the old one after the resilvering has finished (that way, you are hopefully redundant during the whole process).

the most likely reason why grub complains after removing the disk is that it already does not see one of the disks in your pool (which is fine, it's a raidz1 after all, so you can boot with N-1 disks). once you remove sdb, grub probably sees N-2 disks, which is not enough to read the pool, and hence not enough to boot. you can verify this by checking how many disks the grub shell sees (with "ls"). usually this is a consequence of the bios boot (order) settings, with many mainboards/bios implementations being limited to a certain number of devices. if there is a disk that is not currently selected as boot disk in your bios, try selecting it (in addition, or if not possible, instead of your "sdb" disk).

 

[swimboy]

Since I don't have any available SATA ports, putting in the new drive before removing the old drive isn't an option. I've double-checked with "ls" at the boot prompt, and all the disks are there. I've also checked my bios settings to make sure that all of the drives in the array are marked as a potential boot disk.

 

[fortechitsolutions]

Possible solution? (a) Buy a new hard drive outright to replace the failing drive (b) remove and replace; let rebuild complete (c) return bad part under warranty, and eventually get a replacement back in the mail (d) now you have a cold-spare drive on hand for quick and easy use the next time a drive inevitably fails, since all drives fail over time...

 

[fabian]

could you please post the following output from the grub rescue shell when you remove sdb?

first list the detected disks and their partitions:

ls

then, run the following the get more output:

set debug=all

then, print the config:

set

and try to continue the boot:

insmod normal

the latter should print debug output, e.g., I get the following on a raidz1 where I removed 2 of 5 disks:

kern/disk.c:196: Opening `hd0' ...
kern/fs.c:56: Detecting zfs...
fs/zfs.c:1192: label ok 0
fs/zfs.c:1192: label ok 1
fs/zfs.c:1192: label ok 2
fs/zfs.c:1192: label ok 3
kern/fs.c:78: zfs detection failed.
kern/disk.c:295: Closing `hd0'.
error: unknown filesystem.

please include all the output!

 

[guletz]

Maybe, you can try this:

- boot your server with the failing disk
- remove the ssd from zfs (cache/slog) - nothing will be broke
- stop the server
- replace the ssd with another disk
- start/boot the server
- in zfs, replace the failing disk withe the new disk (spare)
- after the resilvering process is done, reinstall your grub
- stop the server
- remove the bad disk and add again the ssd
- after boot, re-add the cache/slog SSD

PS: it will be nice to have a spare disk in a closet

 

[Magneto]

Hi, just to mention, just because the system does not have nice 'hot swap easy remove sleds' does not inherently mean the system can't support hot-plugging disks, ie,
- rip open case while it is powered up
- un-cable the bad drive
- remove it physically, assuming screws are not inaccessible while system is powered and assembled more or less
- attach new replacement drive
- happy days, proceed

most motherboards since a long time now have supported this without a great deal of fuss. Possibly you will get a new SDX assigned to the disk on the port since it is a 'new device' (ie, old disk removed is SDA, new disk added is SDE - until the next reboot, possibly)

potentially you are better off for a very brief power cycle, but presumably that is not desirable here; which is why you are posting ?

anyhow. Just because it isn't a pretty disk sled layout, does not mean it isn't hot swap. Just means it is more work to get the drive out, usually. (plus you run some risk of "oops, dropped a screw on the motherboard, oops, shorted something out, oopsie".

ie, be careful

If you want to play-practice, use a throw-away box first (ie, old desktop as long as it is <5 years old, SATA cabled disks) and test this out with a simple linux live CD to observe how things are logged in dmesg as you unplug and replug disks while system is up. Loads of fun!


Tim

I wouldn't recommend this at all. A rackmount server I recently worked on made some sparks on the chassis when I opened it, while running. Luckily nothing bad happened but you never know.