Disk replacement procedure for a ZFS Raid 1 install.

[QPidduck]

Hi team,

We've (prior to 3.4) had out Proxmox OS's on a separate SSD and the Data managed by a Hardware raid controller.
The concept of a ZFS raid one install is appealing as if helps us out of a tricky spot if we have a failure with the single SSD.

Initial testing has shown that selecting ZFS RAID 1 during install (onto a 1TB disk) creates on each disk 3 Partitions
1. 1MB Grub Boot partition
2. 128MB EFI-System Partition
3. Remainder PVE-ZFS-Partition (in RPOOL mirror with other disk)

My assumption is that there is no mirroring of the other partitions so a kernal update etc wouldn't be reflected in the grub of the 2nd disk (please correct me if i'm wrong.)

After simulating a disk failure I can install a new disk and partition it in a way where the 3rd partition is happy in a mirror again, but short of 'dd' ing the part-table and the other partitions I can't find a way to bring my replacement disk to the point where I could boot from it if the other original disk were to now fail.

Is there something I'm not seeing should we look at better ways to manage the non-pool partitions on a ZFS Raid 1 install?

Many thanks,
Quentin Pidduck
Technologywise Ltd.
Tauranga, New Zealand.

 

[dietmar]

I guess you boot with grub? Ifs so, simply install grub on the new disk:

# grub-install /dev/thenewdisk

 

[QPidduck]

Grub is the default for that set-up yes, but I disagree your approach is the best way to handle this.

1) If I install Proxmox in the ZFS raid 1 and spend most the server's life booting from and upgrading software on /dev/sda (or bios preferred boot device) would this not mean I have to re-install grub on the 2nd disk after every-time a new kernel version is applied using apt?

2) If my primary disk fails I would need to install it's replacement while running off the remaing disk then dd 'ing the partition table and first 129MB from the remaining to the replacement disk. Seems like a major doesn't it?

 

[dietmar]

Grub is the default for that set-up yes, but I disagree your approach is the best way to handle this.

Grub should be already configured to install on both devices. If not, simply run

# grub-pc --reconfigure

and select both devices.

 

[QPidduck]

Thanks for the quick reply,

Any thoughts on the EFI-System-Partition on both original disks and how best to recreate that on a replacement disk?

 

[dietmar]

Any thoughts on the EFI-System-Partition on both original disks and how best to recreate that on a replacement disk?

I guess you just need to copy the ESP. But it is unclear to me how EFI booting should work with mirrored disk - I found no good documentation about that.

 

[QPidduck]

I'll have to do some testing then.
Would Proxmox benefit from a script that assists users in introducing a new disk into a broken ZFS Raid-1 installation then running everything required to bring it to a bootable state?

 

[dietmar]

Would Proxmox benefit from a script that assists users in introducing a new disk into a broken ZFS Raid-1 installation then running everything required to bring it to a bootable state?

yes.

 

[QPidduck]

Working on something now. Looking very promising.

 

[QPidduck]

Initial tests are still working well, but i need a bit of input.

Our servers recognise hdd's as sda,sdb,sdc etc.
I know old IDE based systems use hdX
What other drive identifiers should I include?

Also our servers report multiple instances of a Sata disk in /dev/disk/by-id because of scsi emulation.
eg.
root@XXXXXXXXXXXXXXX:/dev/disk/by-id# ls /dev/disk/by-id/ | grep WX20C7981909
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909-part1
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909-part2
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909-part3
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909-part1
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909-part2
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909-part3

Am i better to address a disk via ata or scsi?

 

[shutt1e]

hi, any news on this? i'm looking also for a simple solution to replace a failing disk in a mirror installation.
if the script could not be done, is possible to have a step by step guide for this?
btw. i cannot find "# grub-pc --reconfigure" on my installation. has that changed on last releases?

thanks

 

[QPidduck]

Hi Shutt1e.

I've written a script, but am awaiting further input before I release it to you all,

All the commands are the same as outlined here
http://forum.proxmox.com/archive/index.php/t-20912.html

hope that helps you in the meantime!

Cheers,
Quentin Pidduck.

 

[vkhera]

All the commands are the same as outlined here
http://forum.proxmox.com/archive/index.php/t-20912.html

In the new forum, the above link is broken. What was the original topic called (or even better the updated link)?

 

[vkhera]

I think this is the new link: https://forum.proxmox.com/threads/replace-disk-on-zfs-pool-i-did-not-look-through.20912/

 

[QPidduck]

Can confirm.

Not sure if I have my scripts to automate the process around anymore. Please let me know if I should dig for them.

 

[fireon]

I know it is a littel pit late, but i do this easy way. For the first copy the partitiontable from another disk in the same pool. For example. SDB is the new disk:

sgdisk -R /dev/sdb /dev/sda

So everything including grub is on the new disk.
The UUID is copy with, so we must set a new one:

sgdisk -G /dev/sdb

And finaly start the resliverprocess:

zpool replace -f rpool sdb2 sdb2

Make sure to wait until resilver is done before rebooting. Diskbyid are also possible. (/dev/disk/by-id/)

 

[vkhera]

Nice. I did not know about the sgdisk utility. I ended up manually copying the GPT partitions by comparing the output of gdisk 'p' command on other systems. I don't think it would have been good here since I was adding a larger drive, and would need a larger partition 2.

Basically, I created with gdisk utility: 1G BIOS boot partition (EF02), "rest of disk" size ZFS partition (BF01), and an approximately 8G "reserved" partition (BF07).

To make the first partition, I had to go to expert mode and set alignment to 1. I have no idea why proxmox creates the partition 9 reserved space.

Then I just ran grub-install and then did the zpool attach (I was adding a drive).

 

[fireon]

Should work with autoexpand. Not tested with rpool, but works fine on an extrapool. ZFS Autoreplace and Autoexpand, should it work?

 

[vkhera]

I did not test with auto expand. From what I read that does not work on root pool. I had to reboot, then run "zpool online -e DEVICE" then it grew to full size.

 

[vigilian]

is zfs support natively now? and how do you manage to do a migration to a raid zfs without wiping out the system?
Why zfs would it bt better than mdadm since mdadm can do the same thing since you can actually do a raid with partition and not disk?