Disk replacement procedure for a ZFS Raid 1 install.

Q

QPidduck

New Member
Jan 8, 2014
17
1
1
Hi team,

We've (prior to 3.4) had out Proxmox OS's on a separate SSD and the Data managed by a Hardware raid controller.
The concept of a ZFS raid one install is appealing as if helps us out of a tricky spot if we have a failure with the single SSD.

Initial testing has shown that selecting ZFS RAID 1 during install (onto a 1TB disk) creates on each disk 3 Partitions
1. 1MB Grub Boot partition
2. 128MB EFI-System Partition
3. Remainder PVE-ZFS-Partition (in RPOOL mirror with other disk)

My assumption is that there is no mirroring of the other partitions so a kernal update etc wouldn't be reflected in the grub of the 2nd disk (please correct me if i'm wrong.)

After simulating a disk failure I can install a new disk and partition it in a way where the 3rd partition is happy in a mirror again, but short of 'dd' ing the part-table and the other partitions I can't find a way to bring my replacement disk to the point where I could boot from it if the other original disk were to now fail.

Is there something I'm not seeing should we look at better ways to manage the non-pool partitions on a ZFS Raid 1 install?

Many thanks,
Quentin Pidduck
Technologywise Ltd.
Tauranga, New Zealand.
 
I guess you boot with grub? Ifs so, simply install grub on the new disk:

# grub-install /dev/thenewdisk
 
Grub is the default for that set-up yes, but I disagree your approach is the best way to handle this.

1) If I install Proxmox in the ZFS raid 1 and spend most the server's life booting from and upgrading software on /dev/sda (or bios preferred boot device) would this not mean I have to re-install grub on the 2nd disk after every-time a new kernel version is applied using apt?

2) If my primary disk fails I would need to install it's replacement while running off the remaing disk then dd 'ing the partition table and first 129MB from the remaining to the replacement disk. Seems like a major doesn't it?
 
QPidduck said:
Grub is the default for that set-up yes, but I disagree your approach is the best way to handle this.
Click to expand...

Grub should be already configured to install on both devices. If not, simply run

# grub-pc --reconfigure

and select both devices.
 
Thanks for the quick reply,

Any thoughts on the EFI-System-Partition on both original disks and how best to recreate that on a replacement disk?
 
QPidduck said:
Any thoughts on the EFI-System-Partition on both original disks and how best to recreate that on a replacement disk?
Click to expand...

I guess you just need to copy the ESP. But it is unclear to me how EFI booting should work with mirrored disk - I found no good documentation about that.
 
I'll have to do some testing then.
Would Proxmox benefit from a script that assists users in introducing a new disk into a broken ZFS Raid-1 installation then running everything required to bring it to a bootable state?
 
Initial tests are still working well, but i need a bit of input.

Our servers recognise hdd's as sda,sdb,sdc etc.
I know old IDE based systems use hdX
What other drive identifiers should I include?

Also our servers report multiple instances of a Sata disk in /dev/disk/by-id because of scsi emulation.
eg.
root@XXXXXXXXXXXXXXX:/dev/disk/by-id# ls /dev/disk/by-id/ | grep WX20C7981909
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909-part1
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909-part2
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909-part3
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909-part1
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909-part2
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909-part3

Am i better to address a disk via ata or scsi?
 
hi, any news on this? i'm looking also for a simple solution to replace a failing disk in a mirror installation.
if the script could not be done, is possible to have a step by step guide for this?
btw. i cannot find "# grub-pc --reconfigure" on my installation. has that changed on last releases?

thanks
 
Can confirm.

Not sure if I have my scripts to automate the process around anymore. Please let me know if I should dig for them.
 
I know it is a littel pit late, but i do this easy way. For the first copy the partitiontable from another disk in the same pool. For example. SDB is the new disk:
Code: 
sgdisk -R /dev/sdb /dev/sda
So everything including grub is on the new disk.
The UUID is copy with, so we must set a new one:
Code: 
sgdisk -G /dev/sdb
And finaly start the resliverprocess:
Code: 
zpool replace -f rpool sdb2 sdb2
Make sure to wait until resilver is done before rebooting. Diskbyid are also possible. (/dev/disk/by-id/)
 
  • Like
Reactions: tl5k5 and chrone
Nice. I did not know about the sgdisk utility. I ended up manually copying the GPT partitions by comparing the output of gdisk 'p' command on other systems. I don't think it would have been good here since I was adding a larger drive, and would need a larger partition 2.

Basically, I created with gdisk utility: 1G BIOS boot partition (EF02), "rest of disk" size ZFS partition (BF01), and an approximately 8G "reserved" partition (BF07).

To make the first partition, I had to go to expert mode and set alignment to 1. I have no idea why proxmox creates the partition 9 reserved space.

Then I just ran grub-install and then did the zpool attach (I was adding a drive).
 
I did not test with auto expand. From what I read that does not work on root pool. I had to reboot, then run "zpool online -e DEVICE" then it grew to full size.
 
is zfs support natively now? and how do you manage to do a migration to a raid zfs without wiping out the system?
Why zfs would it bt better than mdadm since mdadm can do the same thing since you can actually do a raid with partition and not disk?
 
You must log in or register to reply here.
Top Bottom