Disk replacement procedure for a ZFS Raid 1 install.

QPidduck

New Member
Jan 8, 2014
17
1
1
Hi team,

We've (prior to 3.4) had out Proxmox OS's on a separate SSD and the Data managed by a Hardware raid controller.
The concept of a ZFS raid one install is appealing as if helps us out of a tricky spot if we have a failure with the single SSD.

Initial testing has shown that selecting ZFS RAID 1 during install (onto a 1TB disk) creates on each disk 3 Partitions
1. 1MB Grub Boot partition
2. 128MB EFI-System Partition
3. Remainder PVE-ZFS-Partition (in RPOOL mirror with other disk)

My assumption is that there is no mirroring of the other partitions so a kernal update etc wouldn't be reflected in the grub of the 2nd disk (please correct me if i'm wrong.)

After simulating a disk failure I can install a new disk and partition it in a way where the 3rd partition is happy in a mirror again, but short of 'dd' ing the part-table and the other partitions I can't find a way to bring my replacement disk to the point where I could boot from it if the other original disk were to now fail.

Is there something I'm not seeing should we look at better ways to manage the non-pool partitions on a ZFS Raid 1 install?

Many thanks,
Quentin Pidduck
Technologywise Ltd.
Tauranga, New Zealand.
 
I guess you boot with grub? Ifs so, simply install grub on the new disk:

# grub-install /dev/thenewdisk
 
Grub is the default for that set-up yes, but I disagree your approach is the best way to handle this.

1) If I install Proxmox in the ZFS raid 1 and spend most the server's life booting from and upgrading software on /dev/sda (or bios preferred boot device) would this not mean I have to re-install grub on the 2nd disk after every-time a new kernel version is applied using apt?

2) If my primary disk fails I would need to install it's replacement while running off the remaing disk then dd 'ing the partition table and first 129MB from the remaining to the replacement disk. Seems like a major doesn't it?
 
Grub is the default for that set-up yes, but I disagree your approach is the best way to handle this.

Grub should be already configured to install on both devices. If not, simply run

# grub-pc --reconfigure

and select both devices.
 
Thanks for the quick reply,

Any thoughts on the EFI-System-Partition on both original disks and how best to recreate that on a replacement disk?
 
Any thoughts on the EFI-System-Partition on both original disks and how best to recreate that on a replacement disk?

I guess you just need to copy the ESP. But it is unclear to me how EFI booting should work with mirrored disk - I found no good documentation about that.
 
I'll have to do some testing then.
Would Proxmox benefit from a script that assists users in introducing a new disk into a broken ZFS Raid-1 installation then running everything required to bring it to a bootable state?
 
Initial tests are still working well, but i need a bit of input.

Our servers recognise hdd's as sda,sdb,sdc etc.
I know old IDE based systems use hdX
What other drive identifiers should I include?

Also our servers report multiple instances of a Sata disk in /dev/disk/by-id because of scsi emulation.
eg.
root@XXXXXXXXXXXXXXX:/dev/disk/by-id# ls /dev/disk/by-id/ | grep WX20C7981909
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909-part1
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909-part2
ata-WDC_WD1600HLFS-75G6U1_WD-WX20C7981909-part3
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909-part1
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909-part2
scsi-SATA_WDC_WD1600HLFS-_WD-WX20C7981909-part3

Am i better to address a disk via ata or scsi?
 
hi, any news on this? i'm looking also for a simple solution to replace a failing disk in a mirror installation.
if the script could not be done, is possible to have a step by step guide for this?
btw. i cannot find "# grub-pc --reconfigure" on my installation. has that changed on last releases?

thanks
 
Can confirm.

Not sure if I have my scripts to automate the process around anymore. Please let me know if I should dig for them.
 
I know it is a littel pit late, but i do this easy way. For the first copy the partitiontable from another disk in the same pool. For example. SDB is the new disk:
Code:
sgdisk -R /dev/sdb /dev/sda
So everything including grub is on the new disk.
The UUID is copy with, so we must set a new one:
Code:
sgdisk -G /dev/sdb
And finaly start the resliverprocess:
Code:
zpool replace -f rpool sdb2 sdb2
Make sure to wait until resilver is done before rebooting. Diskbyid are also possible. (/dev/disk/by-id/)
 
  • Like
Reactions: tl5k5 and chrone
Nice. I did not know about the sgdisk utility. I ended up manually copying the GPT partitions by comparing the output of gdisk 'p' command on other systems. I don't think it would have been good here since I was adding a larger drive, and would need a larger partition 2.

Basically, I created with gdisk utility: 1G BIOS boot partition (EF02), "rest of disk" size ZFS partition (BF01), and an approximately 8G "reserved" partition (BF07).

To make the first partition, I had to go to expert mode and set alignment to 1. I have no idea why proxmox creates the partition 9 reserved space.

Then I just ran grub-install and then did the zpool attach (I was adding a drive).
 
I did not test with auto expand. From what I read that does not work on root pool. I had to reboot, then run "zpool online -e DEVICE" then it grew to full size.
 
is zfs support natively now? and how do you manage to do a migration to a raid zfs without wiping out the system?
Why zfs would it bt better than mdadm since mdadm can do the same thing since you can actually do a raid with partition and not disk?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!