How to add drive to increase redundancy to Proxmox 5.0 host with ZFS

C

Chicken76

Renowned Member
Jun 26, 2017
57
2
73
46
I had only one hard drive (/dev/sda) when I installed Proxmox 5.0 and I chose ZFS for storage.
Now I've bought and added another identical drive (/dev/sdb) and want to turn rpool into a mirror and make both drives bootable so the machine can survive a drive failure.

Here's how /dev/sda looks now:
Code: 
root@pve:/dev# gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk /dev/sda: 1465149168 sectors, 698.6 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 2FF979EB-653C-48FB-92B3-51898EBE979C
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1465149134
Partitions will be aligned on 8-sector boundaries
Total free space is 0 sectors (0 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02
   2            2048      1465132749   698.6 GiB   BF01  zfs
   9      1465132750      1465149134   8.0 MiB     BF07


I would like some help with the steps to be taken in order to achieve this. What I'm thinking of is this:

1. Initialize /dev/sdb with a gpt partition table and create partitions identical to what's on /dev/sda now.
Code: 
sgdisk /dev/sda -R /dev/sdb
sgdisk -G /dev/sdb

2. Add /dev/sdb2 to rpool as a mirror of /dev/sda2
Code: 
zpool attach rpool /dev/sda2 /dev/sdb2
Can/should I use UUID instead of /dev/sdb2, to make it resistant to drives being initialized in a different order, or cable swaps?

3. Fix boot loader. I need help with this. I'd like it to be able to boot in case either drive dies.

4. What do I need to do about the third partition? I see it's only 8 MB big. What is it used for?
 
Chicken76 said:
I had only one hard drive (/dev/sda) when I installed Proxmox 5.0 and I chose ZFS for storage.
Now I've bought and added another identical drive (/dev/sdb) and want to turn rpool into a mirror and make both drives bootable so the machine can survive a drive failure.

...

I would like some help with the steps to be taken in order to achieve this. What I'm thinking of is this:
Click to expand...

while your plan below looks reasonable, I recommend playing through the scenario in a VM a few times for you to be comfortable.

1. Initialize /dev/sdb with a gpt partition table and create partitions identical to what's on /dev/sda now.
Code: 
sgdisk /dev/sda -R /dev/sdb
sgdisk -G /dev/sdb

2. Add /dev/sdb2 to rpool as a mirror of /dev/sda2
Code: 
zpool attach rpool /dev/sda2 /dev/sdb2
Can/should I use UUID instead of /dev/sdb2, to make it resistant to drives being initialized in a different order, or cable swaps?
Click to expand...
you can, but it is not necessary.

3. Fix boot loader. I need help with this. I'd like it to be able to boot in case either drive dies.
Click to expand...
update-grub followed by grub-install /dev/sdb (but again, I recommend testing this part in a virtual environment!)

4. What do I need to do about the third partition? I see it's only 8 MB big. What is it used for?
Click to expand...

it is mainly used as a marker in ZFS internally to signify that this disk as a whole was formatted by ZFS (it is used to set certain internal parameters, and I think also for autoexpand?).
 
fabian said:
while your plan below looks reasonable, I recommend playing through the scenario in a VM a few times for you to be comfortable.

you can, but it is not necessary.

update-grub followed by grub-install /dev/sdb (but again, I recommend testing this part in a virtual environment!)

it is mainly used as a marker in ZFS internally to signify that this disk as a whole was formatted by ZFS (it is used to set certain internal parameters, and I think also for autoexpand?).
Click to expand...

Thank you for your answers. I have followed your advice and installed Proxmox in a VM with a single drive (scsi0), shut down, added a second drive (scsi1), performed steps 1,2 and 3. So far so good. Rpool is now mirrored.

Then I shut down the VM and modified the boot order and set the first boot device from 'scsi0' to 'scsi1', and the VM doesn't boot. ('scsi0' is not in the boot order)

inze4I8.jpg


I think something extra needs to be done to make it boot from the second drive, in addition to the grub stuff.
Doesn't grub write only the master boot record? Something should be done to /dev/sdb1, should I dd sda1 into it?
 
Chicken76 said:
Thank you for your answers. I have followed your advice and installed Proxmox in a VM with a single drive (scsi0), shut down, added a second drive (scsi1), performed steps 1,2 and 3. So far so good. Rpool is now mirrored.

Then I shut down the VM and modified the boot order and set the first boot device from 'scsi0' to 'scsi1', and the VM doesn't boot. ('scsi0' is not in the boot order)

inze4I8.jpg


I think something extra needs to be done to make it boot from the second drive, in addition to the grub stuff.
Doesn't grub write only the master boot record? Something should be done to /dev/sdb1, should I dd sda1 into it?
Click to expand...

sorry, forgot to mention that virtio-scsi only allows booting from the first disk. choose the LSI scsi controller for this test ;)
 
  • Like
Reactions: GadgetPig
fabian said:
it is mainly used as a marker in ZFS internally to signify that this disk as a whole was formatted by ZFS (it is used to set certain internal parameters, and I think also for autoexpand?).
Click to expand...


From what I remember (maybe I am wrong), this let say extra partition have 2 functions:
1. to be compatible with some unix-like disro(who use this 2 partition schema), so it will be possible to import a pool in unix (with all hdd space alocated for zfs)
2. to be able to replace / add a new hdd to the pool with the same capacity, but with a minor difference (usually is about under 1 Mb size difference )

For sure this extra partition has nothing to do with auto-expand .
 
fabian said:
sorry, forgot to mention that virtio-scsi only allows booting from the first disk. choose the LSI scsi controller for this test ;)
Click to expand...
I used the 'SCSI' controller. This is how I set up the virtual drives:
cdGnChF.png


I chose SCSI out of the options in the pic below because the guides available online recommended it for Linux guests. Although not related to this thread, does the SCSI controller recommendation still stand for version 5.0?
fOZR9Ak.png
 
yes. but in addition to the bus type, you can also set the controller type for SCSI (und options). there, virtio-scsi (default for linux guests) or virtio-scsi-single (if you want iothread support) is recommended for linux guests - but it has the limitation of only allowing boot from scsi0 (which is normally not a problem in VMs - but for tests like here it matters).
 
fabian said:
yes. but in addition to the bus type, you can also set the controller type for SCSI (und options). there, virtio-scsi (default for linux guests) or virtio-scsi-single (if you want iothread support) is recommended for linux guests - but it has the limitation of only allowing boot from scsi0 (which is normally not a problem in VMs - but for tests like here it matters).
Click to expand...
Ah, I see now what you meant. I've tried LSI 53C810 and it didn't work (wouldn't boot), then tried LSI 53C895A and it worked! I've even removed 'scsi0' virtual drive from the VM and it booted from 'scsi1' fine.
Now off to perform the same thing on the real machine.
Thank you again for your help.
 
guletz said:
From what I remember (maybe I am wrong), this let say extra partition have 2 functions:
1. to be compatible with some unix-like disro(who use this 2 partition schema), so it will be possible to import a pool in unix (with all hdd space alocated for zfs)
2. to be able to replace / add a new hdd to the pool with the same capacity, but with a minor difference (usually is about under 1 Mb size difference )

For sure this extra partition has nothing to do with auto-expand .
Click to expand...
So since I don't plan to take these hard drives to any UNIX machine, would you say I don't need to do anything about that partition? Should I delete it from /dev/sdb as it has nothing in it (partition table was cloned) to avoid confusing any zfs utilities?
 
Chicken76 said:
So since I don't plan to take these hard drives to any UNIX machine, would you say I don't need to do anything about that partition? Should I delete it from /dev/sdb as it has nothing in it (partition table was cloned) to avoid confusing any zfs utilities?
Click to expand...

I do not know the answer. But I do not see what will be the reason to delete this extra partition.
 
guletz said:
I do not know the answer. But I do not see what will be the reason to delete this extra partition.
Click to expand...


Now I know the corect answer:


Brian Behlendorf (ZoL dev/maintainer):
"Partition #9 is created by the tools to be consistent with the Illumos
behavior but is otherwise completely unused. You should be able to
remove it safely without ill effect at least when using ZoL. I'm not
sure how things will behave on other platforms."

https://groups.google.com/a/zfsonlinux.org/forum/#!msg/zfs-discuss/gedcS4rfoDg/Eo4hOwWPxuYJ
 
You must log in or register to reply here.
Top Bottom
Back