Adding 2nd boot drive: is anything needed on partition 1?

[MrPete]

I see info discussing the Installer layout of ProxMox EFI boot drives, with 3 partitions:

• a 1 MB BIOS Boot Partition (gdisk type EF02)
• a 512 MB EFI System Partition (ESP, gdisk type EF00)
• a third partition spanning the set hdsize parameter or the remaining space used for the chosen storage type

I want to add a second mirror drive. There are complex methods, but I'm seeking something simple for myself and others. To that end, a few questions:

1) Is there a script to duplicate what the Installer does, on another drive?

2) Does any content need to be injected into the first 1 MB partition?

3) Do I have it right that adding the 2nd partition via proxmox-boot-tool is really all that's needed there, once the partition is defined?
(eg proxmox-boot-tool format /dev/<drive>2 )

Last step is adding partition 3 to the existing zfs making a mirror. I'll finish up this straightforward set of steps once I have these nailed down.

 

[Dunuin]

You basically have to follow the chapter "Adding a failed bootable device" but instead of running that "zpool replace" you do a "zpool attach" to turn that single disk ZFS pool into a mirror: https://pve.proxmox.com/wiki/ZFS_on_Linux#_zfs_administration

Partition 1 is needed in case you are booting in legacy mode using grub instead of EFI.

 

[MrPete]

You basically have to follow the chapter "Adding a failed bootable device"...

There is a SERIOUS error in the instructions on the wiki! I'm working my way back to repair the damage...
sgdisk <healthy bootable device> -R <new device>

SHOULD be
sgdisk --replicate=<new device> <healthy bootable device>

Supposedly the wiki version works. Did NOT for me. It wiped my partition table.

Best practice: don't use sgdisk at all. Instead use the interactive gdisk version:

gdisk <healthy bootable device>
(use p and/or i to verify you're looking at the right device)
(use b and save a copy of the partition table. In fact, save it on another drive -- GREAT emergency recovery info!)
(quit)
gdisk <new device>
(use p and/or i to verify this is actually a blank device)
(use r to enter recovery menu)
(use l to load the backup partition table you saved)
(use g to change the disk GUID)
(now examine the partition table and ensure all is well. If the disk is not identical you may need to adjust the 3rd partition size! Remember to reserve at least 30-35 sectors at the end for the backup GPT table.)

 

[Dunuin]

-R, --replicate=second_device_filename
Replicate the main device's partition table on the specified second device. Note that the replicated partition table is an exact copy, including all GUIDs; if the device should have its own unique GUIDs, you should use the -G option on the new disk.

So "sgdisk <healthy bootable device> -R <new device>" will clone the partition table from "<healthy bootable device>" (so the existing and working disk with PVE on it) to "<new device>" (your empty or factory-prepartitioned new disk).

Reverse that and you will wipe youe working PVE installation.

Maybe you just got confused and used the wrong disks/partitions with those commands?
Everyone else had no problems doing the same stuff with those commands.

 

[MrPete]

OK, I have rescued my partitions*** and am up and running again. However, there's still this nagging question...

So "sgdisk <healthy bootable device> -R <new device>" will clone the partition table from "<healthy bootable device>" (so the existing and working disk with PVE on it) to "<new device>" (your empty or factory-prepartitioned new disk).

Reverse that and you will wipe youe working PVE installation.

Maybe you just got confused and used the wrong disks/partitions with those commands?
Everyond else had no problems doing the same stuff with those commands.

I understand. That's honestly how it is supposed to work. And I seriously questioned my sanity.
Honestly, it didn't work that way for me. Makes me nervous about sgdisk and/or parameter parsing and/or something deeper in Linux...or in the hardware/firmware. Why would I suggest this?

As a reasonably experienced person with these tools (been doing disks at a low level for many decades, I have reprocessed everything in my head and logs etc.

There is one anomaly that stands out for me even now, and it made me nervous from the get-go:

• Platform I'm on: HP Z2 G5 SFF
• Motherboard has two NVMe slots, labeled SSD0 and SSD1
• Original (HP/Samsung) storage is in SSD0
• I added new (Crucial) storage in SSD1
• In the BIOS, strangely, it is always listing the Crucial first. Even in the list of PCIe channels.
• In ProxMox Linux, /dev/nvme1n1 is the Samsung, /dev/nvme0n1 is the Crucial
• Using SystemRescue 11.00 the devices are swapped: /dev/nvme0n1 is the Samsung, /dev/nvme1n1 is the Crucial

That just made me nervous.
To be safe in the future with such a crucial step, I'm going to do it by saving the partition table in a file off-disk, then load into the new disk. gdisk does that quite nicely.

*** For those who are interested: how to rescue the disk partitions on a ProxMox EFI boot drive formatted in the standard way:

• As of today, while the ProxMox installer has a "rescue" mode, it assumes you have good partitions on your boot drive
• Any of several bootable-USB rescue systems can take care of things. I used SystemRescue

Partition Setup is as follows:
Number, Start (Sector), End (sector), Type Code
1, 34, 2047, EF02
2, 2048, 1050623, EF00
3, 1050624, <end>, BF01

Note: the type codes are a gdisk/sgdisk short hand for standard GPT UUID's. The lookup table is available online.

Steps to recovery:

• Do you have the GUID of your boot disk? If so, you can 100% recover without having any other data. The one absolutely necessary tool is gdisk. (NOTE: I tried using testdisk -- it seriously was confused by the drive. A waste of time.)
• Install a recovery system on a USB and configure the host hardware so you can boot from the USB

General Sequence:

1. Ensure you're working on the correct disk, Run gdisk
2. (Create a blank GPT table if needed)
3. Change sector alignment to 1 to load the standard table
4. Create all partitions
5. Change sector alignment to 8 (the standard used by ProxMox)
6. If you have the disk GUID, set it
7. Verify correctness
8. Write to disk and exit
9. Done

The above steps are not too hard:

1. lsblk shows all devices. Run gdisk /dev/<device>
2. use p to verify no partitions. Use o to make fresh GPT table if needed.
3. use x for expert menu, then l ("L") and 1 to set alignment to 1. Now "m" back to normal menu.
4. "n" for new, provide partition number (1 to 3), start and end sectors, and type code. For partition 3 accept the default end-of-disk number.
5. use x for expert menu, then l ("L") and 8 to set alignment to 8.
6. If you have disk GUID, "g" in expert menu lets you set it.
7. "p" in any menu prints the restored partition table and disk GUID. Verify everything.
8. "w" writes to disk and exits.
9. Reboot as normal!

 

[Dunuin]

There is one anomaly that stands out for me even now, and it made me nervous from the get-go:

• Platform I'm on: HP Z2 G5 SFF
• Motherboard has two NVMe slots, labeled SSD0 and SSD1
• Original (HP/Samsung) storage is in SSD0
• I added new (Crucial) storage in SSD1
• In the BIOS, strangely, it is always listing the Crucial first. Even in the list of PCIe channels.
• In ProxMox Linux, /dev/nvme1n1 is the Samsung, /dev/nvme0n1 is the Crucial
• Using SystemRescue 11.00 the devices are swapped: /dev/nvme0n1 is the Samsung, /dev/nvme1n1 is the Crucial

That just made me nervous.

I always use "/dev/disk/by-id/..." instead of "/dev/sdX" or "/dev/nvmeXnY". That way it doesn't matter how those disks are enumerated and you won't do stuff on the wrong disk if its something like "/dev/disk/by-id/nvme-Samsung-SomeModel-SomeSerial" instead on just "/dev/nvme1n1".

 

[MrPete]

A few hints maybe should make their way into that part of the wiki at some point. Obviously, overwriting partition tables is Dangerous Ground

Something perhaps-safer: here's how to set up a drive the ProxMox Way, without doing a copy at all

#find the device
ls /dev/disk/by-id
# (choose your drive. No question which one. Safer to use that throughout the following instead of /dev/sdx)

#initialize GPT
sgdisk /dev/sdx -o

# define the partitions, using 1 sector alignment to get it exact (NOTE: all will have new ID's, as is appropriate)
sgdisk /dev/sdx -a 1 -n 1:34:2047 -t 1:ef02
sgdisk /dev/sdx -a 1 -n 2::1050623 -t 2:ef00
sgdisk /dev/sdx -a 1 -n 3 -t 3:bf01

# if you have a specific GUID, set it:
sgdisk /dev/sdx -U <guid>

# verify result
sgdisk /dev/sdx -p

For testing...

# create a test file. Use ./test.img instead of /dev/sdx in the above...
truncate -S 100G test.img

Save and restore partition tables

sgdisk /dev/sdx -b ./part-backupfile.bin # save existing part'n table to disk
sgdisk /dev/sdx -l ./part-backupfile.bin # load part'n table from backup file

 

[Dunuin]

I agree, backing up partition tables would be a good idea.

 

[MrPete]

Final notes:
Once the above (see "A few hints..." (NON-copying) sgdisk sequence is performed to create the new GPT and three partitions on the new disk) is complete, the job is finished after these commands (change sdx to your drive, or better to disk/by-id/xxxx)

proxmox-boot-tool format /dev/sdx
proxmox-boot-tool init /dev/sdx

I have now finished, including migrating rpool to the second (slightly smaller and therefore not-quite directly compatible) NVMe, then creating a mirror. I'll make a new post with the complete instructions. Learning a lot! And now this host is more solid than ever.