System drive (ZFS) replacement

[listhor]

Currently PVE runs on 256GB NVME M.2 drive installed on PCI adapter. I've purchased Intel HPE IBM P3600 1.6TB HHHL PCI-E NVMe and I would like to install it and replace system drive.
My plan is to remove current drive and connect it via USB3 (adapter) and put in empty PCI slot a new drive.
What's the best way to clone ZFS drive, use clonezilla (dd) to move everything over to bigger drive?

 

[Kingneutron]

https://github.com/kneutron/ansites...-replace-zfs-mirror-boot-disks-with-bigger.sh

DO NOT just blindly run the script, read through it.

Basically you set autoexpand=on, use ' zpool replace ' and then fix UEFI boot on the new/larger drive with proxmox boot utility.

DO NOT ATTEMPT THIS WITHOUT A FULL BACKUP. You have been warned.

If you want to re-purpose the original/smaller drive afterward you should create a new GPT partition table on it and wipefs -a , otherwise zpool import could get confused.

 

[justinclift]

Oh, maybe check if the USB3 adapter plays around with the logical layout of the drive it's passing through?

Saying that from remembering a USB-drive adapter things from years ago (for SATA drives) that decided it was going to secretly store metadata about the drive in the first few sectors, and pass through everything else.

Naturally, that screwed up any hope of using that drive directly for booting when the adapter was removed from the chain.

 

[listhor]

Oh, maybe check if the USB3 adapter plays around with the logical layout of the drive it's passing through?

I could also temporarily remove other NVME drives (storage) for a period of replication. Anyway, it's a valid point.

https://github.com/kneutron/ansites...-replace-zfs-mirror-boot-disks-with-bigger.sh

DO NOT just blindly run the script, read through it.

Basically you set autoexpand=on, use ' zpool replace ' and then fix UEFI boot on the new/larger drive with proxmox boot utility.

DO NOT ATTEMPT THIS WITHOUT A FULL BACKUP. You have been warned.

If you want to re-purpose the original/smaller drive afterward you should create a new GPT partition table on it and wipefs -a , otherwise zpool import could get confused.

Thanks, later I will go through it more carefully. As for now, it seems that more or less does a job described in: https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_change_failed_dev ?

------
Depending on how Proxmox VE was installed it is either using systemd-boot or GRUB through proxmox-boot-tool [2] or plain GRUB as bootloader (see Host Bootloader). You can check by running:
# proxmox-boot-tool status
The first steps of copying the partition table, reissuing GUIDs and replacing the ZFS partition are the same. To make the system bootable from the new disk, different steps are needed which depend on the bootloader in use.

1. # sgdisk <healthy bootable device> -R <new device>
2. # sgdisk -G <new device>
3. # zpool replace -f <pool> <old zfs partition> <new zfs partition>

Use the zpool status -v command to monitor how far the resilvering process of the new disk has progressed.

With proxmox-boot-tool:
# proxmox-boot-tool format <new disk's ESP>
------

It looks like 2 or maybe 3 options are left:

• manual / scripted copying - these options do require live Linux CD/USB with ZFS support? It can't be done from within booted Proxmox?
• clonezilla https://clonezilla.org/show-live-doc-content.php?topic=clonezilla-live/doc/03_Disk_to_disk_clone
• add the new drive as a mirror to the original drive (resilver it) then offline the old drive - would that work?

 

[justinclift]

Hmmm, if your USB3 adapter just passed through the drive without mucking around with it (as per my post above), you should be fine to do it all from your running Proxmox install.

Actually, you probably should do it from your running Proxmox install, as it'll need to be running the ZFS root pool ("rpool") in order to live add your USB drive as a mirror (which you want).

The steps pretty much boil down to this:

1. Copy the partition structure from your existing, working drive to the new one
   • 1st partition is just a super small thing (1MB) for grub to store data in
   • 2nd partition is a small (1GB?) EFI partition to hold boot data
     • At least, that's what it is on my systems
   • 3rd partition is your root partition (/)
2. Add the 3rd partition of your new drive as a mirror
   • This does the bulk of the data copying as a background task, so be super careful to let this finish copying before rebooting.
   • You can check the copy progress with zpool status
3. Fill in the contents of the first two partitions so all of the bootup files are there and ready to go
4. Shutdown, do the physical drive-swap thing, start up and maybe select the new drive in the BIOS as your boot device

 

[listhor]

After looking closer at all options, I think the most safe is clonezilla. But If I will manage to have more time, I'll clone system drive to network folder and then play with manual data transfer as you guys described. Thanks!

 

[listhor]

So, it's been some time since I decided to change SSD's. And I run into some issues during this process.
First of all, If you use Intel's P3600 SSD and Supermicro mobo you need to change following settings in BIOS:

M.2 with Intel AIC P4800/P4600 or similar NVMe AIC type.
a. Change slot#x, where the card is installed, OPROM to EFI
b. Change M.2 OPROM to EFI
c. Change BIOS setup menu >> Advanced>>PCI/PCIe/PnP Configuration>>NVMe Firmware Source>>AMI Native Support.

I cloned my old system's SSD but due to sector size difference (between old 512 and new 4096) I couldn't use it to restore any partition; neither I could use these commands to copy partition table and uuid:

sgdisk <healthy bootable device> -R <new device>
sgdisk -G <new device>

So I created (using fdisk) in new SSD 1G (EFI) partition and second for remaining capacity. And issued following commands:

proxmox-boot-tool format <new disk's EFI>
proxmox-boot-tool init <new disk's EFI>

I wasn't sure how to set up properly mirror for old zfs partition, so I used replace command to transfer (make sure to enable at first autoexpand, check: zpool get autoexpand <pool>) data:

zpool replace -f <pool> <old zfs partition> <new zfs partition>

Once resilvering was finished (monitored by zpool status -v), I rebooted and changed boot options in bios to reflect change of boot drive.
Once it was successfully rebooted, I did clean up of EFIs uuid:

proxmox-boot-tool clean

I hope it will continue working fine

EDIT:
added autoexpand