Add disk and create ZFS Mirror?

[aconnor]

Hi,
I am new to Proxmox and very impressed so far.

I want to upgrade my machine- it currently has a 2TB Intel NVMe disk with everything on it.

Goal- add a second 2TB Intel NVMe disk as a mirror without having to wipe and re-install

I don't really care that much if I have to wipe and rebuild, but I'm trying to learn more about disk management and I'm finding the docs a bit difficult to parse into usable info

Happy to read links, docs, whatever you've got

 

[jsterr]

I guess a reinstall would be the easiest way for you if yo dont care about wipe and rebuild. You can select ZFS Mirror on Install under Advanced Options, make sure you put in the right size for your disks.



I guess your having an ext4 install not a zfs single disk setup right?

A untestet other way might be to create a zfs single disk (via GUI). move content to that disk and then add the old disk via cli to make it a mirror: https://coderwall.com/p/zpb89a/creating-a-mirrored-zfs-pool-out-of-a-single-disk-rpool

 

[Stoiko Ivanov]

In addition to what @jsterr said - should you have installed PVE on ZFS (single disk/RAID0) you can follow the reference documentation for creating the correct partitions on the new NVMe :
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_zfs_administration (Changing a failed bootable device)

and then attach the correct partition to the pool (as linked by @jsterr)

In any case - make sure you have a working backup, if you care about the data!

 

[Dunuin]

Proxmox also only needs around 16GB of storage. If you got some old 120/240 GB SSD laying around and free slots in your case you could install PVE to these and use the NVMes as a storage for VMs only. That way you can easily wipe the VM pool, restore VMs/LXCs from backups without needing to install and setup your PVE again.

 

[aconnor]

this is awesome, thanks for your help!
I will do as per the first couple of replies- I have set up a second unit with mirrored ZFS volume, but the first unit only started with one.
I was pretty sure I made first unit a ZFS single drive because this was always in the plans, but if it blows up there's nothing major lost, and yes I could backup first!

 

[Dunuin]

Also keep in mind that features like bit rot protection require some kind of parity. With just a single ZFS disk it can tell you that you lost data to bit rot but it can't repair it. So it is always a good idea to use atleast a simple mirror. Not only for the case that a drive dies.

 

[batrovich]

Hi, i'm in the same situation here, have PBS with just one ext4 disk, and want to add a second one in mirror. I think the best thing it's reinstall everything from scratch. But i would like to "backup my bakcups" into and external USB and then, after de reinstallation restore them.
It's that possible? Thanks a lot!

 

[Dunuin]

Yes, but not using the webUI.
You can...
- attach and mount USB Disk
- backup your config files and keys (especially "/etc/proxmox-backup/") to USB disk
- verify all backup snapshots
- stop the PBS services (so your datastore won't be accessed while backing up)
- copy your entire PBS datastore folder to your USB disk
- wipe your PBS disks
- install a fresh PBS ontop of a HW raid1 or using a ZFS mirror
- copy the datastore folder from your USB disk to your local disks
- edit your "/etc/proxmox-backup/datastore.cfg" so it points to your copied datastore folder
- unplug USB disk
- reboot your PBS
- run a verify job again but force it to re-verif, to check backup integrity is still fine

 

[batrovich]

Thanks a lot for your replay!

Just a question, is there any other config files and keys beside proxmox-backup?

 

[batrovich]

Me again, i already make de backups, but i'm having some doubts about de reinstallation process of PBS. I have one ssd disk where i plan to install proxmox, and then the two 4 TB disks that i want to put in mirror.

How should i proceed with the installation?

thanks a lot for any help!

 

[Dunuin]

Then you don't need to reinstall PBS when PBS is installed on its own dedicated disk.
What is your disk storing the datastore formated with? If it is already ZFS you might even create it into a mirror without deleting anything.

 

[batrovich]

Ahhh too late. I realize that didn't reinstalll everything, but i already wipe both 4TB disks and create a zfs mirror pool with them. Now i'm copying back de backups. How could i've done that without wiping the disk?

 

[Dunuin]

If you didn't previously used ZFS you would have to wipe it anyway.
Have a look at the zpool command: https://openzfs.github.io/openzfs-docs/man/8/zpool-attach.8.html
With that you can attach another disk to an existing vdev creating a mirror of an existing single disk vdev.
Something like: zpool attach YourPool /dev/disk/by-id/YourSingleDisk /dev/disk/by-id/NewDiskToAdd

 

[batrovich]

Yeah didn't previously used zfs. I'll check that out.
Thank you for your help!

 

[eminemfan]

Writing the full process for the sake of ease as already mentioned by @Stoiko Ivanov and @jsterr :
1. Using the instructions for Changing a failed bootable device we run sgdisk <old device> -R <new device>
2. This copies the partition table from the old drive to the new one, when you go to into <your node> -> Disks you should see both drives having the same partitions
3. Again from the instructions for Changing a failed bootable device we runsgdisk -G <new device>. This randomizes the GUIDs of the partitions and we are going to use them when attaching new drive to our pool.
4. Run ls /dev/disk/by-id/ to find the id for the ZFS partition on the new drive. It also lists the ZFS partition on the old drive and we can make sure not to mix them by using running zpool status.
5. Finally run zpool attach -f <name of the pool> /dev/disk/by-id/<old device> /dev/disk/by-id/<new device>
and wait for the resilver to finish.

Whole thing should look like

root@pve:~# sgdisk /dev/nvme0n1 -R /dev/nvme1n1
root@pve:~# sgdisk -G /dev/nvme1n1
root@pve:~# ls /dev/disk/by-id/
nvme-eui.00000000000000008ce38e03006425f4     
nvme-eui.00000000000000008ce38e03006425f4-part1
nvme-eui.00000000000000008ce38e03006425f4-part2
nvme-eui.00000000000000008ce38e03006425f4-part3
nvme-eui.00000000000000008ce38e03006425fa     
nvme-eui.00000000000000008ce38e03006425fa-part1
nvme-eui.00000000000000008ce38e03006425fa-part2
nvme-eui.00000000000000008ce38e03006425fa-part3
root@pve:~# zpool attach -f rpool nvme-eui.00000000000000008ce38e03006425f4-part3 nvme-eui.00000000000000008ce38e03006425fa-part3

EDIT: To be able to boot from the new drive you also need to run proxmox-boot-tool format <EFI partition on the new drive> and proxmox-boot-tool init <EFI partition on the new drive> . You can get the names of the partition from <your node> -> Disks