ZFS mirror replacement

[ZooKeeper]

So, I read where I can replace failed ZFS drive from mirror, but I would like to do something different.

I have two 250gb SSD VM drive in ZFS mirror set up. I want to replace each with 1TB one by one. How do I remove one drive 250gb first and then add first new one and then remove other 250gb and repeat same process. It isn't production unit but my pfsense is on it so looking for a way to make sure I don't screw up.

I have to remove 1 of 250gb first as I am trading with my brother for 1TB. so I may need to restart in between.

This is how zpool status as of now.



Thanks in advance.

 

[aaron]

Which pool? The root pool or the VMs one?

In general the idea for this is to remove one of the disks in the pool and replace it with the larger one. Replace the missing disk with the new one. If you have partitions on the disks you will have to recreate or clone them. When you clone them make sure to create new GUIDs for them.
Then wait for the resilvering to be done before you change the second disk. Keep the disks you take out on the side until everything is done just to have another copy should something go south.

Commands on how to clone partitions and create new GUIDs are described in the admin manual: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_zfs_administration

 

[ZooKeeper]

It is VM disk I want to change. But what would be commands to remove one of the disk. Shutdown computer and then plug in 1Tb and then add.

 

[jim.bond.9862]

First things first.
Before you start replacing the drives , turn the zfs auto expand ON on the pool.
This way once you replace all the disk the pool will expand to fill the whole new space.
Second if you have the ports and space mount the new disks in the box. Get the disks uuid and run zfs replace command.
Google it for correct syntax.
This will disconnect the drive you replacing and add the new one.
Repeat after resilvering is done.

By the look of it, your VM pool is non bootable so no partitioning or other prep is needed.

If you can not add new disks before removing the old one , just offline it, remove the disk, add new one and run replace command. Zfs support hot replace. That is if your hardware support hot plug unplug of disks you can keep the server running and do all on working server. No downtime at all. I had replaced my system boot disks with larger once without shutting down the machine at all.

 

[ZooKeeper]

Yes it is non bootable device. I just did was
Zpool detach VMs sdg

It isn't something i can't live so if it breaks, will learn.

But I forgot to expand on though. Is it going to matter when I add my 1Tb now or can I run command after?

 

[jim.bond.9862]

Doesnt matter , but untill you add new device you are running unprotected. There is no mirror.
Also detach is ok but again it adds time as it has to redo the meta data and such to dissolve the mirror and later it has to recreate the mirror , where actuall replace just updates the meta data to new device and resilvering / balance the mirror with protection. Can save up to half an hour on small disks and more on large one.

 

[aaron]

Don't forget that now you dissolved the mirror and you will have to run the `attach` command with first the current disk and then the new one so ZFS know which disk you want to mirror. zpool attach VMs <current_disk > <new disk>

If you only run zpool attach VMs <new disk> you will have made a RAID 0 like pool and there is no easy way to repair that except recreating the pool.

 

[ZooKeeper]

Thanks @aaron guess what, i had to start over all because i believe i put new disk name before old disk..Better luck next time.