Advice Needed for Replacing HDDs in Mirrored ZFS Pool on Proxmox

[ana_mera]

Hello Proxmox Community,

I’m currently running a Proxmox VE setup on a Hetzner AX161 server and am planning to upgrade my storage. Here’s an overview of my current configuration and what I intend to do:

Current Setup:​

• Boot Disk: Mirrored SSDs (rpool):
  • Purpose: Hosting the Proxmox system, root filesystem, and subvolumes for containers (CTs).
  • Configuration: Mirrored ZFS pool (rpool), identified as SSDs.
  • Datasets: rpool/ROOT/pve-1 is mounted as /, and several container subvolumes are under rpool/data.
• Data Disk: Mirrored HDDs (backup):
  • Purpose: Used primarily for storing VZDump backups.
  • Configuration: Mirrored ZFS pool (backup) with two 2 TB HDDs.

Planned Upgrade:​

I want to replace the existing 2 TB HDDs in the backup pool with new 8 TB HDDs. I’m considering the following approach:

1. Add the First 8 TB HDD:
   • Confirm there is available space or temporarily remove one 2 TB HDD to add the new 8 TB HDD.
   • Add the new 8 TB HDD to the server.
2. Replace One of the 2 TB HDDs:
   • Use the ZFS replace command to replace the first 2 TB HDD with the new 8 TB HDD. (example: zpool replace backup /dev/sdd /dev/sdc)
   • Monitor and complete the resilvering process. (via pool status backup)
3. Remove the Replaced 2 TB HDD:
   • Once resilvering is done, remove the old 2 TB HDD.
4. Add the Second 8 TB HDD:
   • Add the second 8 TB HDD to the server.
5. Replace the Remaining 2 TB HDD:
   • Use the ZFS replace command to replace the remaining 2 TB HDD with the second 8 TB HDD.
   • Monitor and complete the resilvering process.
6. Expand the ZFS Pool:
   • After both drives are replaced and resilvering is complete, expand the backup pool to utilize the full 8 TB capacity using the ZFS autoexpand feature (example: zpool set autoexpand=on backup)

Questions:​

1. Is this step-by-step approach the best way to replace my existing 2 TB HDDs with 8 TB HDDs while ensuring data integrity and minimal downtime?
2. Are there any risks or potential issues I should be aware of during this process?
3. Do I need to manually set autoexpand=on before or after the drive replacement, or is it automatic?
4. Given my current setup, is there anything else I should consider or prepare for before starting the upgrade?

Thank you for your advice and support!

Ana

 

[ggoller]

Use the ZFS replace command to replace the first 2 TB HDD with the new 8 TB HDD. (example: zpool replace backup /dev/sdd /dev/sdc)

I recommend using the disk-id (/dev/disk/by-id/*) instead of the label. (The label could change, the disk-id doesn't.)

Is this step-by-step approach the best way to replace my existing 2 TB HDDs with 8 TB HDDs while ensuring data integrity and minimal downtime?

AFAIK yes.

Are there any risks or potential issues I should be aware of during this process?

There shouldn't be.

Do I need to manually set autoexpand=on before or after the drive replacement, or is it automatic?

You can set autoexpand=on before replacing the devices or you could execute zpool online -e pool device to manually expand the zpool (after replacing the devices obviously).

Otherwise this approach looks good!

 

[ubu]

My recipe:



Zpool replace disk
==================

Get disk IDs
ls -l /dev/disk/by-id/*


Get zpool status:

zpool status

this assumes the following disk layout:

Part 1: BIOS Boot
Part 2: EFI
Part 3: ZFS

Copy Partitions from working to new disk, without copying label and UUIDs:

sfdisk -d /dev/WORKING | sed 's/, uuid.*//; /label-id/d;' |sfdisk /dev/REPLACEMENT

Replace Disk, give ZFS Partition
zpool replace zp_pve /dev/disk/by-id/nvme-OLD-part3 /dev/disk/by-id/nvme-iREPLACEMENT-part3


Check status, should resilver:
zpool status


Rewrite Bootloader:
proxmox-boot-tool format /dev/disk/by-id/nvme-REPLACEMENT-part2
proxmox-boot-tool init /dev/disk/by-id/nvme-REPLACEMENT-part2
proxmox-boot-tool status

Clean /etc/kernel/proxmox-boot-uuids of old entries

proxmox-boot-tool status
proxmox-boot-tool refresh
proxmox-boot-tool clean

 

[ana_mera]

Thank you for quick responce!

I have acctually two more questions:

1. Should I detach the disk from the pool before i remove it? (via zpool detach command)

2. Which approach is recommended:

-Remove one of the 2 TB HDDs first, add the new 8 TB HDD, and then mirror the data from the remaining old 2 TB HDD?
-Add the 8 TB HDD first, mirror it with one of the old 2 TB HDDs, and then remove the old 2 TB HDD?


Thank you in advance!

Ana

 

[ana_mera]

My recipe:



Zpool replace disk
==================

Get disk IDs
ls -l /dev/disk/by-id/*


Get zpool status:

zpool status

this assumes the following disk layout:

Part 1: BIOS Boot
Part 2: EFI
Part 3: ZFS

Copy Partitions from working to new disk, without copying label and UUIDs:

sfdisk -d /dev/WORKING | sed 's/, uuid.*//; /label-id/d;' |sfdisk /dev/REPLACEMENT

Replace Disk, give ZFS Partition
zpool replace zp_pve /dev/disk/by-id/nvme-OLD-part3 /dev/disk/by-id/nvme-iREPLACEMENT-part3


Check status, should resilver:
zpool status


Rewrite Bootloader:
proxmox-boot-tool format /dev/disk/by-id/nvme-REPLACEMENT-part2
proxmox-boot-tool init /dev/disk/by-id/nvme-REPLACEMENT-part2
proxmox-boot-tool status

Clean /etc/kernel/proxmox-boot-uuids of old entries

proxmox-boot-tool status
proxmox-boot-tool refresh
proxmox-boot-tool clean

Thank you for your recipe

I think i do not need all of that, since my current HDDs are not bootable at all, but my SSDs are.

So the only command that i need is acctually zpool replace

 

[ubu]

I would:

Remove one of the 2 TB HDDs first, add the new 8 TB HDD, and then mirror the data from the remaining old 2 TB HDD

but both should work, if you have the free slot

 

[ana_mera]

I recommend using the disk-id (/dev/disk/by-id/*) instead of the label. (The label could change, the disk-id doesn't.)


AFAIK yes.


There shouldn't be.


You can set autoexpand=on before replacing the devices or you could execute zpool online -e pool device to manually expand the zpool (after replacing the devices obviously).

Otherwise this approach looks good!

Hi Gabriel,

Thank you for answering

I’ve just received confirmation from Hetzner that there are no additional slots available for new drives. This means I need to remove one of the existing 2 TB HDDs before I can add the new 8 TB HDD.

My Questions:​

1. Is it safe for Hetzner to simply remove the old 2 TB HDD directly?
   • Should I perform any commands on Proxmox or ZFS before they physically remove the drive to ensure the system handles the change correctly?
2. Hot Swapping Concerns:
   • Hetzner has indicated that hot swapping is supported on their end. However, I’m unsure if Proxmox fully supports hot swapping for ZFS pools and if it is advisable in my scenario.
   • Do you recommend using hot swapping for this replacement, or should I consider scheduling a downtime for the replacement to minimize risks?

Thank you for your advice and support!

Ana

 

[ggoller]

HI!
in this case I'd use zpool detach and zpool attach. Before removing the old drive, detach it from the pool like this: zpool detach <pool> <device>, then you can physically remove it. After that insert the new one and execute zpool attach <pool> <other-device-in-mirror> <new-device>. The pool will automatically resilver and should work again.

Hetzner has indicated that hot swapping is supported on their end. However, I’m unsure if Proxmox fully supports hot swapping for ZFS pools and if it is advisable in my scenario.

You are not really hotswapping though as (if you go with the zpool attach/detach route) you are manually removing the disk from the zpool and adding another one.

Do you recommend using hot swapping for this replacement, or should I consider scheduling a downtime for the replacement to minimize risks?

Technically this should work without any downtime, but it is up to you if you feel comfortable doing it or if you need to schedule a downtime.

 

[ana_mera]

HI!
in this case I'd use zpool detach and zpool attach. Before removing the old drive, detach it from the pool like this: zpool detach <pool> <device>, then you can physically remove it. After that insert the new one and execute zpool attach <pool> <other-device-in-mirror> <new-device>. The pool will automatically resilver and should work again.


You are not really hotswapping though as (if you go with the zpool attach/detach route) you are manually removing the disk from the zpool and adding another one.

Technically this should work without any downtime, but it is up to you if you feel comfortable doing it or if you need to schedule a downtime.

Hi Gabriel!

Thanks for your answer!

Ok I understand, then I will use zpool detach and zpool attach. The swap will be done offline to ensure safety.

My Questions:​

1. How long does the zpool detach command typically take to complete?
   • It’s crucial for me to know the duration because the physical removal of the HDD by Hetzner is scheduled shortly after I perform the detach operation.
2. Here’s the plan I’ve prepared for tonight’s operation. Can you please review and confirm if it looks correct?

# Enable autoexpand for the pool
zpool set autoexpand=on <zpool>

# Detach the old disk from the pool
zpool detach <zpool> /dev/disk/by-id/<old-device-id>

# Shutdown the server safely
shutdown -h now

# (After swapping the disks and booting up the server)
# Attach the new disk to the pool
zpool attach <zpool> /dev/disk/by-id/<existing-device-id> /dev/disk/by-id/<new-device-id>

Thank you for your assistance!
Ana

 

[ggoller]

Hi!

How long does the zpool detach command typically take to complete?

I can't tell you that exactly, but the detach/attach should be quite fast. The resilvering after the attach will take a while though.

The plan looks fine to me!