ZFS reserved and boot partition

pixelpoint · 2024-10-10T09:03:53+0200

Hello dear Proxmox Backup users!

One of our 4 HDDs failed in our Proxmox Backup Server.
We replaced this with a new HDD and then re-silvered the whole pool.

Now, after resilvering, I realised I may have made a mistake.

See this screenshot:

You can see here that every HDD has the following partitions:

ZFS
EFI
BIOS boot

All, except for one. The one we replaced.
The replacement disk shows only 2 partitions: ZFS and ZFS reserved

I am not sure if this is a problem or not.
Does ZFS reserved mean that the EFI and BIOS boot partitions are inside the ZFS reserved partition or are they missing?
If this is inside the ZFS reserved partition, will they work when rebooting and re-silvering?

I guess ZFS, as a RAID manager, volume manager and filesystem knows what it does when re-silvering and therefore everything should be fine, right?

Best regards
pixelpoint

RichardH · 2024-10-10T09:26:56+0200

Hi,

you replaced a boot disk. That is not as straight forward as replacing a failed disk in a regular data-pool.

See: https://pve.proxmox.com/wiki/ZFS_on_Linux
Section: Changing a failed bootable device

Br,
Richard

pixelpoint · 2024-10-10T11:19:49+0200

Thank you for your answer.

Seeing as I am replacing a disk with itself, I did the following:

Code:

# the wrongly partitioned disk: /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT-part1

# removing the disk
zpool offline rpool /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT-part1
zpool labelclear -f /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT-part1

# from the wiki
sgdisk /dev/disk/by-id/ata-ST16000NM003G-2KH113_ZL2CANF3 -R /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT
sgdisk -G /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT
zpool replace -f rpool /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT-part1 /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT

I am now waiting for the resilver to finish (probably going to take another 7 - 14 days), so I can execute the following command:

Bash:

# from the wiki
proxmox-boot-tool format <new disk's ESP>

The ESP in question should then be partition 2 as far as I understand, correct?

Right now zpool status -v shows:

Bash:

Every 1.0s: zpool status -v                                                                                                                                                                                                                               backup: Thu Oct 10 11:16:05 2024

  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Oct 10 10:51:20 2024
        5.04G / 48.1T scanned at 3.48M/s, 0B / 48.1T issued
        0B resilvered, 0.00% done, no estimated completion time
config:

        NAME                                               STATE     READ WRITE CKSUM
        rpool                                              DEGRADED     0     0     0
          raidz1-0                                         DEGRADED     0     0     0
            ata-ST16000NM003G-2KH113_ZL2C9L19-part3        ONLINE       0     0     0
            replacing-1                                    DEGRADED     0     0     0
              ata-ST16000NM001J-2TW113_ZRS1QMPT-part1/old  OFFLINE      0     0     0
              ata-ST16000NM001J-2TW113_ZRS1QMPT            ONLINE       0     0     0
            ata-ST16000NM003G-2KH113_ZL2CABRZ-part3        ONLINE       0     0     0
            ata-ST16000NM003G-2KH113_ZL2CANF3-part3        ONLINE       0     0     0

errors: No known data errors

Thank you for your help, it's very appreciated.
Never used ZFS before PVE / PBS.

Best regards,
pixelpoint

RichardH · 2024-10-10T11:35:22+0200

pixelpoint said:
The ESP in question should then be partition 2 as far as I understand, correct?

According to the documentation it should be. Take your time and double check and it should all go fine

.

pixelpoint said:
Thank you for your help, it's very appreciated.
Never used ZFS before PVE / PBS.

No prob, it was new for me also at one point. We all started somewhere.

BR,
Richard

news · 2024-10-10T11:37:40+0200

No this is wrong:

zpool replace -f rpool /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT-part1 /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT

look on all partition table of the disks, one is different!

check cfdisk <device-name>

ubu · 2024-10-10T11:43:58+0200

pixelpoint said:

Thank you for your answer.

Seeing as I am replacing a disk with itself, I did the following:

Code:

# the wrongly partitioned disk: /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT-part1

# removing the disk
zpool offline rpool /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT-part1
zpool labelclear -f /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT-part1

# from the wiki
sgdisk /dev/disk/by-id/ata-ST16000NM003G-2KH113_ZL2CANF3 -R /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT
sgdisk -G /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT
zpool replace -f rpool /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT-part1 /dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS1QMPT

I am now waiting for the resilver to finish (probably going to take another 7 - 14 days), so I can execute the following command:

Bash:

# from the wiki
proxmox-boot-tool format <new disk's ESP>

The ESP in question should then be partition 2 as far as I understand, correct?

Right now zpool status -v shows:

Bash:

Every 1.0s: zpool status -v                                                                                                                                                                                                                               backup: Thu Oct 10 11:16:05 2024

  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Oct 10 10:51:20 2024
        5.04G / 48.1T scanned at 3.48M/s, 0B / 48.1T issued
        0B resilvered, 0.00% done, no estimated completion time
config:

        NAME                                               STATE     READ WRITE CKSUM
        rpool                                              DEGRADED     0     0     0
          raidz1-0                                         DEGRADED     0     0     0
            ata-ST16000NM003G-2KH113_ZL2C9L19-part3        ONLINE       0     0     0
            replacing-1                                    DEGRADED     0     0     0
              ata-ST16000NM001J-2TW113_ZRS1QMPT-part1/old  OFFLINE      0     0     0
              ata-ST16000NM001J-2TW113_ZRS1QMPT            ONLINE       0     0     0
            ata-ST16000NM003G-2KH113_ZL2CABRZ-part3        ONLINE       0     0     0
            ata-ST16000NM003G-2KH113_ZL2CANF3-part3        ONLINE       0     0     0

errors: No known data errors

Thank you for your help, it's very appreciated.
Never used ZFS before PVE / PBS.

Best regards,
pixelpoint

Small improvement:

Delete the uuid and label-id,while copying, so new ones get created, so no doubles:

sfdisk -d /dev/WORKING | sed 's/, uuid.*//; /label-id/d;' |sfdisk /dev/REPLACEMENT

---

My whole recipe:

Zpool replace disk
==================

Get disk IDs
ls -l /dev/disk/by-id/*

If you replace sdb in a sda/ sdb mirror

ls -l /dev/disk/by-id/* |grep sdb

I use the /dev/disk/by-id names eg:

ls -l /dev/disk/by-id/* |grep sda
lrwxrwxrwx 1 root root 9 Okt 10 09:23 /dev/disk/by-id/ata-SAMSUNG_MZ7KM1T9HAJM-00005_S2HNNX0H638608 -> ../../sda

Get Zpool Status:

zpool status

this assumes the following disk layout:

Part 1: BIOS Boot
Part 2: EFI
Part 3: ZFS

Copy Partitions from working to new disk, without copying label and UUIDs:

sfdisk -d /dev/WORKING | sed 's/, uuid.*//; /label-id/d;' |sfdisk /dev/REPLACEMENT

Replace Disk, give ZFS Partition
zpool replace zp_pve /dev/disk/by-id/nvme-OLD-part3 /dev/disk/by-id/nvme-REPLACEMENT-part3

Check status, should resilver:
zpool status

Rewite Bootloader:
proxmox-boot-tool format /dev/disk/by-id/nvme-REPLACEMENT-part2
proxmox-boot-tool init /dev/disk/by-id/nvme-REPLACEMENT-part2
proxmox-boot-tool status

Clean /etc/kernel/proxmox-boot-uuids of old entries

proxmox-boot-tool status
proxmox-boot-tool refresh
proxmox-boot-tool clean

Search

Search

ZFS reserved and boot partition

pixelpoint

Member

RichardH

Member

pixelpoint

Member

RichardH

Member

news

Active Member

ubu

Renowned Member