replace harddrive in zfs raid

R

robotto

Active Member
May 9, 2019
6
1
43
Hello,
yesterday I replace a harddrive in my zfs raid. After this I execute the following command.

zpool replace rpool 18419091268690890705 /dev/disk/by-id/ata-WDC_WD4005FFBX-68CAUN0_WD-AS05X86R

Tonight the process was successfully and the ZFS Raid is online again.

Now I installed the last new kernel and the new kernel was not copy to the new harddrive.

What is wrong?

root@srv4:/home/frank# apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
proxmox-kernel-6.17 proxmox-kernel-6.8
The following packages will be upgraded:
libfreetype6 libgstreamer-plugins-base1.0-0 libpve-cluster-api-perl libpve-cluster-perl libpve-common-perl libpve-notify-perl pve-cluster
7 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 1,773 kB of archives.
After this operation, 7,168 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://security.debian.org trixie-security/main amd64 libfreetype6 amd64 2.13.3+dfsg-1+deb13u1 [452 kB]
Get:2 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 libpve-common-perl all 9.1.8 [125 kB]
Get:3 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 libpve-notify-perl all 9.1.1 [9,076 B]
Get:4 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 libpve-cluster-api-perl all 9.1.1 [45.7 kB]
Get:5 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 libpve-cluster-perl all 9.1.1 [27.0 kB]
Get:6 http://security.debian.org trixie-security/main amd64 libgstreamer-plugins-base1.0-0 amd64 1.26.2-1+deb13u1 [997 kB]
Get:7 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 pve-cluster amd64 9.1.1 [117 kB]
Fetched 1,773 kB in 0s (8,829 kB/s)
Reading changelogs... Done
(Reading database ... 88803 files and directories currently installed.)
Preparing to unpack .../0-libfreetype6_2.13.3+dfsg-1+deb13u1_amd64.deb ...
Unpacking libfreetype6:amd64 (2.13.3+dfsg-1+deb13u1) over (2.13.3+dfsg-1) ...
Preparing to unpack .../1-libgstreamer-plugins-base1.0-0_1.26.2-1+deb13u1_amd64.deb ...
Unpacking libgstreamer-plugins-base1.0-0:amd64 (1.26.2-1+deb13u1) over (1.26.2-1) ...
Preparing to unpack .../2-libpve-common-perl_9.1.8_all.deb ...
Unpacking libpve-common-perl (9.1.8) over (9.1.7) ...
Preparing to unpack .../3-libpve-notify-perl_9.1.1_all.deb ...
Unpacking libpve-notify-perl (9.1.1) over (9.0.7) ...
Preparing to unpack .../4-libpve-cluster-api-perl_9.1.1_all.deb ...
Unpacking libpve-cluster-api-perl (9.1.1) over (9.0.7) ...
Preparing to unpack .../5-libpve-cluster-perl_9.1.1_all.deb ...
Unpacking libpve-cluster-perl (9.1.1) over (9.0.7) ...
Preparing to unpack .../6-pve-cluster_9.1.1_amd64.deb ...
Unpacking pve-cluster (9.1.1) over (9.0.7) ...
Setting up libgstreamer-plugins-base1.0-0:amd64 (1.26.2-1+deb13u1) ...
Setting up libpve-common-perl (9.1.8) ...
Setting up pve-cluster (9.1.1) ...
Setting up libfreetype6:amd64 (2.13.3+dfsg-1+deb13u1) ...
Setting up libpve-cluster-perl (9.1.1) ...
Setting up libpve-cluster-api-perl (9.1.1) ...
Setting up libpve-notify-perl (9.1.1) ...
Processing triggers for pve-manager (9.1.6) ...
Processing triggers for systemd (257.9-1~deb13u1) ...
Processing triggers for man-db (2.13.1-1) ...
Processing triggers for procps (2:4.0.4-9) ...
Processing triggers for pve-ha-manager (5.1.1) ...
Processing triggers for libc-bin (2.41-12+deb13u2) ...
root@srv4:/home/frank# apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
proxmox-kernel-6.17 proxmox-kernel-6.8
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
root@srv4:/home/frank# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
proxmox-kernel-6.17.13-2-pve-signed proxmox-kernel-6.8.12-20-pve-signed
The following packages will be upgraded:
proxmox-kernel-6.17 proxmox-kernel-6.8
2 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 228 MB of archives.
After this operation, 1,569 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 proxmox-kernel-6.17.13-2-pve-signed amd64 6.17.13-2 [125 MB]
Get:2 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 proxmox-kernel-6.17 amd64 6.17.13-2 [11.7 kB]
Get:3 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 proxmox-kernel-6.8.12-20-pve-signed amd64 6.8.12-20 [103 MB]
Get:4 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 proxmox-kernel-6.8 all 6.8.12-20 [11.4 kB]
Fetched 228 MB in 18s (12.6 MB/s)
Reading changelogs... Done
Selecting previously unselected package proxmox-kernel-6.17.13-2-pve-signed.
(Reading database ... 88803 files and directories currently installed.)
Preparing to unpack .../proxmox-kernel-6.17.13-2-pve-signed_6.17.13-2_amd64.deb ...
Unpacking proxmox-kernel-6.17.13-2-pve-signed (6.17.13-2) ...
Preparing to unpack .../proxmox-kernel-6.17_6.17.13-2_amd64.deb ...
Unpacking proxmox-kernel-6.17 (6.17.13-2) over (6.17.9-1) ...
Selecting previously unselected package proxmox-kernel-6.8.12-20-pve-signed.
Preparing to unpack .../proxmox-kernel-6.8.12-20-pve-signed_6.8.12-20_amd64.deb ...
Unpacking proxmox-kernel-6.8.12-20-pve-signed (6.8.12-20) ...
Preparing to unpack .../proxmox-kernel-6.8_6.8.12-20_all.deb ...
Unpacking proxmox-kernel-6.8 (6.8.12-20) over (6.8.12-18) ...
Setting up proxmox-kernel-6.8.12-20-pve-signed (6.8.12-20) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.8.12-20-pve /boot/vmlinuz-6.8.12-20-pve
update-initramfs: Generating /boot/initrd.img-6.8.12-20-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/1E9D-E419
No initrd-image /boot/initrd.img-6.17.13-2-pve found - skipping
Copying kernel 6.17.9-1-pve
Copying kernel 6.8.12-20-pve
Removing old version 6.17.4-2-pve
Removing old version 6.8.12-18-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.17.9-1-pve
Found initrd image: /boot/initrd.img-6.17.9-1-pve
Found linux image: /boot/vmlinuz-6.8.12-20-pve
Found initrd image: /boot/initrd.img-6.8.12-20-pve
Adding boot menu entry for UEFI Firmware Settings ...
done
WARN: /dev/disk/by-uuid/1EA0-B974 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 6.8.12-20-pve /boot/vmlinuz-6.8.12-20-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 6.8.12-20-pve /boot/vmlinuz-6.8.12-20-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/1E9D-E419
No initrd-image /boot/initrd.img-6.17.13-2-pve found - skipping
Copying kernel 6.17.9-1-pve
Copying kernel 6.8.12-20-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.17.9-1-pve
Found initrd image: /boot/initrd.img-6.17.9-1-pve
Found linux image: /boot/vmlinuz-6.8.12-20-pve
Found initrd image: /boot/initrd.img-6.8.12-20-pve
Adding boot menu entry for UEFI Firmware Settings ...
done
WARN: /dev/disk/by-uuid/1EA0-B974 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 6.8.12-20-pve /boot/vmlinuz-6.8.12-20-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.17.13-2-pve
/usr/sbin/grub-probe: error: unknown filesystem.
Found linux image: /boot/vmlinuz-6.17.9-1-pve
Found initrd image: /boot/initrd.img-6.17.9-1-pve
Found linux image: /boot/vmlinuz-6.17.4-2-pve
Found initrd image: /boot/initrd.img-6.17.4-2-pve
Found linux image: /boot/vmlinuz-6.8.12-20-pve
Found initrd image: /boot/initrd.img-6.8.12-20-pve
Found linux image: /boot/vmlinuz-6.8.12-18-pve
Found initrd image: /boot/initrd.img-6.8.12-18-pve
/usr/sbin/grub-probe: error: unknown filesystem.
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Setting up proxmox-kernel-6.17.13-2-pve-signed (6.17.13-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.17.13-2-pve /boot/vmlinuz-6.17.13-2-pve
update-initramfs: Generating /boot/initrd.img-6.17.13-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/1E9D-E419
Copying kernel 6.17.13-2-pve
Copying kernel 6.17.9-1-pve
Copying kernel 6.8.12-20-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.17.13-2-pve
Found initrd image: /boot/initrd.img-6.17.13-2-pve
Found linux image: /boot/vmlinuz-6.17.9-1-pve
Found initrd image: /boot/initrd.img-6.17.9-1-pve
Found linux image: /boot/vmlinuz-6.8.12-20-pve
Found initrd image: /boot/initrd.img-6.8.12-20-pve
Adding boot menu entry for UEFI Firmware Settings ...
done
WARN: /dev/disk/by-uuid/1EA0-B974 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 6.17.13-2-pve /boot/vmlinuz-6.17.13-2-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 6.17.13-2-pve /boot/vmlinuz-6.17.13-2-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/1E9D-E419
Copying kernel 6.17.13-2-pve
Copying kernel 6.17.9-1-pve
Copying kernel 6.8.12-20-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.17.13-2-pve
Found initrd image: /boot/initrd.img-6.17.13-2-pve
Found linux image: /boot/vmlinuz-6.17.9-1-pve
Found initrd image: /boot/initrd.img-6.17.9-1-pve
Found linux image: /boot/vmlinuz-6.8.12-20-pve
Found initrd image: /boot/initrd.img-6.8.12-20-pve
Adding boot menu entry for UEFI Firmware Settings ...
done
WARN: /dev/disk/by-uuid/1EA0-B974 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 6.17.13-2-pve /boot/vmlinuz-6.17.13-2-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.17.13-2-pve
Found initrd image: /boot/initrd.img-6.17.13-2-pve
/usr/sbin/grub-probe: error: unknown filesystem.
Found linux image: /boot/vmlinuz-6.17.9-1-pve
Found initrd image: /boot/initrd.img-6.17.9-1-pve
Found linux image: /boot/vmlinuz-6.17.4-2-pve
Found initrd image: /boot/initrd.img-6.17.4-2-pve
Found linux image: /boot/vmlinuz-6.8.12-20-pve
Found initrd image: /boot/initrd.img-6.8.12-20-pve
Found linux image: /boot/vmlinuz-6.8.12-18-pve
Found initrd image: /boot/initrd.img-6.8.12-18-pve
/usr/sbin/grub-probe: error: unknown filesystem.
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Setting up proxmox-kernel-6.8 (6.8.12-20) ...
Setting up proxmox-kernel-6.17 (6.17.13-2) ...
Click to expand...

There is also a partition missing I guess.

sda is the existing harddrive
sdb is the new harddrive

lrwxrwxrwx 1 root root 9 Mar 20 13:07 ata-WDC_WD4005FFBX-68CAUN0_WD-AS05X86R -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar 20 13:18 ata-WDC_WD4005FFBX-68CAUN0_WD-AS05X86R-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Mar 20 13:18 ata-WDC_WD4005FFBX-68CAUN0_WD-AS05X86R-part9 -> ../../sdb9
lrwxrwxrwx 1 root root 9 Mar 20 13:07 ata-WDC_WD40EFAX-68JH4N1_WD-WXD2D53JA7E1 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 20 13:07 ata-WDC_WD40EFAX-68JH4N1_WD-WXD2D53JA7E1-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 20 13:07 ata-WDC_WD40EFAX-68JH4N1_WD-WXD2D53JA7E1-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Mar 20 13:07 ata-WDC_WD40EFAX-68JH4N1_WD-WXD2D53JA7E1-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Mar 20 13:07 usb-WD_Elements_323331333139383032303433-0:0 -> ../../sdc
lrwxrwxrwx 1 root root 9 Mar 20 13:07 wwn-0x50014ee215d7d91c -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 20 13:07 wwn-0x50014ee215d7d91c-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 20 13:07 wwn-0x50014ee215d7d91c-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Mar 20 13:07 wwn-0x50014ee215d7d91c-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Mar 20 13:07 wwn-0x50014ee2171933c6 -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar 20 13:18 wwn-0x50014ee2171933c6-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Mar 20 13:18 wwn-0x50014ee2171933c6-part9 -> ../../sdb9
Click to expand...
 
  • Like
Reactions: news and leesteken
Btw: "WD40EFAX" is a SHINGLED disk. It may be problematic to use it with ZFS...
 
  • Like
Reactions: news and leesteken
Hello,
with proxmox-boot-tool I got the following output.
root@srv4:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
1E9D-E419 is configured with: grub (versions: 6.17.13-2-pve, 6.17.9-1-pve, 6.8.12-20-pve)
WARN: /dev/disk/by-uuid/1EA0-B974 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
Click to expand...

Can I now execute?

sgdisk <healthy bootable device> -R <new device>
sgdisk -G <new device>

What happend than to the zfs pool?
 
robotto said:
What happend than to the zfs pool?
Click to expand...
The specific pool member (sdb9?) gets (logically) destroyed because the partition table on the "<new device>" get overwritten. From that moment on there is no redundancy. Of course you should update your backups first.

You need to follow the guide to get back to a "normal"/documented setup. If you do not establish redundant boot capabilities you'll get into trouble when (not: "if"!) the only bootable device dies.

We are talking about a mirror-vdev, right? You did not tell us the pool topology, so I'm just guessing! You may post the output of zpool status; zpool list -v to show us more details.
 
Hello,
here is the output.
root@srv4:/home/frank# zpool status; zpool list -v
pool: rpool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: resilvered 508G in 12:19:03 with 0 errors on Sat Mar 21 01:37:18 2026
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD40EFAX-68JH4N1_WD-WXD2D53JA7E1-part3 ONLINE 0 0 0
ata-WDC_WD4005FFBX-68CAUN0_WD-AS05X86R ONLINE 0 0 0

errors: No known data errors
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 3.62T 508G 3.13T - - 45% 13% 1.00x ONLINE -
mirror-0 3.62T 508G 3.13T - - 45% 13.7% - ONLINE
ata-WDC_WD40EFAX-68JH4N1_WD-WXD2D53JA7E1-part3 3.64T - - - - - - - ONLINE
ata-WDC_WD4005FFBX-68CAUN0_WD-AS05X86R 3.64T - - - - - - - ONLINE
Click to expand...
 
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom
Back