New system disks on host instead of old ones

A

alexc

Renowned Member
Apr 13, 2015
138
4
83
I have PVE host running for 5+ years. It was set up on couple of 2Tb HDDs in zfs mirror, and now it's time to change these disks with newer ones. I brought 10Tb HDDs, and plan the following:
- partition new disks so it has layout of old 2Tb disks, but the last partition will accomodate all 10Tb of space.
- remove one "old" disk from rpool pool
- add "new" disk to rpool
- wait until resilver is done
- repeat the same for second pait of "old" and "new" disk.

Seems this is not completely safe since there will be time when I have only 1 disk in pool, but this seems to be easy to understand step-by-step plan, and it also can be done online, without interrupting PVE work for long.

The only alternative I can see is booting from LiveCD, then install PVE on new disks (but I need to rename rpool before that, right?) and copy data to new pool from old one, isn't it?
 
alexc said:
Seems this is not completely safe since there will be time when I have only 1 disk in pool, but this seems to be easy to understand step-by-step plan, and it also can be done online, without interrupting PVE work for long.
Click to expand...
You can make mirrors with more than two drives. You can attach both new drives (make sure use the right command to mirror instead of stripe), wait for them to resilver and then remove both old drives.
If the pool is the rpool and needs to be bootable, you need additional steps as described in the Proxmox manual.
 
First turn autoexpand on with zpool set autoexpand=on <yourpool> and then go on like leesteken described.
 
  • Like
Reactions: leesteken
leesteken said:
You can make mirrors with more than two drives. You can attach both new drives (make sure use the right command to mirror instead of stripe), wait for them to resilver and then remove both old drives.
If the pool is the rpool and needs to be bootable, you need additional steps as described in the Proxmox manual.
Click to expand...
yes, making it bootable is the question. Acually, now I have 3 partitons on each disk used by ZFS (3rd partiton is in pool, while 1 and 2 is not), so I can copy disk partition scheme for 1 and 2 to new disks, and then - use https://pve.proxmox.com/wiki/Host_Bootloader , right?
 
alexc said:
yes, making it bootable is the question. Acually, now I have 3 partitons on each disk used by ZFS (3rd partiton is in pool, while 1 and 2 is not), so I can copy disk partition scheme for 1 and 2 to new disks, and then - use https://pve.proxmox.com/wiki/Host_Bootloader , right?
Click to expand...
Why not follow the section "Changing a failed bootable device" in the manual: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_change_failed_dev
Maybe resize the third partition (with gdisk) after the sgdisk step but you can also do it later. Or create the partitions yourself with gdisk instead of using sgdisk.
Also remove the old ESPs from the proxmox-boot-tool once you are sure you can boot from the new drives.
 
leesteken said:
Why not follow the section "Changing a failed bootable device" in the manual: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_change_failed_dev
Maybe resize the third partition (with gdisk) after the sgdisk step but you can also do it later. Or create the partitions yourself with gdisk instead of using sgdisk.
Also remove the old ESPs from the proxmox-boot-tool once you are sure you can boot from the new drives.
Click to expand...
Sorry for my dumb questioning, I copies partitions boundaries from old disks to new one, so sda1 and sda2 9sda is one of new disks) are same-sized as old disk sdc1 and sdc2 (sdc is one of old disks). Now I need to add sd3 instead of sdc3, I can do zfs replace rpool ... ... or zpool attach rpool ... ... and then zpool remove rpool ... ... .
Now I read that zfs won't be happy with this attach, since it'll consider sda3 as whold vdev, not as partition, and try to use it accordingly. If this is the case, or attach/resilver/remove is working way?

And, which is more, I did dd /dev/sdc[1,2] -> /dev/sda[1,2], and the sabe for sdd[1,2] -> sdb[1,2]. Then I run
Code: 
proxmox-boot-tool status

and got

Code: 
System currently booted with legacy bios
7ACC-88E0 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)
7AD0-CC70 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)

if I read docs right, now I should do

Code: 
proxmox-boot-tool format /dev/sda2
proxmox-boot-tool init /dev/sda2

I really fuzzy here, and very afraid to break the system. Like System currently booted with legacy bios frustrates me very much, not sure if commands above are fit in the case.

Another way I can find to "move" zpool from old disk mirror (sdc+sdd) to new mirror (sda+sdb), it appears I can do

Code: 
zpool add rpool mirror /dev/sda3 /dev/sdd3

wait for resilver, and then do
Code: 
zpool remove rpool /dev/sdc3 /dev/sdd3

But this appears somehow too fantastic, as from point of view of old-raid-cards-world :)

THANK YOU VERY, VERY MUCH FOR YOUR ATTENTION!!!
 
Last edited:
alexc said:
I really fuzzy here, and very afraid to break the system. Like System currently booted with legacy bios frustrates me very much, not sure if commands above are fit in the case.
Click to expand...
This is normal when your system does not boot in UEFI mode.

alexc said:
Another way I can find to "move" zpool from old disk mirror (sdc+sdd) to new mirror (sda+sdb), it appears I can do

Code: 
zpool add rpool mirror /dev/sda3 /dev/sdd3

wait for resilver, and then do
Code: 
zpool remove rpool /dev/sdc3 /dev/sdd3

But this appears somehow too fantastic, as from point of view of old-raid-cards-world :)
Click to expand...
Maybe first add both new partitions before removing both old ones. It's also better to use /dev/disk/by-id/ata-.... instead of /dev/sd.... . And maybe remove the old drives after everything is booting and working fine. You can even remove the old drives to test if everything boots fine and then remove then from the 4-way mirror.

alexc said:
Code: 
proxmox-boot-tool status

and got

Code: 
System currently booted with legacy bios
7ACC-88E0 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)
7AD0-CC70 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)

if I read docs right, now I should do

Code: 
proxmox-boot-tool format /dev/sda2
proxmox-boot-tool init /dev/sda2
Click to expand...
That will initialize one ESP on the new drive for booting Proxmox. You also need to do that for the other new drive (otherwise you cannot boot when sda fails).
Once everything boots from the new drives (maybe you need to change boot order in your motherboard BIOS), old then remove the old ESPs from proxmox-boot-tool.

alexc said:
And, which is more, I did dd /dev/sdc[1,2] -> /dev/sda[1,2], and the sabe for sdd[1,2] -> sdb[1,2].
Click to expand...
There is no point in doing that. And I don't understand why you would do that.

alexc said:
Sorry for my dumb questioning, I copies partitions boundaries from old disks to new one, so sda1 and sda2 9sda is one of new disks) are same-sized as old disk sdc1 and sdc2 (sdc is one of old disks). Now I need to add sd3 instead of sdc3, I can do zfs replace rpool ... ... or zpool attach rpool ... ... and then zpool remove rpool ... ... .
Now I read that zfs won't be happy with this attach, since it'll consider sda3 as whold vdev, not as partition, and try to use it accordingly. If this is the case, or attach/resilver/remove is working way?
Click to expand...
This I don't really understand. Did you change the last sector of the third partition for ZFS to fill the rest of the drive? Assuming that your old installation also had 3 or 4 partitions.
Maybe show the output of lsblk -o+PARTTYPENAME to give people here an overview of your drives and partitions?
 
leesteken said:
This is normal when your system does not boot in UEFI mode.


Maybe first add both new partitions before removing both old ones. It's also better to use /dev/disk/by-id/ata-.... instead of /dev/sd.... . And maybe remove the old drives after everything is booting and working fine. You can even remove the old drives to test if everything boots fine and then remove then from the 4-way mirror.


That will initialize one ESP on the new drive for booting Proxmox. You also need to do that for the other new drive (otherwise you cannot boot when sda fails).
Once everything boots from the new drives (maybe you need to change boot order in your motherboard BIOS), old then remove the old ESPs from proxmox-boot-tool.


There is no point in doing that. And I don't understand why you would do that.


This I don't really understand. Did you change the last sector of the third partition for ZFS to fill the rest of the drive? Assuming that your old installation also had 3 or 4 partitions.
Maybe show the output of lsblk -o+PARTTYPENAME to give people here an overview of your drives and partitions?
Click to expand...
 
Here is the output:
Code: 
root@server:~# lsblk -o+PARTTYPENAME
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT PARTTYPENAME
sda           8:0    0   9.1T  0 disk
├─sda1        8:1    0  1007K  0 part            BIOS boot
├─sda2        8:2    0   512M  0 part            EFI System
└─sda3        8:3    0   9.1T  0 part            Solaris /usr & Apple ZFS
sdb           8:16   0   9.1T  0 disk
├─sdb1        8:17   0  1007K  0 part            BIOS boot
├─sdb2        8:18   0   512M  0 part            EFI System
└─sdb3        8:19   0   9.1T  0 part            Solaris /usr & Apple ZFS
sdc           8:32   0   1.8T  0 disk
├─sdc1        8:33   0  1007K  0 part            BIOS boot
├─sdc2        8:34   0   512M  0 part            EFI System
└─sdc3        8:35   0   1.8T  0 part            Solaris /usr & Apple ZFS
sdd           8:48   0   1.8T  0 disk
├─sdd1        8:49   0  1007K  0 part            BIOS boot
├─sdd2        8:50   0   512M  0 part            EFI System
└─sdd3        8:51   0   1.8T  0 part            Solaris /usr & Apple ZFS
 
alexc said:
Here is the output:
Code: 
root@server:~# lsblk -o+PARTTYPENAME
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT PARTTYPENAME
sda           8:0    0   9.1T  0 disk
├─sda1        8:1    0  1007K  0 part            BIOS boot
├─sda2        8:2    0   512M  0 part            EFI System
└─sda3        8:3    0   9.1T  0 part            Solaris /usr & Apple ZFS
sdb           8:16   0   9.1T  0 disk
├─sdb1        8:17   0  1007K  0 part            BIOS boot
├─sdb2        8:18   0   512M  0 part            EFI System
└─sdb3        8:19   0   9.1T  0 part            Solaris /usr & Apple ZFS
sdc           8:32   0   1.8T  0 disk
├─sdc1        8:33   0  1007K  0 part            BIOS boot
├─sdc2        8:34   0   512M  0 part            EFI System
└─sdc3        8:35   0   1.8T  0 part            Solaris /usr & Apple ZFS
sdd           8:48   0   1.8T  0 disk
├─sdd1        8:49   0  1007K  0 part            BIOS boot
├─sdd2        8:50   0   512M  0 part            EFI System
└─sdd3        8:51   0   1.8T  0 part            Solaris /usr & Apple ZFS
Click to expand...
Initialize the EFI System Partitions (ESP) with proxmox-boot-tool (as you showed before). Attach both third partitions to the rpool (as you showed before) but use /dev/disk/by-id/ata-....-part3 and wait for resilver. Shutdown the system, remove/disconnect both old drives, and check if the system boots properly. Once that works you can remove the both old partitions from the rpool and remove both old ESP from proxmox-boot-tool.
 
leesteken said:
That will initialize one ESP on the new drive for booting Proxmox. You also need to do that for the other new drive (otherwise you cannot boot when sda fails).
Click to expand...
Code: 
root@server:~# proxmox-boot-tool format /dev/sda2
UUID="7ACC-88E0" SIZE="536870912" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="sda" MOUNTPOINT=""
E: '/dev/sda2' contains a filesystem ('vfat') - exiting (use --force to override)
root@server:~# proxmox-boot-tool init /dev/sda2
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
UUID="7ACC-88E0" SIZE="536870912" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="sda" MOUNTPOINT=""
Mounting '/dev/sda2' on '/var/tmp/espmounts/7ACC-88E0'.
Installing grub i386-pc target..
Installing for i386-pc platform.
Installation finished. No error reported.
Unmounting '/dev/sda2'.
Adding '/dev/sda2' to list of synced ESPs..
Refreshing kernels and initrds..
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Copying and configuring kernels on /dev/disk/by-uuid/7ACC-88E0
        Copying kernel 5.13.19-6-pve
        Copying kernel 5.15.39-1-pve
        Copying kernel 5.15.39-3-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.39-3-pve
Found initrd image: /boot/initrd.img-5.15.39-3-pve
Found linux image: /boot/vmlinuz-5.15.39-1-pve
Found initrd image: /boot/initrd.img-5.15.39-1-pve
Found linux image: /boot/vmlinuz-5.13.19-6-pve
Found initrd image: /boot/initrd.img-5.13.19-6-pve
done
Copying and configuring kernels on /dev/disk/by-uuid/7AD0-CC70
        Copying kernel 5.13.19-6-pve
        Copying kernel 5.15.39-1-pve
        Copying kernel 5.15.39-3-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.39-3-pve
Found initrd image: /boot/initrd.img-5.15.39-3-pve
Found linux image: /boot/vmlinuz-5.15.39-1-pve
Found initrd image: /boot/initrd.img-5.15.39-1-pve
Found linux image: /boot/vmlinuz-5.13.19-6-pve
Found initrd image: /boot/initrd.img-5.13.19-6-pve
done
root@server:~#
This is what I got. I very afraid of pointing to /dev/sda2, that is, will /dev/sda be bootable after all? Boot seems to be on sda1, isn't it?

P.S. Actually, I will use by-id name scheme, not sda/sdb, I use sda as short reference to disk.
 
alexc said:
This is what I got. I very afraid of pointing to /dev/sda2, that is, will /dev/sda be bootable after all? Boot seems to be on sda1, isn't it?
Click to expand...
proxmox-boot-tool should take care of everything. Note that your old drives will be unchanged and still be bootable. You can easily overcome your fear by trying to boot from sda. If that does not work then just boot from sdc or sdd again.
 
leesteken said:
Alternatively (instead of this hard-holding on Linux drive management stuff not specific to Proxmox), you can just install a fresh Proxmox on the new drives and restore your VMs from backup. Just make sure to install on the correct drives and not wipe your old ones accidentally.
Click to expand...
This was my first intention to do but I afraid two rpool's in system is not good idea. And yes, I have no place to do backup, this is why I want to do that "in place" (almost).

I just added two new disks to rpool, and resilver started. The server... well, become very slow, so it seems data flows as intended. I run
Code: 
proxmox-boot-tool init /dev/sda2
proxmox-boot-tool init /dev/sdb2
now I see
Code: 
root@server:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
7ACC-88E0 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)
7AD0-CC70 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)
root@server:~#

7ACC-88E0 and 7AD0-CC70 refers to new disks, no old disks listed here (??).
 
alexc said:
Code: 
proxmox-boot-tool init /dev/sda2
proxmox-boot-tool init /dev/sdb2
now I see
Code: 
root@server:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
7ACC-88E0 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)
7AD0-CC70 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)
root@server:~#

7ACC-88E0 and 7AD0-CC70 refers to new disks, no old disks listed here (??).
Click to expand...
You did not do a proxmox-boot-tool format and you did do a dd before. Therefore the new ESPs might have the same IDs as the old ones which is ill advised.
What is the output of lsblk -o+PARTTYPENAME -o+UUID?
 
leesteken said:
You did not do a proxmox-boot-tool format and you did do a dd before
Click to expand...
You're right!

Code: 
root@server:~# lsblk -o+PARTTYPENAME -o+UUID
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT UUID
sda           8:0    0   9.1T  0 disk
├─sda1        8:1    0  1007K  0 part
├─sda2        8:2    0   512M  0 part            7ACC-88E0
└─sda3        8:3    0   9.1T  0 part            10820367462338150560
sdb           8:16   0   9.1T  0 disk
├─sdb1        8:17   0  1007K  0 part
├─sdb2        8:18   0   512M  0 part            7AD0-CC70
└─sdb3        8:19   0   9.1T  0 part            10820367462338150560
sdc           8:32   0   1.8T  0 disk
├─sdc1        8:33   0  1007K  0 part
├─sdc2        8:34   0   512M  0 part            7ACC-88E0
└─sdc3        8:35   0   1.8T  0 part            10820367462338150560
sdd           8:48   0   1.8T  0 disk
├─sdd1        8:49   0  1007K  0 part
├─sdd2        8:50   0   512M  0 part            7AD0-CC70
└─sdd3        8:51   0   1.8T  0 part            10820367462338150560

I did proxmox-boot-tool format /dev/sdb2 --force and proxmox-boot-tool format /dev/sda2 --force and finally seems its ok:
Code: 
root@server:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
7ACC-88E0 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)
7AD0-CC70 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)
834A-70DC is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)
83A6-25C6 is configured with: grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-3-pve)
root@server:~#

Now wait for resilver to reboot and test!

THANK YOU!
 
If I may disturb your futher I can tell I did added two new drives to rpool so the mirror pool rpool now listed 4 drives.

I rebooted and find it works well.

Code: 
root@server:~# zpool status rpool
  pool: rpool
 state: ONLINE
  scan: resilvered 1.45T in 05:04:53 with 0 errors on Tue Nov 26 10:22:57 2024
config:

        NAME                                                  STATE     READ WRITE CKSUM
        rpool                                                 ONLINE       0     0     0
          mirror-0                                            ONLINE       0     0     0
            ata-WDC_WD2000FYYZ-01UL1B2_WD-WCC1PEXKXVAT-part3  ONLINE       0     0     0
            ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4AUP65P-part3    ONLINE       0     0     0
            ata-MG06ACA10TE_60P0A0QAFL3G-part3                ONLINE       0     0     0
            ata-MG06ACA10TE_61F0A0EFFR7H-part3                ONLINE       0     0     0

errors: No known data errors

Now it's time to remove two old drives (actually, two old drive partition #3 each), I run

Code: 
root@server:~# zpool remove rpool ata-WDC_WD2000FYYZ-01UL1B2_WD-WCC1PEXKXVAT-part3
cannot remove ata-WDC_WD2000FYYZ-01UL1B2_WD-WCC1PEXKXVAT-part3: operation not supported on this type of pool

the same is when I tried remove both old partitons:

Code: 
root@server:~# zpool remove rpool ata-WDC_WD2000FYYZ-01UL1B2_WD-WCC1PEXKXVAT-part3
cannot remove ata-WDC_WD2000FYYZ-01UL1B2_WD-WCC1PEXKXVAT-part3: operation not supported on this type of pool
root@server:~# zpool remove rpool ata-WDC_WD2000FYYZ-01UL1B2_WD-WCC1PEXKXVAT-part3 ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4AUP65P-part3
cannot remove ata-WDC_WD2000FYYZ-01UL1B2_WD-WCC1PEXKXVAT-part3: operation not supported on this type of pool
cannot remove ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4AUP65P-part3: operation not supported on this type of pool

All I was able to do is make drives offline by running
Code: 
zpool offline rpool ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4AUP65P-part3

zpool offline rpool ata-WDC_WD2000FYYZ-01UL1B2_WD-WCC1PEXKXVAT-part3

and now I have this status:

Code: 
root@server:~# zpool status rpool
  pool: rpool
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: resilvered 1.45T in 05:04:53 with 0 errors on Tue Nov 26 10:22:57 2024
config:

        NAME                                                  STATE     READ WRITE CKSUM
        rpool                                                 DEGRADED     0     0     0
          mirror-0                                            DEGRADED     0     0     0
            ata-WDC_WD2000FYYZ-01UL1B2_WD-WCC1PEXKXVAT-part3  OFFLINE      0     0     0
            ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4AUP65P-part3    OFFLINE      0     0     0
            ata-MG06ACA10TE_60P0A0QAFL3G-part3                ONLINE       0     0     0
            ata-MG06ACA10TE_61F0A0EFFR7H-part3                ONLINE       0     0     0

errors: No known data errors

So to say, this is not the intended configuration. I plan to remove extra (old) disks from the server, and see rpool autoextended.

Please advice how to finish this journey, I'm afraid the miracle of zfs a bit unpredictable for me.

Thank you in advance!

UPD: I find a way, and it was zpool detach

Code: 
zpool detach rpool ata-WDC_WD2000FYYZ-01UL1B2_WD-WCC1PEXKXVAT-part3
zpool detach rpool ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4AUP65P-part3

Now the mirrored pool consists of two drives only, yet I struggle with its size (rpool still appears sized of 1.44Tb, not 9+Tb)
 
Last edited:
You must log in or register to reply here.
Top Bottom