Node after hardware changes

[digidax]

Hello,

I have to replace a storage controller of a node. After doing this, I expect that the node will not boot because ZFS with RAID-10 was used.

        NAME                                              STATE     READ WRITE CKSUM
        rpool                                             DEGRADED     0     0     0
          mirror-0                                        ONLINE       0     0     0
            scsi-36003005702306c0025ea3079135e3ac2-part3  ONLINE       0     0     0
            scsi-36003005702306c0025ea3079135e8cf5-part3  ONLINE       0     0     0
          mirror-1                                        DEGRADED     0     0     0
            scsi-36003005702306c0025ea3079135edf8f        FAULTED      7   334     0  too many errors
            scsi-36003005702306c0025ea3079135f3275        ONLINE       0     0     0

The disks are named as "by-id" to the ZFS storage pool and I have read, that this information will be lost after changing the storage controller.
https://wiki.archlinux.org/index.php/persistent_block_device_naming#by-id_and_by-path
"the by-id will change when the device is plugged into a port of a hardware controller subject to another subsystem"

So my plan is to do a new installation of PVE.

Question: The node will have the same IP address and name. May I have to remove it from the cluster "pvecm delnode NodeName" or can I leave it?

Thanks, Frank

 

[AKA]

You'll need to delete it.

Everything can be found here: https://pve.proxmox.com/wiki/Cluster_Manager
Under "Remove a Cluster Node"

If, for whatever reason, you want this server to join the same cluster again,
you have to


reinstall Proxmox VE on it from scratch

then join it, as explained in the previous section.





After removal of the node, its SSH fingerprint will still reside in the known_hosts of the other nodes. If you receive an SSH error after rejoining a node with the same IP or hostname, run pvecm updatecerts once on the re-added node to update its fingerprint cluster wide.

 

[digidax]

Thanks, other than expected the node did booting and starting up normally. But ZFS have now used a non persistent addressing of the disks:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda3    ONLINE       0     0     0
            sdb3    ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
        spares
          sde       AVAIL  
          sdf       AVAIL

Is there a way to change the members of the pool from /dev/sda to /dev/disk/by-id/ ?

# ls -l /dev/disk/by-id/ata*
lrwxrwxrwx 1 root root  9 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C64WE -> ../../sda
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C64WE-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C64WE-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C64WE-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C65N5 -> ../../sdd
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C65N5-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C65N5-part9 -> ../../sdd9
lrwxrwxrwx 1 root root  9 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C7WDS -> ../../sdf
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C7WDS-part1 -> ../../sdf1
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C7WDS-part9 -> ../../sdf9
lrwxrwxrwx 1 root root  9 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C92GE -> ../../sde
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C92GE-part1 -> ../../sde1
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C92GE-part9 -> ../../sde9
lrwxrwxrwx 1 root root  9 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C95PF -> ../../sdb
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C95PF-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C95PF-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1C95PF-part3 -> ../../sdb3
lrwxrwxrwx 1 root root  9 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1CA7H6 -> ../../sdc
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1CA7H6-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Oct 22 11:59 /dev/disk/by-id/ata-ST1000DM010-2EP102_ZN1CA7H6-part9 -> ../../sdc9

Thanks, Frank

 

[AKA]

Hi, it doesn't really matter.

If ZFS finds the pool anyway, you don't need to worry.
Otherwise you can tell ZFS with the -d option in the import command where to find disks for the pool.
for example: zpool import -d /dev/disk/by-id/ rpool

 

[digidax]

Thanks a lot good to know - really nice features.

Which disks should have the bootloader of the RAID 10 array?
Am I right, that it will be the members of mirror-0: sda and sdb?
In the case of replacing on of the two disks, I have to install grub on the new one or does it do ZFS automaticly?
Now, grub is on sda and sdb:

# dd bs=512 count=1 if=/dev/sda 2>/dev/null | strings
ZRr=
`|f
\|f1
GRUB
Geom
Hard Disk
Read
 Error
# dd bs=512 count=1 if=/dev/sdb 2>/dev/null | strings
ZRr=
`|f
\|f1
GRUB
Geom
Hard Disk
Read
 Error

 

[AKA]

It only needs to be on one disk (However that would be stupid in terms of disk failure).
Technically it can be on all disks in the pool, so the server can boot from all disks.

ZFS should handle the rest (I'm assuming your root in on ZFS too).