[SOLVED] Bootable ZFS disk's upgrade question.

[jim.bond.9862]

Hello everyone.
I am running a pve 5.3-11
It works well so at this point let it stay. Last time my upgrade didn't go well and I had to rebuild whole thing.
And my lxc's did not transferd so rebuild those. A pain.

That said however, my question is on zfs.

My setup is simple.
The system is loaded on 2 SSD in zfs mirror. The whole system. Than I have a zfs mirror pool for all containers on HDD.
And I also have several zfs mirror pools for all my data.
The proxmox server acts as vm host and data server via mapped volumes to lxc machines. Most of my data is media so it is shared via emby or jellyfin vm.

Last week one of the system SSDs in rpool failed.
Original disk's were 120gb.

I got several replacements that are 240gb.
I swapped the failed already and all seams to be working but I have a question. Since I now have disk's of different sizes in the pool, I want to replace the second disk to but is it better to not expand the pool size and leave the rest of the SSD to allow for wear management?
That is if the space is not used for anything will the drive use it for wear leveling , hence prolong the drives live till failure. Or it is not going to work and I might as well expand the pool to full size of the disks.

Also what is the best way to test bootabiluty of the setup after disk swap?
I mean I swapped one disk and system rebooted ok, but how do I know that it used new drive and not booted from the old one?

Thanks. Vl.


Last week one of

 

[jim.bond.9862]

Bump.
What no one can help me here?

Please guys, I need to finish this...

 

[jim.bond.9862]

Bump,

Any one?

 

[tom]

Please do not "Bump" your threads.

Instead, read the docs:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_zfs

And I recommend you test all this by installing Proxmox VE as a VM and you play around with replacing disks, so you can learn how to do all this before you risk your data.

 

[jim.bond.9862]

Please do not "Bump" your threads.

Instead, read the docs:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_zfs

And I recommend you test all this by installing Proxmox VE as a VM and you play around with replacing disks, so you can learn how to do all this before you risk your data.

Tom, my question is not about replacing disks. I already read all the wiki and know how to replace the disk.
My questions is specifically for using SSD based rpool and when disk replacement is larger than original disk , should the pool be expand or should I leave all the extra space to SSD wearleveling. Will it increases the ssd life span or will I be just waisting space.

Where in the docs I can find that info?

 

[budy]

Tom, my question is not about replacing disks. I already read all the wiki and know how to replace the disk.
My questions is specifically for using SSD based rpool and when disk replacement is larger than original disk , should the pool be expand or should I leave all the extra space to SSD wearleveling. Will it increases the ssd life span or will I be just waisting space.

Where in the docs I can find that info?

That would be the latter… SSDs will have to take care of their wear levelling themselves. This is nothing you leave to the file system. If you've got bigger SSDs and you can male good use of them as boot disks, by all means, go for it. However, I always prefer to keep the boot disks as small as possible since I don't like to carry surplus data on them.

In regards of replacing the failed mirror drive, you did perform a grub-install /dev/drive on the replacement drive after you had ZFS resilver the pool, right? And you also checked, if the pve host will also boot from the replaced drive alone, should you other fail as well?

Just my 2c, of course.

 

[jim.bond.9862]

That would be the latter… SSDs will have to take care of their wear levelling themselves. This is nothing you leave to the file system. If you've got bigger SSDs and you can male good use of them as boot disks, by all means, go for it. However, I always prefer to keep the boot disks as small as possible since I don't like to carry surplus data on them.

In regards of replacing the failed mirror drive, you did perform a grub-install /dev/drive on the replacement drive after you had ZFS resilver the pool, right? And you also checked, if the pve host will also boot from the replaced drive alone, should you other fail as well?

Just my 2c, of course.

Thanks, so in your opinion I should just update the second drive to larger one to keep disks the same in pool, and expand the pool to new size. No need to leave any empty space on disks as it will not have with wearleveling . Right?

As to your question , no I did not run grub-install as it was not on the guide I had. Will do it now. I also did not check if I can boot from new drive as I was waiting to get my question answered. I figured the best time to test is when I replacing the second disk. Why reboot the system extra time.

 

[budy]

Thanks, so in your opinion I should just update the second drive to larger one to keep disks the same in pool, and expand the pool to new size. No need to leave any empty space on disks as it will not have with wearleveling . Right?

Well… I don't know, what a "massive" rpool on a pve host will give you, since you're using different ZPOOLs for your VMs/containers anyway, but yes - the wear-levelling is up to the drives internals and you shouldn't need to worry about it nowadays. Long gone are days, when we had to short-stroke SSDs for that…

As to your question , no I did not run grub-install as it was not on the guide I had. Will do it now. I also did not check if I can boot from new drive as I was waiting to get my question answered. I figured the best time to test is when I replacing the second disk. Why reboot the system extra time.

I don't know which guide you used, but did it tell you that you would be getting a bootable ZFS mirror from it? The thing is, that there're usually 3 partions on your drive… and only one of them, that is the rpool, which makes up your root in pve, is managed by ZFS. The reasoning to go ahead and actually try to boot the system with only either of these mirror disks operational is simply, that you will proove that the setup works as intended, especially, since you already replaced one of those disks. But this is of course up to you…

 

[jim.bond.9862]

Well… I don't know, what a "massive" rpool on a pve host will give you, since you're using different ZPOOLs for your VMs/containers anyway, but yes - the wear-levelling is up to the drives internals and you shouldn't need to worry about it nowadays. Long gone are days, when we had to short-stroke SSDs for that…



I don't know which guide you used, but did it tell you that you would be getting a bootable ZFS mirror from it? The thing is, that there're usually 3 partions on your drive… and only one of them, that is the rpool, which makes up your root in pve, is managed by ZFS. The reasoning to go ahead and actually try to boot the system with only either of these mirror disks operational is simply, that you will proove that the setup works as intended, especially, since you already replaced one of those disks. But this is of course up to you…

Well I don't know if I call 240gb massive , by today's standards

But that is what triggered my confusion and this post.
I was kind of going with my old-school knowledge. I remember reading some.time ago that if you building system boot on SSD it is a good idea not to format the disk to full capacity and leave some unused space , so the disks wearleveling algorithm have more space to manage bad blocks and read/write calls thus extending usable disk life expectancy.

As for the guide I used, it did told me that in the end I should have a bootable system.

I used gdisk to copy partition table and randomize the I'd.
I use DD to copy boot partition from the good SSD to new one.
And use zfs replace to replace the bad disk with new one.
Each and every step worked as described.

But since the old disks were 120gb and new are 240gb. I think I should replace the old disk with new one to keep a matching pair.

Thanks for your help.

 

[jim.bond.9862]

Ok have to resurect this thread as I just found out my new disk is not bootable. I run grub install and nothing.
Reading the guide again and the docs and not sure how to proceed next.

A few more questions for gurus.

1. What is the safest way to upgrade system from 5.3 to 6 ,
I guess?

2. Best backup how to of the system so I do not loose the current setup.
I lost it last time when moving from 5.1 to 5.3.

It is zfs but I do not have other zfs system or external zfs disk.

Thanks

 

[budy]

Can you show how your rpool is setup? Also which disks are involved?

 

[jim.bond.9862]

root@pve2:~# zpool status rpool
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 5.41M in 0h0m with 0 errors on Sat Apr 25 12:43:11 2020
config:

        NAME                                   STATE     READ WRITE CKSUM
        rpool                                  ONLINE       0     0     0
          mirror-0                             ONLINE       0     0     0
            ata-SATA_SSD_20011624000987-part2  ONLINE       0     0     0
            sdr2                               ONLINE       0     0     6

errors: No known data errors
root@pve2:~#

The are 2 SSD. Sdr and sdp
Sdp is the new one and I added by Id.
Sdr is original working.


lsblk
...
sdp 8:240 0 223.6G 0 disk
├─sdp1 8:241 0 1007K 0 part
├─sdp2 8:242 0 111.8G 0 part
└─sdp9 8:249 0 8M 0 part
...
sdr 65:16 0 111.8G 0 disk
├─sdr1 65:17 0 1007K 0 part
├─sdr2 65:18 0 111.8G 0 part
└─sdr9 65:25 0 8M 0 part


________
I used this guide
https://edmondscommerce.github.io/replacing-failed-drive-in-zfs-zpool-on-proxmox/

To replace the failed one

 

[budy]

And you did ran grub-install /dev/sdp ? I am actually not a 100% sure, if the disks being of different sizes does matter, since I only used that on disks of equal sizes.

 

[jim.bond.9862]

As far as I know it shouldn't matter.but I will re run the grub and retry boot
Let you know

 

[jim.bond.9862]

nope, run grub-install on both disks . boots from original not from new.

this is result of pveversion -V if it helps

root@pve2:/# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-12-pve)
pve-manager: 5.3-11 (running version: 5.3-11/d4907f84)
pve-kernel-4.15: 5.3-3
pve-kernel-4.15.18-12-pve: 4.15.18-35
pve-kernel-4.15.18-10-pve: 4.15.18-32
pve-kernel-4.15.18-8-pve: 4.15.18-28
pve-kernel-4.13.13-2-pve: 4.13.13-33
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-47
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-12
libpve-storage-perl: 5.0-39
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-23
pve-cluster: 5.0-33
pve-container: 2.0-35
pve-docs: 5.3-3
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-18
pve-firmware: 2.0-6
pve-ha-manager: 2.0-8
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-2
pve-xtermjs: 3.10.1-2
qemu-server: 5.0-47
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo1

 

[jim.bond.9862]

Ok, I am marking this as solved.
Turned out it was my bios that is totally screwed up. It not allows more than 2 boot disks, really? A server MB only allows 2 boot devices, but I had to fiddle with settings for like 3 hours untill I found where and what need to be set for it to work.