What now? - Please avoid using zpool upgrade on the "rpool" (root pool) itself, when upgrading to ZFS 2.0

efinley

Member
Jul 16, 2018
25
1
23
54
In the Proxmox VE 6.4 release notes known issues section it says:

Please avoid using zpool upgrade on the "rpool" (root pool) itself, when upgrading to ZFS 2.0 on a system booted by GRUB in legacy mode, as that will break pool import by GRUB.

ZFS 2.0.4 is also available in 6.3-6 which I'm running on a handful of hosts. The procedure was:
  • upgrade to 6.3-6 (from 6.3-3 which used ZFS 0.8.5)
  • reboot
  • zpool upgrade -a (which enabled all new features in ZFS 2.0.4 in both rpool and tank)
These systems are booted by GRUB in legacy mode, so now I'm thinking I won't be able to reboot successfully.

Anyone know of a way to recover from this that doesn't involve wiping and re-installing PVE?
 
Depending on when the systems were setup you might be able to simply switch over to booting with grub from the 2nd partition prepared by the installer (the installer created these since PVE 5.4):
* check with lsblk if your rpool drives have a 512MB partition (e.g. sda2)
* if yes try to intialize it with proxmox-boot-tool as described in the docs:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_proxmox_boot_tool
* afterwards try mounting it and checking that it does contain a grub config and the desired kernel-images

If the host was setup before pve 5.4 (or simply does not have the 512M partition) you could try setting up an usb-drive for booting (also with proxmox-boot-tool)

sadly with those issues and with hardware there always can be some problems - so make sure that you have a working and tested backup!

I hope this helps!
 
Just to clarify, does PVE with ZFS on rpool only work with UEFI boot as of PVE 6.4?
 
Last edited:
Just to clarify, does PVE with ZFS on rpool only work with UEFI boot?

This has been a source of confusion for me as well.

I know that during a 6.3.x (mid cycle) upgrade the rpool was offered a ZFS pool upgrade that at that time REQUIRED UEFI or it would not boot.

I was able to change my bios boot mode to UEFI then it works.

Moving forward I am confused if this was resolved on a new fresh install on legacy bios or what.

EDIT:

Just read the next line in the release notes that may answer:

Setups installed with the Proxmox VE 6.4 ISO are not affected, as there the installer always sets up an easier to handle, vfat formatted, ESP to boot.

This is interesting. I may want to reinstall in order to get this functionality on legacy BIOS. I have an older server that does have UEFI but its handling of UEFI devices is a bit weird.
 
Last edited:
I have this server running PVE 6.3 with ZFS 2.0.4 that I did 'zpool upgrade -a' on and didn't expect it to be able to reboot, but it did:

Code:
root@lindonhq-vsys01:~# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.106-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-8
pve-kernel-helper: 6.3-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-4.15: 5.4-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.13-3-pve: 5.3.13-3
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-9
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-5
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-10
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

Code:
root@lindonhq-vsys01:~# zpool get all rpool
NAME   PROPERTY                       VALUE                          SOURCE
rpool  size                           222G                           -
rpool  capacity                       36%                            -
rpool  altroot                        -                              default
rpool  health                         ONLINE                         -
rpool  guid                           11740802552937118642           -
rpool  version                        -                              default
rpool  bootfs                         rpool/ROOT/pve-1               local
rpool  delegation                     on                             default
rpool  autoreplace                    off                            default
rpool  cachefile                      -                              default
rpool  failmode                       wait                           default
rpool  listsnapshots                  off                            default
rpool  autoexpand                     off                            default
rpool  dedupratio                     1.00x                          -
rpool  free                           141G                           -
rpool  allocated                      81.0G                          -
rpool  readonly                       off                            -
rpool  ashift                         12                             local
rpool  comment                        -                              default
rpool  expandsize                     -                              -
rpool  freeing                        0                              -
rpool  fragmentation                  7%                             -
rpool  leaked                         0                              -
rpool  multihost                      off                            default
rpool  checkpoint                     -                              -
rpool  load_guid                      11913772034070002647           -
rpool  autotrim                       on                             local
rpool  feature@async_destroy          enabled                        local
rpool  feature@empty_bpobj            active                         local
rpool  feature@lz4_compress           active                         local
rpool  feature@multi_vdev_crash_dump  enabled                        local
rpool  feature@spacemap_histogram     active                         local
rpool  feature@enabled_txg            active                         local
rpool  feature@hole_birth             active                         local
rpool  feature@extensible_dataset     active                         local
rpool  feature@embedded_data          active                         local
rpool  feature@bookmarks              enabled                        local
rpool  feature@filesystem_limits      enabled                        local
rpool  feature@large_blocks           enabled                        local
rpool  feature@large_dnode            enabled                        local
rpool  feature@sha512                 enabled                        local
rpool  feature@skein                  enabled                        local
rpool  feature@edonr                  enabled                        local
rpool  feature@userobj_accounting     active                         local
rpool  feature@encryption             enabled                        local
rpool  feature@project_quota          active                         local
rpool  feature@device_removal         enabled                        local
rpool  feature@obsolete_counts        enabled                        local
rpool  feature@zpool_checkpoint       enabled                        local
rpool  feature@spacemap_v2            active                         local
rpool  feature@allocation_classes     enabled                        local
rpool  feature@resilver_defer         enabled                        local
rpool  feature@bookmark_v2            enabled                        local
rpool  feature@redaction_bookmarks    enabled                        local
rpool  feature@redacted_datasets      enabled                        local
rpool  feature@bookmark_written       enabled                        local
rpool  feature@log_spacemap           active                         local
rpool  feature@livelist               enabled                        local
rpool  feature@device_rebuild         enabled                        local
rpool  feature@zstd_compress          enabled                        local

Code:
root@lindonhq-vsys01:~# zpool status rpool
  pool: rpool
state: ONLINE
  scan: scrub repaired 0B in 00:01:25 with 0 errors on Sun Apr 11 00:25:26 2021
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda2    ONLINE       0     0     0
            sdb2    ONLINE       0     0     0

errors: No known data errors

Code:
root@lindonhq-vsys01:~# lsblk /dev/sda
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 223.6G  0 disk
├─sda1   8:1    0  1007K  0 part
├─sda2   8:2    0 223.6G  0 part
└─sda9   8:9    0     8M  0 part

root@lindonhq-vsys01:~# lsblk /dev/sdb
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb      8:16   0 223.6G  0 disk
├─sdb1   8:17   0  1007K  0 part
├─sdb2   8:18   0 223.6G  0 part
└─sdb9   8:25   0     8M  0 part

Code:
root@lindonhq-vsys01:~# efibootmgr -v
EFI variables are not supported on this system.

Now I'm wondering why it worked and if there is something specific that I can check to see if the others will also work?

Is not booting with GRUB legacy an rpool->zfs 2.0.4 thing or a PVE->6.4 thing?
 
Last edited:
I have this server running PVE 6.3 with ZFS 2.0.4 that I did 'zpool upgrade -a' on and didn't expect it to be able to reboot, but it did:
I'll try to summarize the conditions needed for a rpool not to be bootable by grub (not exclusive list - since there can also be issues with grub not being able to read past 2TB on some RAID-controllers (older generation HP for example)):

* ZFS changes the on-disk format with so called zpool features - quite well documented in the man-page (`man zpool-features`)
* Feature additions can happen (but need not) on every minor version upgrade (e.g. from 0.7.x -> 0.8.x , or 0.8.x -> 2.0.x) - see https://github.com/openzfs/zfs/blob/master/RELEASES.md
* running `zpool upgrade` on a pool sets the features to 'enabled' (meaning they can be used)
* some features are READ-ONLY COMPATIBLE (see `man zpool-features`) - they should not cause a problem
* activating a feature sets it to 'active' (e.g. by setting the compression to zstd with ZFS 2.0.0)
* grub is not able to read data from a pool which has an incompatible (and not read-only compatible) 'active' feature - this renders installs booting from grub to end up in the grub_recovery shell
** the list of features supported by grub can be seen in:
https://openzfs.github.io/openzfs-docs/Getting Started/Debian/Debian Buster Root on ZFS.html
http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/zfs/zfs.c#n276

So in theory running `zpool upgrade` on rpool booted by grub does not automatically render it unbootable - however setting some pool-property can happen as a side-effect causing the system to only get to the grub_rescue shell.
Check the zpool-features manpage for which features can cause the issue.

In any case,since booting with grub from a rpool is quite fragile we changed the booting to happen from a vfat partition with grub as well (with the PVE 6.4 ISO), systems booting with UEFI (` mount |grep efivarfs`) are booting from a vfat partition (with systemd-boot) since the PVE 5.4 ISO

I hope this explains it.
 
  • Like
Reactions: Ansy
Thanks for more info on this, but there is still confusion for me.

If we're running 6.3 created by an original ISO, on Legacy BIOS do you suggest we reinstall 6.4 from scratch to get vfat on grub?

Edit: I see post #2 has some info on trying to use the 512MB partition and initialize with proxmox-boot-tool. I guess I can try that and if it doesnt work I can reinstall. But it seems there is work to be done, we should not just assume everything is ok unless we do these steps for future.
 
Last edited:
If we're running 6.3 created by an original ISO, on Legacy BIOS do you suggest we reinstall 6.4 from scratch to get vfat on grub?
We're currently writing a HOWTO activate booting from vfat with grub for systems installed with Proxmox VE 5.4 or higher. (will try to post here once we put it online)

If reinstallation can be done easily in your situation (w.r.t. downtime, backup+restore time ...) this is certainly also a clean solution.

Edit: I see post #2 has some info on trying to use the 512MB partition and initialize with proxmox-boot-tool.
if the partition is clean and unused - running `proxmox-boot-tool format <partition> && proxmox-boot-tool init <partition>` should be enough
to get everything setup correctly (`proxmox-boot-tool status` should afterwards print which partition is used for booting with grub)

keep in mind that you could render your system unbootable or even delete important data - so make sure you have a working and tested backup!
 
We're currently writing a HOWTO activate booting from vfat with grub for systems installed with Proxmox VE 5.4 or higher. (will try to post here once we put it online)

@Stoiko: Please do not forget about that :)
 
would be great if you tell us if it worked well for you
Thanks!

Aha, great. Of course i will tell, but i don't think this will be this weekend.

EDIT: But when i now see this requirement for not having a problem:
  • System is not using ZFS as the root filesystem
I only use ZFS for my data storage (for VM's).

1620982996797.png

Only VMSTORE is ZFS based. My root (/dev/mapper/pve-root) is ext4. But i admit that i do not remember if i have UEFI boot mode enabled in BIOS. Most likely it is old (grub) boot type.

If so, does that mean that i can safely upgrade without problem? Thank you.

EDIT2: Eh, i (just need to read little bit further) that everything should be ok in my case.

Br
 
Last edited:
If so, does that mean that i can safely upgrade without problem? Thank you.
Usually yes - the issue with grub+ZFS is only relevant if grub needs to read from ZFS to get to the kernelimage and initrd (which are usually on the root filesystem)
 
Ok, for me this is great. Also, since grub reads from ext4 in my case i can also safely run

zpool upgrade

after upgrade & not expect any problem after?
Thank you!
 
Updating went fine. Now my PMox box is on 6.4-5, as it seems, everything is ok. More i can say after few days, when i will also run

zpool upgrade

Best regards!
 
Last edited:
The proxmox documentation kind of conflicts itself, and I feel the resolution is not the best one proposed.

The proposed resolution leaves people without a spare partition down a dark alley with no way out, yet we can see from the explanations here there is a far simpler resolution, simply make sure the incompatible feature is disabled on the zfs pool that is used by the root filesystem, thats it, no need for uefi conversion, and no need for fiddling with existing partitions.

Sadly its not been discovered what the incompatible feature is, but since I am planning to update to 6.4 so I can use persistent l2arc, I will test locally and post my findings here, hopefully then documentation can be updated.

I do accept of course that not running the upgrade command ensures the feature is off but it also keeps off all the other new features as well.
 
Last edited:
Well seems I had the foresight to leave unpartitioned space on both drives (25 gig free).

So I can make a 512meg vfat on both, but I still plan to test locally what zfs flags break grub, I think its probably the zstd compression.
 
I think its probably the zstd compression.
yes this would break grub compatibility - as would all other features not marked as 'read-only compatible yes' in `zpool-properties(5)`
 
Duplicated the setup by installing on 6.2 and destroying the EFI partition, will update to 6.4, and see if any carnage ensues.
 
Last edited:
Ok guys I have an update, it is good news.

First of all some output to show the configuration been tested.

Code:
# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
WARN: /dev/disk/by-uuid/7C8B-41CC does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
WARN: /dev/disk/by-uuid/7C8B-9B78 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping

This shows its been booted by grub (I also see grub on local display), and the 2 missing uuid's are the EFI partitions I deleted to mimic the server I have concerns with.

Code:
# ls /sys/firmware/efi
ls: cannot access '/sys/firmware/efi': No such file or directory

Further confirmation not an efi boot.

I initially just did a basic 6.2 to 6.4 update to have it boot from openzfs 2.0 on a system that was installed on zfs on linux 0.8x, and the system booted albeit with disabled features.

Code:
# dpkg -l | grep zfs
ii  libzfs4linux                         2.0.5-pve1~bpo10+1            amd64        OpenZFS filesystem library for Linux
ii  zfs-initramfs                        2.0.5-pve1~bpo10+1            all          OpenZFS root filesystem capabilities for Linux - initramfs
ii  zfs-zed                              2.0.5-pve1~bpo10+1            amd64        OpenZFS Event Daemon
ii  zfsutils-linux                       2.0.5-pve1~bpo10+1            amd64        command-line tools to manage OpenZFS filesystems

Code:
# zpool status
  pool: rpool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
config:

        NAME                                                 STATE     READ WRITE CKSUM
        rpool                                                ONLINE       0     0     0
          mirror-0                                           ONLINE       0     0     0
            ata-SAMSUNG_SSD_830_Series_S0VYNEAC704291-part3  ONLINE       0     0     0
            ata-SAMSUNG_SSD_830_Series_S0WJNEAC705087-part3  ONLINE       0     0     0

Code:
Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(5) for details.

POOL  FEATURE
---------------
rpool
      redaction_bookmarks
      redacted_datasets
      bookmark_written
      log_spacemap
      livelist
      device_rebuild
      zstd_compress

I then ran zpool upgrade to update it, with all new features set to enabled.

Code:
# zpool upgrade rpool
This system supports ZFS pool feature flags.

Enabled the following features on 'rpool':
  redaction_bookmarks
  redacted_datasets
  bookmark_written
  log_spacemap
  livelist
  device_rebuild
  zstd_compress

confirmed with zpool get all rpool and zpool status

Code:
# zpool get all   
NAME   PROPERTY                       VALUE                          SOURCE
rpool  size                           19G                            -
rpool  capacity                       8%                             -
rpool  altroot                        -                              default
rpool  health                         ONLINE                         -
rpool  guid                           13857213527387179236           -
rpool  version                        -                              default
rpool  bootfs                         rpool/ROOT/pve-1               local
rpool  delegation                     on                             default
rpool  autoreplace                    off                            default
rpool  cachefile                      -                              default
rpool  failmode                       wait                           default
rpool  listsnapshots                  off                            default
rpool  autoexpand                     off                            default
rpool  dedupratio                     1.00x                          -
rpool  free                           17.5G                          -
rpool  allocated                      1.54G                          -
rpool  readonly                       off                            -
rpool  ashift                         12                             local
rpool  comment                        -                              default
rpool  expandsize                     -                              -
rpool  freeing                        0                              -
rpool  fragmentation                  2%                             -
rpool  leaked                         0                              -
rpool  multihost                      off                            default
rpool  checkpoint                     -                              -
rpool  load_guid                      5200106711538973796            -
rpool  autotrim                       off                            default
rpool  feature@async_destroy          enabled                        local
rpool  feature@empty_bpobj            active                         local
rpool  feature@lz4_compress           active                         local
rpool  feature@multi_vdev_crash_dump  enabled                        local
rpool  feature@spacemap_histogram     active                         local
rpool  feature@enabled_txg            active                         local
rpool  feature@hole_birth             active                         local
rpool  feature@extensible_dataset     active                         local
rpool  feature@embedded_data          active                         local
rpool  feature@bookmarks              enabled                        local
rpool  feature@filesystem_limits      enabled                        local
rpool  feature@large_blocks           enabled                        local
rpool  feature@large_dnode            enabled                        local
rpool  feature@sha512                 enabled                        local
rpool  feature@skein                  enabled                        local
rpool  feature@edonr                  enabled                        local
rpool  feature@userobj_accounting     active                         local
rpool  feature@encryption             enabled                        local
rpool  feature@project_quota          active                         local
rpool  feature@device_removal         enabled                        local
rpool  feature@obsolete_counts        enabled                        local
rpool  feature@zpool_checkpoint       enabled                        local
rpool  feature@spacemap_v2            active                         local
rpool  feature@allocation_classes     enabled                        local
rpool  feature@resilver_defer         enabled                        local
rpool  feature@bookmark_v2            enabled                        local
rpool  feature@redaction_bookmarks    enabled                        local
rpool  feature@redacted_datasets      enabled                        local
rpool  feature@bookmark_written       enabled                        local
rpool  feature@log_spacemap           active                         local
rpool  feature@livelist               enabled                        local
rpool  feature@device_rebuild         enabled                        local
rpool  feature@zstd_compress          enabled                        local
root@AMD:~# zpool status
  pool: rpool
 state: ONLINE
config:

        NAME                                                 STATE     READ WRITE CKSUM
        rpool                                                ONLINE       0     0     0
          mirror-0                                           ONLINE       0     0     0
            ata-SAMSUNG_SSD_830_Series_S0VYNEAC704291-part3  ONLINE       0     0     0
            ata-SAMSUNG_SSD_830_Series_S0WJNEAC705087-part3  ONLINE       0     0     0

errors: No known data errors
.
rebooted again and absolutely no problems.

So the issue is when the flags are active rather than enabled, we already know zstd_compress is one of those flags, not sure about the other new features.

So it seems the issue is nowhere near as bad as has been indicated. As long as you dont blindly start using the new features without converting to the new proxmox boot tool first, everything should be fine, especially making sure to not start using zstd on the root zfs.

Creating a new pool will of course use zstd as its default, but that would be ok as the pool isnt required to be loaded by grub, creating a new dataset or volume would inherit the property from the rpool root. This is the unknown as to whether they inherit "on" with lz4 been the default or "on" with zstd been the default, to play safe I would specifically set the rpool to lz4 to ensure any child datasets and volumes will use lz4.

It is sad to see that by default proxmox still doesnt put grub on both vdev mirror's meaning if disk 1 fails I dont think proxmox would boot, I might file a bug report on that. However if proxmox by default on 6.4 and 7+ now uses proxmox-boot moving forwards then this lack of grub redundancy issue might be obsoleted.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!