[SOLVED] Autoexpand on rpool (ZFS)

fireon

Distinguished Member
Oct 25, 2010
4,123
388
153
41
Austria/Graz
iteas.at
Hallo,

i tested autoexpand a time ago on extra pool. This was working great. But what is with the rpool? Does this also work with autoexpand? We need to replace six 2TB HDDs with six 8TB HDDs. Actual PVE with nosubscription repo.

Thanks
 
I highly recommend testing it in a VM first (nested PVE with ZFS does not work with virtio controllers for the vdevs, but with e.g. sata it does), but in principle it should work. You have to take care to replicate the boot/grub partitions and MBR to the new devices as well, otherwise you will probably end up with an unbootable system.
 
It does not work, but it should. What have i done:
PVE install with 6 Disks in zfs raidz10, so installed PVE first on 3 disks and then add additional 3 disks. All disks are on 32GB. I set autexpand to on. After that i replaced disk for disk:
Code:
sgdisk -R /dev/sdf /dev/sda
sgdisk -G /devsdf
zpool replace rpool 10714300945297318711 sdf2
grub-install /dev/sdf
But after replace all the disk with 100GB disks, the pool is not growing up ;( What can i do?
Code:
NAME  SIZE  ALLOC  FREE  EXPANDSZ  FRAG  CAP  DEDUP  HEALTH  ALTROOT 
rpool  191G  1.66G  189G  -  0%  0%  1.00x  ONLINE  -

zpool status  
 pool: rpool 
state: ONLINE 
 scan: resilvered 6.34M in 0h0m with 0 errors on Thu Nov 24 11:05:16 2016 
config: 

 NAME  STATE  READ WRITE CKSUM 
 rpool  ONLINE  0  0  0 
 raidz1-0  ONLINE  0  0  0 
 sda2  ONLINE  0  0  0 
 sdb2  ONLINE  0  0  0 
 sdc2  ONLINE  0  0  0 
 raidz1-1  ONLINE  0  0  0 
 sdd2  ONLINE  0  0  0 
 sde2  ONLINE  0  0  0 
 sdf2  ONLINE  0  0  0 

errors: No known data errors
 
when you replicate with sgdisk -R, you leave the rest of the new bigger disk empty.. you did not resize the partitions, did you? or did you leave something out? ;)

anyway, I did a quick test with a file-based pool and a striped raidz1 like you have and it worked without issues:
Code:
# cd /backups
# for i in {1..6}; truncate -s 10g vdev$i.img
# zpool create testpool raidz1 /backups/vdev1.img /backups/vdev2.img /backups/vdev3.img raidz1 /backups/vdev4.img /backups/vdev5.img /backups/vdev6.img
# zpool status testpool
  pool: testpool
 state: ONLINE
  scan: none requested
config:

  NAME                    STATE     READ WRITE CKSUM
  testpool                ONLINE       0     0     0
    raidz1-0              ONLINE       0     0     0
      /backups/vdev1.img  ONLINE       0     0     0
      /backups/vdev2.img  ONLINE       0     0     0
      /backups/vdev3.img  ONLINE       0     0     0
    raidz1-1              ONLINE       0     0     0
      /backups/vdev4.img  ONLINE       0     0     0
      /backups/vdev5.img  ONLINE       0     0     0
      /backups/vdev6.img  ONLINE       0     0     0

errors: No known data errors
# zpool list testpool
NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
testpool  59.5G   126K  59.5G         -     0%     0%  1.00x  ONLINE  -
# zfs create testpool/test
# dd if=/dev/urandom of=/testpool/test/testfile bs=4M count=256
# for i in {7..9}; truncate -s 20g vdev$i.img
# zpool set autoexpand=on testpool
# zpool replace testpool /backups/vdev4.img /backups/vdev7.img
# zpool replace testpool /backups/vdev5.img /backups/vdev8.img
# zpool replace testpool /backups/vdev6.img /backups/vdev9.img
# zpool status testpool
  pool: testpool
 state: ONLINE
  scan: resilvered 257M in 0h0m with 0 errors on Thu Nov 24 11:58:00 2016
config:

  NAME                    STATE     READ WRITE CKSUM
  testpool                ONLINE       0     0     0
    raidz1-0              ONLINE       0     0     0
      /backups/vdev1.img  ONLINE       0     0     0
      /backups/vdev2.img  ONLINE       0     0     0
      /backups/vdev3.img  ONLINE       0     0     0
    raidz1-1              ONLINE       0     0     0
      /backups/vdev7.img  ONLINE       0     0     0
      /backups/vdev8.img  ONLINE       0     0     0
      /backups/vdev9.img  ONLINE       0     0     0

errors: No known data errors
# zpool list -v testpool
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
testpool  89.5G  1.50G  88.0G         -     0%     1%  1.00x  ONLINE  -
  raidz1  29.8G   766M  29.0G         -     1%     2%
    /backups/vdev1.img      -      -      -         -      -      -
    /backups/vdev2.img      -      -      -         -      -      -
    /backups/vdev3.img      -      -      -         -      -      -
  raidz1  59.8G   771M  59.0G         -     0%     1%
    /backups/vdev7.img      -      -      -         -      -      -
    /backups/vdev8.img      -      -      -         -      -      -
    /backups/vdev9.img      -      -      -         -      -      -
 
I search in the manpage from sgdisk for help, but nothing found. So how must i setup the new bigger disk that this is working for me too? ... maybe the "N" option from sgdisk can help me? The only problem is this sh...y little bootpartition...

sgdisk -N -a /dev/sde2

does not work, but also no error... or should i create the partitions manually with cfdisk?
 
Last edited:
for the boot disk, we (currently) format them like this in the installer to mimic the ZFS partition layout while allowing grub to be installed:
Code:
sgdisk -Z /dev/DISK
sgdisk -a1 -n1:34:2047 -t1:EF02 -n9:-8M:0 -t9:BF07 -n2:2048:0 -t2:BF01 -c 2:zfs

that basically means:
Code:
1: BIOS Boot partition (<=1M)
2: ZFS partition (all free space)
9: reserved space (8MB)

so if you use this layout on your new disks and then replace, it should work as expected (including installing grub into the BIOS boot partition). for non-boot disks, you could just add the whole disk ;)
 
I Test it, and it works like a charm. The only one thing was i must do an extra install grub on every disk, because the system was not bootable from ne disks. In my case, for example for every disk:
Code:
sgdisk -Z /dev/sdf
sgdisk -a1 -n1:34:2047 -t1:EF02 -n9:-8M:0 -t9:BF07 -n2:2048:0 -t2:BF01 -c 2:zfs /dev/sdf
zpool replace rpool 10714300945297318711 sdf2
grub-intall /dev/sdf
Thanks for the help Fabian.
 
I Test it, and it works like a charm. The only one thing was i must do an extra install grub on every disk, because the system was not bootable from ne disks. In my case, for example for every disk:
Code:
sgdisk -Z /dev/sdf
sgdisk -a1 -n1:34:2047 -t1:EF02 -n9:-8M:0 -t9:BF07 -n2:2048:0 -t2:BF01 -c 2:zfs /dev/sdf
zpool replace rpool 10714300945297318711 sdf2
grub-intall /dev/sdf
Thanks for the help Fabian.

yes, the grub install is to be expected - the disk is blank when you start, and zfs only cares about the contents of the second partition which is the actual vdev of the pool. it is a lot easier for non root pools, because then you can just "sgdisk -Z" and "zpool replace" ;)
 
for the boot disk, we (currently) format them like this in the installer to mimic the ZFS partition layout while allowing grub to be installed:
Code:
sgdisk -Z /dev/DISK
sgdisk -a1 -n1:34:2047 -t1:EF02 -n9:-8M:0 -t9:BF07 -n2:2048:0 -t2:BF01 -c 2:zfs

Can you tell me the command for EFI installation too? Thanks a lot!
Code:
partx -s /dev/sda 
NR  START  END  SECTORS  SIZE NAME  UUID 
1  2048  4095  2048  1M Grub-Boot-Partition  5c7c9f75-bf87-486d-9596-920024590b20 
2  4096  266239  262144  128M EFI-System-Partition 6f8a295f-15ea-4b77-959c-2abbc8e10190 
3 266240 1953523119 1953256880 931.4G PVE-ZFS-Partition  3fb748eb-7de3-4a2b-ac6e-800b052ceb5e
 
that one must have been done by you (or with a very old installer, IIRC there was at some point experimental ZFS+UEFI support?). we don't support ZFS+UEFI out of the box, because the EFI partition is only mounted (and updated) from (/on) one vdev with the current implementation in Grub. if you want, you can of course set such systems up yourself and take care of updating the EFI partition on all boot vdevs manually on each grub update.

that said, neither ZFS nor PVE care much about the (exact) partition layout. ZFS does prefer "whole disks", but that is not compatible with having the bootloader on the same disk anyway... so feel free to experiment and see what works for you
 
that one must have been done by you (or with a very old installer, IIRC there was at some point experimental ZFS+UEFI support?). we don't support ZFS+UEFI out of the box, because the EFI partition is only mounted (and updated) from (/on) one vdev with the current implementation in Grub. if you want, you can of course set such systems up yourself and take care of updating the EFI partition on all boot vdevs manually on each grub update.
that said, neither ZFS nor PVE care much about the (exact) partition layout. ZFS does prefer "whole disks", but that is not compatible with having the bootloader on the same disk anyway... so feel free to experiment and see what works for you
Oh my god... o_O ok i do an new installation, it would be easier. This was an default install with... i think a PVE 3.X Version, upgraded to 4.3 for a while...

So, actual UEFI and ZFS not a good idea. Ok, hope i can disable UEFI on this machine.
:) :)
 
Oh my god... o_O ok i do an new installation, it would be easier. This was an default install with... i think a PVE 3.X Version, upgraded to 4.3 for a while...

So, actual UEFI and ZFS not a good idea. Ok, hope i can disable UEFI on this machine.
:) :)

UEFI and ZFS are not a problem per se - you can boot from ZFS using UEFI just fine. the problem is that UEFI does not support redundancy out of the box and nobody seems to care ;) so you have your nice root pool with your desired redundancy, but the EFI partition is only current (or even only available!) on one vdev - if you lose that one you have to do manual recovery or you cannot boot at all. the PVE installer does not support EFI for ZFS setups to prevent users from running into this problem. as soon as a solution is available, we will re-add it to the installer.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!