Different size disks for zfs RAIDZ-1 root file-system

Shrinky999

New Member
Jul 9, 2015
2
0
1
Hi all!

I'm trying to install PVE 3.4 on a small server with the 3 spare disks I have lying around (1TB, 2TB, 3TB).
What I would like to do is to install it on a RAIDZ-1 of the three disks (as 1TB is more than enough for the moment and I'd switch out the disks one by one over the next few months).

Unfortunately the installer won't let me create a RAIDZ-1 for different size disks. As I couldn't find anything in the documentation (or googling): does anybody here have an idea how I could solve that problem?

Thank you!
Holger
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
14,280
534
133
the ISO install does not allow it, as this is a quite non-optimal zfs config. if you configure zfs later (manually via CLI), its possible. but of course, then you cannot use is as root partition.

so best practice, use identical disks.
 

Shrinky999

New Member
Jul 9, 2015
2
0
1
Thanks Tom!
I'm quite aware of the shortcomings of that approach.

I'm trying to figure out a safe way to host my data (read: redundant)
I'd prefer adding the raw disks to the VM (like described here: https://pve.proxmox.com/wiki/Physical_disk_to_kvm) and let the guest OS handle the zfs, but write speeds (of course) suffer.
Only other option would be to create the zfs RAID in proxmox, create a (raw?) disk image on it and attach that to a vm, correct? (This seems like a lot of overhead.)
Is there a "best practice" for adding zfs to a VM then?

Once I figure that out, I have enough disks of equal size :)

Cheers!
Holger
 

nixcz

New Member
Sep 16, 2014
11
0
1
the ISO install does not allow it, as this is a quite non-optimal zfs config. if you configure zfs later (manually via CLI), its possible. but of course, then you cannot use is as root partition.

so best practice, use identical disks.
Actually, I have a use case for booting from a (slightly) mismatched mirror set. It used to be pretty common in some sys-admin circles to use drives from different manufacturers in a raid set (not just drives from different production runs of the same manufacturer).

So if you have two drives of similar but not identical size that you want to use as your mirrored rpool, it's possible by adding the second drive manually after the installation.

The important part is to install on the smallest drive first. During the installation, choose ZFS, Raid0, and the smallest boot drive. Run the installation as usual.

After the system boots up, manually partition the second, larger drive and attach it to your ZFS rpool as a mirror.

One simple way to deal with the partitioning is to run `gdisk <first drive>`, hit 'x' for expert options, 'u' (to replicate partition table), enter the second drive, and if there's an error about the secondary header, just hit 'y' to let it fix it automatically.

If the partition size doesn't match exactly after the fix, it's probably also a good idea to manually resize the partition down to match the first as closely as possible, as this will help in drive replacement down the road. [Size mismatch shouldn't happen if the replication is successful, but it doesn't hurt to check.]

Also probably a good idea to change the GUID of the second disk since the replication seems to copy it from the first drive.

Run grub-install on the second drive, and zpool attach it to the first. Make sure you use the the device name ending in '-part2' which is the default GPT partition used on the first drive.

Appears to work for me. I've manually booted from either drive, but haven't yet tested a failed drive scenario. Don't know if grub will cooperate.

Also of course you should be comfortable manipulating drive partitions and have familiarity with ZFS operations in general. For my part, I'm mostly on FreeBSD so I aside from various Proxmox installs, I haven't booted linux on metal in a long time.

The following thread looks like it could be helpful for ZFS recovery and interacting with grub, should it come to it. But I haven't tested that far myself.

http://forum.proxmox.com/threads/21306-Grub2-recovery-on-ZFS-Proxmox-VE-3-4

As I mentioned, I don't spend much time in linux, so if someone sees any deficiencies in this set up, or have good reasons not to do it this way, I'd be interested in hearing them.
 
Last edited:

Erk

Member
Dec 11, 2009
149
3
18
Hi all!

I'm trying to install PVE 3.4 on a small server with the 3 spare disks I have lying around (1TB, 2TB, 3TB).
What I would like to do is to install it on a RAIDZ-1 of the three disks (as 1TB is more than enough for the moment and I'd switch out the disks one by one over the next few months).

Unfortunately the installer won't let me create a RAIDZ-1 for different size disks. As I couldn't find anything in the documentation (or googling): does anybody here have an idea how I could solve that problem?

Thank you!
Holger

Try running the Proxmox installer with only one disk attached, see if will let you create a RAIDZ-1 pool, then after everything is installed, attach the other two disks and add them to the pool from the command line.
 

Jota V.

Member
Jan 29, 2018
42
2
13
44
For people as same situation: We have a server with two different disks: 500GB (/dev/sda) and 1TB (/dev/sdb)

I've installed Proxmox node on smallest disk with ZFS Raid 0 (/dev/sda)

After installation (be careful with disk names in your environment) ;-)

zpool status -v

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0

# Replicate partitions
sgdisk --replicate=/dev/sdb /dev/sda
# Regenerate GUIDS
sgdisk --randomize-guids /dev/sdb
# Install Grub
grub-install /dev/sdb
# Attach disk
zpool attach rpool /dev/sda2 /dev/sdb2

zpool status -v

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
 
  • Like
Reactions: Iosif Bancioiu

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!