Creating RAIDZ2 with different hard drive sizes

[Maggus]

Hello Proxmox Community,

I am new to everything: new to the forum, new to Proxmox, and new to Linux.

For my IT startup, I’ve decided to build my entire IT infrastructure on my server using Proxmox.

Currently, I’m facing what is probably a simple issue: I want to set up a RAIDZ2 cold storage pool, but it consists of different HDD sizes (3x10 TB + 3x12 TB). However, Proxmox gives me an error, stating that it cannot create the RAIDZ2 because the drives have different sizes.





# /sbin/zpool create -o ashift=12 datapool raidz2 /dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707C06 /dev/disk/by-id/ata-ST12000NM000J-2TY103_WV702XDL /dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707BQC /dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0RS5D /dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0RT4M /dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0PTBL
invalid vdev specification
use '-f' to override the following errors:
raidz contains devices of different sizes
TASK ERROR: command '/sbin/zpool create -o 'ashift=12' datapool raidz2 /dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707C06 /dev/disk/by-id/ata-ST12000NM000J-2TY103_WV702XDL /dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707BQC /dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0RS5D /dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0RT4M /dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0PTBL' failed: exit code 1

As a tech enthusiast, naturally, I first turned to ChatGPT for advice on this issue and received the following suggestions:

Option 1 (not recommended?):

zpool create -o ashift=12 datapool raidz2 -f \
/dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707C06 \
/dev/disk/by-id/ata-ST12000NM000J-2TY103_WV702XDL \
/dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707BQC \
/dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0RS5D \
/dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0RT4M \
/dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0PTBL

Option 2:

parted /dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707C06 mklabel gpt
parted /dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707C06 mkpart primary 0% 10TB

(Repeat for each drive...)

zpool create -o ashift=12 datapool raidz2 \
/dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707C06-part1 \
/dev/disk/by-id/ata-ST12000NM000J-2TY103_WV702XDL-part1 \
/dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707BQC-part1 \
/dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0RS5D \
/dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0RT4M \
/dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0PTBL

Since I don’t trust ChatGPT unconditionally, I thought it would be better to seek opinions from experts on the best way to proceed.

My Current Setup:​

Server:

• CPU: 56 x AMD EPYC 7453 28-Core Processor (1 Socket)
• Kernel Version: Linux 6.8.12-4-pve
• RAM: 128 GB

Storage:

• 2 x 512 GB SSD for Proxmox (zfs-raid1 rpool)
• 2 x 1 TB SSD (zfs-raid1 vmpool) for VMs, databases, etc.
• 3 x 10 TB + 3 x 12 TB HDD (raidz2 datapool) for cold storage data (the creation of which is my current problem)

Proxmox VE Version: 8.3.0

No hardware RAID controller is installed, and everything is running on ZFS.

I hope I have provided all the necessary information to help you with my question. If not, I am happy to supply any additional details.

I am very much looking forward to your feedback and assistance, learning a lot of new things, and diving deeper into the Proxmox universe and this forum.

Thank you so much for your help in advance!

Best regards,
Maggus

 

[Kingneutron]

Replace the 3x10TB drives with 3x12TB drives. Bingo, problem solved. (for RAIDZ2 at least)

If you don't have the budget for this, create a mirrored pool instead - but you'll still need 1x12TB drive to replace a 10TB.

So your pool would look like this:

mirror-0 10tb+10tb
mirror-1 12tb+12tb
mirror-2 12tb+12tb

Your I/O won't be balanced, but it will let you create the pool without -f. And you'll have a spare 10tb for when a drive fails.

BTW make sure you're using NAS-rated (or better, like Exos or SAS) drives. You don't want to build a zfs pool with desktop-class (e.g. Barracudas) or God forbid, SMR drives.

 

[ness1602]

You add -f on the whole zpool create sausage

 

[Maggus]

Thank you very much for the input.

The HDDs are Exos, so it seems I’m already on the safe side quality-wise.
Even though it would obviously be easiest to have all drives the same size, unfortunately, buying new ones isn’t an option as my budget is exhausted, and the drives are also brand new.

Is there a technical reason why creating the RAIDZ2 shouldn’t be forced, or why the 12TB drives can’t simply be partitioned smaller using the commands you mentioned? I find it hard to believe or understand why this wouldn’t be technically possible.

For instance, with Synology NAS systems, I know that larger drives are automatically adjusted to match smaller ones when creating a RAID1.
So I don’t yet understand the advantage or necessity of the mirrored pool as you described.

As I mentioned, I’m a beginner and would appreciate an explanation.

Once again, thank you for the input and support.

Best regards,
Maggus

 

[l.leahu-vladucu]

Hey Maggus, the man page for zpool-create explicitly says:

The use of differently-sized devices within a single raidz or mirror
group is also flagged as an error unless -f is specified.

This has probably been added as a sanity check to ensure that you don't make mistakes without noticing. Of course, you will only be able to use the size of the smallest disk. If you are sure that you did everything correctly, you can safely use -f (just make sure to double check that the command is correct before using -f).

 

[Maggus]

Hey l.leahu-vladucu,
thank you very much for your assistance.

Do you perhaps have a link to a resource (a man page for zpool create or similar) where I can read up on this in general? As a beginner, I'm currently finding it a bit difficult to locate the right sources for my questions.

If I understood you correctly, the command I would need to use manually is already mostly predefined by Proxmox, except for the -f parameter:

zpool create -o ashift=12 datapool raidz2 -f \
/dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707C06 \
/dev/disk/by-id/ata-ST12000NM000J-2TY103_WV702XDL \
/dev/disk/by-id/ata-ST12000NM000J-2TY103_WV707BQC \
/dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0RS5D \
/dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0RT4M \
/dev/disk/by-id/ata-ST10000NM018G-3CD103_ZRT0PTBL

Maybe I should just be a bit bolder and experiment more—after all, what’s the worst that could happen? That it doesn’t work, and I have to recreate the RAID. But I’d like to get it right from the start and make as few mistakes as possible, so I don’t have to troubleshoot issues that could have been avoided with a correct setup in the first place.

 

[l.leahu-vladucu]

Hey Maggus, you're welcome! I found the information by typing man zpool-create in my Proxmox VE terminal. You will find the same information here: https://openzfs.github.io/openzfs-docs/man/master/8/zpool-create.8.html

It's good to try your best, but also allow yourself to make some mistakes, as this is crucial in order to learn stuff In case you are trying to do some changes to mission-critical systems, it's of course always a good practice to test your changes in a testing environment before doing it on a real system (also for the future, in case something goes wrong, since you don't want the whole infrastructure of your IT startup to go down). But if you're still testing Proxmox VE and/or it's not an issue if you configure something incorrectly, feel free to try out things and learn! And always consult the man pages first, as they contain very useful information!

Good luck!

 

[ucholak]

in this cases i am using partitions on bigger drives, without problems.
/ your option 2

 

[Maggus]

Hello everyone,
after a longer forced break, I finally managed to continue working on this topic.
I successfully created the RAID-Z2 using the command previously described, and under my Node -> Disks -> ZFS, my new "datapool" is displayed without any errors (as far as I can tell).



I'm just a bit confused as to why my "datapool" isn't showing up as storage in the overview. What am I doing wrong? I seem to recall that when I created my "vmpool" as ZFS within the WebUI, it was immediately displayed as storage.



Does anyone have any idea why this might be?

Thank you in advance for your help.

Best regards,
Maggus

 

[mr44er]

Does anyone have any idea why this might be?

The zfs-pool ist now created, but you have to give it to proxmox. The GUI makes all steps at once, but you needed the console to use -f

Datacenter->Storage->Add->ZFS