How to configure ZFS, 3 drives total - 2 + 1 hot spare

[Jared302]

Hello,

I am quite new to proxmox, but enjoying it so far.

I currently have 3, 2TB SATA hard drives, and I'm trying to set them up in proxmox to be in a RAID where there are 2 main drives with 1 as a hot spare, so if 1 drive fails the hot spare will replace it until I get a new drive. So this should be 4TB of data in total.

I've tried to create a RAIDZ with them, but once it's done it says there is 6TB available which doesn't seem right. There was not an option to select one as a hot spare, I had to select all 3 of them.

I created a dRAID with them (one data dev and one spare), and that allowed me to enter 1 as a spare. It shows as 4TB free which looks right, but when I try and add that 4TB as a hard drive on my Win10 VM, it only says there is only 2TB capacity / available.

I've spent a while researching this but can't seem to find a solution. I do have an IT background, but mostly with Windows GUI servers and hardware raid.

Thanks

 

[Dunuin]

I currently have 3, 2TB SATA hard drives, and I'm trying to set them up in proxmox to be in a RAID where there are 2 main drives with 1 as a hot spare, so if 1 drive fails the hot spare will replace it until I get a new drive. So this should be 4TB of data in total.

You will only get 4TB when using raid0 and there a hot spare would be useless as losing a single disk means all data is lost.

I've tried to create a RAIDZ with them, but once it's done it says there is 6TB available which doesn't seem right. There was not an option to select one as a hot spare, I had to select all 3 of them.

A raidz1 requires 3+ disks like a normal raid5. If you want a hot spare you need 4 disks. And the 6TB could be correct. It depends on the point of view. "zpool" will report size including parity data (so basically the total raw capacity), "zfs" will report size with parity data already subtracted (so the usable capacity). So when it tells you the pool is 6TB this could actually mean you only got 4TB (or actually more like 3.2TB as you shouldn't fill a ZFS pool more than 80%).

I created a dRAID with them (one data dev and one spare), and that allowed me to enter 1 as a spare. It shows as 4TB free which looks right, but when I try and add that 4TB as a hard drive on my Win10 VM, it only says there is only 2TB capacity / available.

Don't use draid unless you got plenty of disks (15+ disks are recommended if I remember correctly).

 

[Jared302]

You will only get 4TB when using raid0 and there a hot spare would be useless as losing a single disk means all data is lost.


A raidz1 requires 3+ disks like a normal raid5. If you want a hot spare you need 4 disks. And the 6TB could be correct. It depends on the point of view. "zpool" will report size including parity data (so basically the total raw capacity), "zfs" will report size with parity data already subtracted (so the usable capacity). So when it tells you the pool is 6TB this could actually mean you only got 4TB (or actually more like 3.2TB as you shouldn't fill a ZFS pool more than 80%).


Don't use draid unless you got plenty of disks (15+ disks are recommended if I remember correctly).

Thanks so much for the quick response! Much appreciated

I just created a RAIDZ with all 3 disks, and indeed it does show 6TB total, however when adding that as an HDD to the Win10 VM, it shows 4TB available so looks like you were right.

So if I RAIDZ three 2TB HDD's, will I still have redundancy / my 4TB of data if one disk fails?

I will be doing a cloud backup of the pool every day, but it would be nice if there was still some local redundancy.

Thanks

 

[Dunuin]

So if I RAIDZ three 2TB HDD's, will I still have redundancy / my 4TB of data if one disk fails?

Yes one might fail without a problem.

But before creating your first VM make sure to increase the "Block size" of the ZFS storage from 8K to 16K. Otherwise you will lose an additional TB due to padding overhead even if you can't see this directly. If you already created that VM you would need to do a backup+restore after increasing the block size so the virtual disks get destroyed and recreated with the correct block size.

 

[Jared302]

Yes one might fail without a problem.

But before creating your first VM make sure to increase the "Block size" of the ZFS storage from 8K to 16K. Otherwise you will lose an additional TB due to padding overhead even if you can't see this directly. If you already created that VM you would need to do a backup+restore after increasing the block size so the virtual disks get destroyed and recreated with the correct block size.

Great, thanks!

I tried re-creating the raid with ashift = 16 size, however the total storage was only 2.90TB when trying to add an HDD to the VM, with ashift = 12 the total storage was 3.87TB.

I'm only going to be using the ZFS storage for a data disk on the VM, as I already have the VM installed on a different SATA SSD.

 

[Dunuin]

I tried re-creating the raid with ashift = 16 size, however the total storage was only 2.90TB when trying to add an HDD to the VM, with ashift = 12 the total storage was 3.87TB.

Not "ashift = 16". "volblocksize = 16K" when using ashift=12 which you do by editing your ZFS storage in the webUI and setting the "Block size" field to "16K".

 

[Jared302]

Not "ashift = 16". "volblocksize = 16K" when using ashift=12 which you do by editing your ZFS storage in the webUI and setting the "Block size" field to "16K".
View attachment 60041

Perfect, thanks for the screenshot!

Looks all good now, thanks for the help!

 

[z40]

Yes one might fail without a problem.

But before creating your first VM make sure to increase the "Block size" of the ZFS storage from 8K to 16K. Otherwise you will lose an additional TB due to padding overhead even if you can't see this directly. If you already created that VM you would need to do a backup+restore after increasing the block size so the virtual disks get destroyed and recreated with the correct block size.

Do I need to do a VM backup in this situation? Isn't it enough to migrate the VM to another resource and then migrate it back?

 

[Dunuin]

Do I need to do a VM backup in this situation? Isn't it enough to migrate the VM to another resource and then migrate it back?

Depends. If it is a shared storage it won't help, as the same vdisk will be referenced by multiple nodes.

 

[z40]

Depends. If it is a shared storage it won't help, as the same vdisk will be referenced by multiple nodes.

I have two servers (SERVER1 and SERVER2) and on each server two ZFS volumes - STORAGE1 and STORAGE2 (identical configuration on each - RAIDZ on 4x HDD 10k). Server-to-server replication is enabled for the VM.

My concept:
I'm turning off replication.
I'm moving all VMs from SERVER1/STORAGE1 to SERVER1/STORAGE2. Then I delete the STORAGE1 on the SERVER1 and create it again with a new block size (16kb).
I'm moving VMs from SERVER1/STORAGE2 to SERVER1/STORAGE1.

Is it OK?

What happens if I enable replication now? SERVER1/STORAGE1 will have a block size of 16kb and SERVER2/STORAGE1 will be left with a block size of 8kb.

 

[Dunuin]

You could simply test it. Use "zfs get volblocksize" to get the volblocksize of the zvol before and after the migration/replication. But as far as I understand, replication will make use of "zfs send | zfs receive" and the volblocksize of the source zvol should be kept on the target pool.
I think the "move disk" should work as it is storage agnostic and will create a new virtual disk and then copy its contents?