ZFS With Raid 1

[Isaac.maag]

Hello! I am new to the raid world. I have two identical 256 gb nvme ssds and want to set them up as a mirrored raid (Raid1). When I go to disks it shows both drives but when I go to create a ZFS storage pool I only see one of the drives (the one that doesn't have the boot partition). Can I not use the hard drive the the boot partition is on? If I can how to I get it to show up? I really need help with this as I can't find a single forum or article that relates to my exact issue. Any help would be awesome! Thank you! If more information is needed I am happy to provide. Below are screenshots of my current configurations.

 

[Gilberto Ferreira]

Hello! I am new to the raid world. I have two identical 256 gb nvme ssds and want to set them up as a mirrored raid (Raid1). When I go to disks it shows both drives but when I go to create a ZFS storage pool I only see one of the drives (the one that doesn't have the boot partition). Can I not use the hard drive the the boot partition is on? If I can how to I get it to show up? I really need help with this as I can't find a single forum or article that relates to my exact issue. Any help would be awesome! Thank you! If more information is needed I am happy to provide. Below are screenshots of my current configurations.
View attachment 57929View attachment 57930

Hi

I suppose you need to do this in the installer, when you booted from the ISO image whit USB stick for instance.
Then you will se the Options to create a RAIDz1.

Like this

Cheers

 

[Isaac.maag]

Hi

I suppose you need to do this in the installer, when you booted from the ISO image whit USB stick for instance.
Then you will se the Options to create a RAIDz1.

Like this
View attachment 57931
Cheers

Thanks for the fast response! Is there no way to do this without a reinstall?

 

[Dunuin]

Then you will se the Options to create a RAIDz1

"Raid1", not "Raidz1" which is Raid5.

Thanks for the fast response! Is there no way to do this without a reinstall?

No, because your PVE system disk is using LVM and not ZFS. You will have to backup your VM + LXCs + "/etc" folder and reinstall it as a ZFS mirror which will wipe both disks.

 

[Isaac.maag]

"Raid1", not "Raidz1" which is Raid5.


No, because your PVE system disk is using LVM and not ZFS. You will have to backup your VM + LXCs + "/etc" folder and reinstall it as a ZFS mirror which will wipe both disks.

Awesome I will do that! Another question, is it a better practice to make a separate user other then root to login? I would prefer to be as secure as possible. I was going to make a randomly generated password for root but don't want to enter that every time I login. What do you recommend?

 

[Dunuin]

I was going to make a randomly generated password for root but don't want to enter that every time I login. What do you recommend?

Thats what password managers are for. A different randomly generated way too long password for everything that will be autofilled to the forms after unlocking the save with your master password.
And yes, you could create another user with nearly full privileges. But it won't be possible to do everything with that user, as some stuff is hardcoded so only root can do it.

 

[Isaac.maag]

Thats what password managers are for. A different randomly generated way too long password for everything that will be autofilled to the forms after unlocking the save with your master password.
And yes, you could create another user with nearly full privileges. Not it won't be possible to do everything with that user, as some stuff is hardcoded so only root can do it.

Awesome thanks a ton for your help! You have been super helpful! Last question, just so I understand how this zfs raid setup works. If I setup my two drives to be in a raid 1 ZFS pool does that protect my Virtural machines and my OS in case one drive fails? Or does this only protect the OS? I want to make sure anything I work on inside a VM is backed up in case a drive fails I don't want to lose everything. Thanks! And is ZFS raid 1 with two drives good enough performance for home use?

 

[LnxBil]

If I setup my two drives to be in a raid 1 ZFS pool does that protect my Virtural machines and my OS in case one drive fails?

Yes

And is ZFS raid 1 with two drives good enough performance for home use?

Performance wise, a raid 1 of two disks is as fast as the slowest disk used alone.

 

[ubu]

RAID is to keep your machine running, when a disk fails, BACKUP is for protection

ZFS is great, but it handles a bit differently

 

[Dakater]

Just want to add some usefull informations on the raid1 informations.

1. Yes, it is recommended to install Proxmox with ZFS as raid.

2. Yes, you better use mechanic disks (hdd) and no ssd! Or buy yourself special industrie ssd. (None Multi-Layer!)

3. If you use it for home improvement you can turn of a lot of ZFS logging which will be a bit better for a "standard ssd". Myself using 2x 120gb soho ssd (12$ each), BUT ...

4. It is NOT recommended to use any md raid, BUT you can. First install Debian 12.2 or higher. Create a normal raid1 with mdadm in the shell or right away in the GUI installer.

5. (u)EFI and a new BIOS can kill your plans. Old mainboards support a "Legacy mode". Then it is the old easy way.
New BIOS are now the first days without this option. You aew forced to EFI (or more harder boot-security).
That leads to an EPS partition, a swap partition and a root partition (at least).

6. There is no way to directly raid1 with EPS, BUT ... After installing with 2x SSD you can create md0 for swap and md1 for root and just ignore the EPS partition. That leads to a raid1 without any EPS failover.
After that just ADD in the active EPS partition in the boot order the 2nd inactive EPS partition. Copy the active boot partition to the inactive one.
-> If sda fails then sdb has an active EPS partition. While 1st is down 2nd will be used.

7. Replacing on ZFS and with mdadm on EXT4 is something total different.

8. LVM containering WILL not prevent you against the EFI limitations. Your EPS will not be "directed" correctly.

9. mdadm with Proxmox CAN lead to a major problem where you could destroy all your data.
ZFS will destroy your hardware very fast, if its not the right one!

10. My toughts: Put your Proxmox on cheap ssd. With raid1 and zfs or "single" and always hold a 1:1 copy. So if you change elementary setup on proxmox then copy again 1:1 to your 2nd ssd. In case of failure you just have to replace. I use a 6x SSD cage in a 5,25" slot. 2x cheap ssd for proxmox (right now hard testing it with md-raid and irs working awesome). 2x 2TB SSD for Data (raid1) and of course your fast NVMe. Put them in a raid1 and use it for the base install of your guest OS (Linux, Windows, Hackintosh)

And always keep in mind:
A raid is never a backup.
A NAS or SAN is never a backup even in HA mode.
Even a cluster spreaded on several actice-active SAN's with HA-iSCSI connection of each guest is not a backup or high security.
One controller error and ...

Wish you luck and I suggest ZFS with GUI install flr easy start plus logging off plus learn to replace a disk before installing active (needed) guests.

 

[Dunuin]

6. There is no way to directly raid1 with EPS, BUT ... After installing with 2x SSD you can create md0 for swap and md1 for root and just ignore the EPS partition. That leads to a raid1 without any EPS failover.
After that just ADD in the active EPS partition in the boot order the 2nd inactive EPS partition. Copy the active boot partition to the inactive one.
-> If sda fails then sdb has an active EPS partition. While 1st is down 2nd will be used.

And these days you use the proxmox-boot-tool to keep your individual ESPs in sync.

 

[Dakater]

And these days you use the proxmox-boot-tool to keep your individual ESPs in sync.

That is right for the build in ZFS-way, but using SSD or NVMe is not recommended this way.

But it would be new for me that the boot-tool can do that with any md-raid on ext4-root plus swap plus EFI. Can it?

 

[Dunuin]

That is right for the build in ZFS-way, but using SSD or NVMe is not recommended this way.

As far as I know you always make use of the proxmox-boot-tool. No matter what type of storage you choose and no matter if it is grub or systemd boot once you install PVE from a PVE 6.4+ ISO.

But it would be new for me that the boot-tool can do that with any md-raid on ext4-root plus swap plus EFI. Can it?

You can't use mdadm raid when installing PVE via the PVE ISO. So you would be forced to install a Debian with its Debian Bootloaders and then of cause such PVE features like the proxmox-boot-tool wouldn't be used.

 

[alexskysilk]

That is right for the build in ZFS-way, but using SSD or NVMe is not recommended this way.

You mention recommendations for quite a few things. who are you quoting as doing the recommending?

 

[Dakater]

As far as I know you always make use of the proxmox-boot-tool. No matter what type of storage you choose and no matter if it is grub or systemd boot once you install PVE from a PVE 6.4+ ISO.


You can't use mdadm raid when installing PVE via the PVE ISO. So you would be forced to install a Debian with its Debian Bootloaders and then of cause such PVE features like the proxmox-boot-tool wouldn't be used.

Right. That is what I wrote. Debian 12.2 with update to PVE. You have to do some special settings but that's all about. After removing the debian kernel you can use your pve kernel with mdadm.
By the way: I do not recommend to use any md-raid on proxmox - but you even can update your pve from an ISO-install with mdadm.
In this case you need the same amount of hdds/ssds like you are using after standard pve-iso.

Anyways. My point was not to use mdadm for fun or why I "think" its better then ZFS (its not). But if your hardware is not the right one for ZFS it will be expensive.
Losing any 12$ SSD is not losing any NVMe.

I prefer to use ZFS for PVE with the right hardware of course. But if you dont have the right hardware and dont want to install 2 additional garbage ssd (or industrial type) and you just need to use your system as it is (a soho system) then md-raid over debian with upate to pve isna great "middle way".

You mention recommendations for quite a few things. who are you quoting as doing the recommending?

Depends on your content.
If you are answering a thread and you did not get the topic during the conversation it can happen.

I did a straight recommendation.
Dont use ZFS on soho hardware. As SSD it should be SLC or at least MLC. But better use a hdd for it.
If you are forced to use soho hardware then dont use ZFS with full logging.
Anyways while some flash storges will allow you 10.000 writing operations each cell a SLC will have 100.000 to 1.000.000 write operations.
Just one Intel 256GB SLC SSD will have a price about 150-200$.

And the guests? If mounted to a ZFS Raid1 (2x 2TB Samsung) you will have to replace those soon. Even with ZFS logging to minimum.

Just saying, while there is a high need on RAID1 and more and more advanced home users are using a PVE they will run in that problem using the storage wrong!
And even if there is a risk with md-raid (read about the synch problem on this board or on google) it is s nice middle way.
You can use your soho-hardware with md-raid on proxmox until the day you setup everything a better way. (e.g. a small HA-SAN for iSCSI for all pve-systems in your network @10gbe or just clustering with the right hardware in your pve system.

Anyways:
1) You can use ZFS Raid1 with soho-hardware, but its not the right way.
2) You can use md-raid with your pve and it will never be the right way - but your hardware will survive much longer.
3) You can buy the right hardware for all guests to use ZFS, but you have to spent a lot of money.
4) Best is a dedicated SAN (HA at least n+1 and a hardware RAID6 or RAID10 depends on what you need) for serving iSCSI to all pve, esxi or whatever virtualization you like to use. That would be of course the "high roller class" including managed switches (stacked at least n+1 with Jumbo frame option) - you will have no matter with ZFS ir whatever filesystem you want to use. Your iSCSI Lun is your "dedicated hdd" and after mapping you can do everything with this "storage" as it is dedicated installed.

Gave you all options now for no more irritations. My english is not best, but I wont use ChatGPT for translations without an emergency. I am always trying to get better in it - whatever

 

[Dunuin]

Dont use ZFS on soho hardware. As SSD it should be SLC or at least MLC. But better use a hdd for it.

SLC and MLC are of cause more durable than TLC and QLC. But there is no manufacturer anymore that is producing SSDs with SLC or MLC NAND. Even with proper Enterprise SSDs you are forced to use QLC or TLC NAND. So using TLC NAND is totally fine for ZFS...but that SSD should have a power-loss protection to reduce wear and performance drop caused by sync writes.

3) You can buy the right hardware for all guests to use ZFS, but you have to spent a lot of money.

Enterprise SSDs aren't that expensive anymore. You can get some nice second-hand eMLC Enterprise SSDs for 60€ per TB. Or even new NVMe TLC Enterprise SSDs starting at 70€ per TB. If you account for the "price per TB written" and not the "price per TB capacity" those "cheap" crappy SSDs are the way worse deal with the higher priced Enteprise SSDs actually being the cheaper option on the long run.

 

[Dakater]

SLC and MLC are of cause more durable than TLC and QLC. But there is no manufacturer anymore that is producing SSDs with SLC or MLC NAND. Even with proper Enterprise SSDs you are forced to use QLC or TLC NAND. So using TLC NAND is totally fine for ZFS...but that SSD should have a power-loss protection to reduce wear and performance drop caused by sync writes.


Enterprise SSDs aren't that expensive anymore. You can get some nice second-hand eMLC Enterprise SSDs for 60€ per TB. Or even new NVMe TLC Enterprise SSDs starting at 70€ per TB. If you account for the "price per TB written" and not the "price per TB capacity" those "cheap" crappy SSDs are the way worse deal with the higher priced Enteprise SSDs actually being the cheaper option on the long run.

Can you link or recommendate me some NVMe or SSD @Enterprise grade?
TLC would be better then QLC anyways, but those eMLC was new to me. Will read about that in detail tomorrow.
Maybe TLC with all that ZFS logging off is the golden soho-middleway or that eMLC.

This year should be an upgrade for our business san at work. I think end of the year I can use iSCSI for myself at home because no one needs old hardware (but me)
In this case would be nice to know if TLC and eMLC is a fine way for the base pve @Raid1 ZFS. Anythink else will be on 2x 4 disks @ Raid10 @ 2x 10gbe in my network (cant await the day of hardware change)

 

[alexskysilk]

You can use ZFS Raid1 with soho-hardware, but its not the right way.

This is just a "truism." hardware exists for all manner of use cases. for a home use case there's nothing wrong with using TLC and even qlc ssds; performance will suffer on sustained writes but on balance zfs is still preferrable on this hardware then other non filesystem checksum aware file systems. its not the "wrong way" either.

You can use md-raid with your pve and it will never be the right way - but your hardware will survive much longer.

no idea what this means. write endurance is write endurance... non cow file systems still write whole data.

You can buy the right hardware for all guests to use ZFS, but you have to spent a lot of money.

everything is relative. making blanket statements in a vacuum make no sense at all. If I have downtime losses in the $10000+/min, I design a solution that has high survivability. Spending $50000 on such a solution is dirt cheap at 5 minute downtime prevented. alternatively, my home server being down is just an annoyance- I'd wonder if its worth spending $100 extra.

Best is a dedicated SAN (HA at least n+1 and a hardware RAID6 or RAID10 depends on what you need) for serving iSCSI to all pve, esxi or whatever virtualization you like to use.

SANs are not well supported in PVE. you are limited to lvm thick unless you want to deploy gfs/ocfs2 which are completely unsupported by the devs- but again, solutions exist for particular problems. just because I dont drive an 18 wheeler doesnt mean I cant go to the hardware store.

 

[alexskysilk]

2. Yes, you better use mechanic disks (hdd) and no ssd!

Anyone saying that never had to sit through a rebuild/resilver with SMR drives

HDDs are never a "better" option for virtualization, as they are an order of magnitude poorer in IOPs. I'd take the cheapest consumer SSDs over HDDs for that use case. QLC SSDs in garbage collection mode still respond quicker than HDDs.

 

[Dakater]

SANs are not well supported in PVE. you are limited to lvm thick unless you want to deploy gfs/ocfs2 which are completely unsupported by the devs- but again, solutions exist for particular problems. just because I dont drive an 18 wheeler doesnt mean I cant go to the hardware store.

Just add open-iscsi to your pve and go for it. I dont see any problems in using it. You can add several LUN's or just make it the easy way.

Reducing it to home use:
Storage with HW raid1 and 2x HDD/SSD. (no HA)
External HDD USB for Backup LUN's directly attached to storage.
10gbe uplink to a single soho unmanaged switch.
And if you are lazy and just using one hardware system for pve you can just add one big LUN. I prefer a LUN each guest. More work, but more flexible and at least 2x active pve hardware systems can each use one LUN on the hw raid.
Anyways. Never used iSCSI before on my PVE but on Debian with bare qemu - so I have hope left for this solution. Was nice after finding the right settings.
No - I would not buy myself a HA-SAN solution. It would lead to the single way above with backup, but with iSCSI like at work. Anyways I need and love my network storage - not just for direct connections like iSCSI as a LUN, but for sharing data accross my home-systems (including TV and remote home) controlling.

My gaming (virtual) system storage will not run on iSCSI but dedicated on my personal system (my gaming nvme storage needs no raid!), but I will test the performance with my Nvidia (@passthrough) in combination with iSCSI. Just for fun and I bet the performance will not be worse. GTA V or D3 should run in full details. Does not matter if dedicated harddisk or iSCSI