Oh, ok... I knew that truly 4K native spinning disks are few, but I'm probably not up to date because I haven't used spinning for a long time.
Yet another "ZFS on HW-RAID" Thread (with benchmarks)
- Thread starter EdoFede
- Start date
-
- Tags
- hardware raid performance zfs
Oh, ok... I knew that truly 4K native spinning disks are few, but I'm probably not up to date because I haven't used spinning for a long time.
> So personally I see two possibilities:
https://www.reddit.com/r/truenas/comments/zz3sfg/hardware_raid_zfs_shoot_me_i_deserve_it/
https://www.truenas.com/community/r...bas-and-why-cant-i-use-a-raid-controller.139/
https://www.reddit.com/r/zfs/comments/135ku0i/zfs_on_top_of_multiple_hwraid0s/
https://openzfs.github.io/openzfs-d...uning/Hardware.html#hardware-raid-controllers
> On another three servers with ZFS on a proper HW-Raid controller, I've got already 2 disk failure without any issue.
Replaced, rebuilded, not a single error.
Every few months, some subgenius drops in and thinks they know everything. No, you've actually built a house of cards on quicksand. The wind just hasn't come by yet. You're going to lose your data when it falls over and sinks - and you won't be able to migrate your disks to anything but the same make and model of RAID card. Please do not encourage others to try replicating your foolishness.
- ZFS is so unreliable that it cannot work properly on hardware solutions (on which every other file systems works fine)
- The recommendation is simply derived from an exaggerated interpretation of what is actually a very interesting feature, but not so critical in its absence
https://www.reddit.com/r/truenas/comments/zz3sfg/hardware_raid_zfs_shoot_me_i_deserve_it/
https://www.truenas.com/community/r...bas-and-why-cant-i-use-a-raid-controller.139/
https://www.reddit.com/r/zfs/comments/135ku0i/zfs_on_top_of_multiple_hwraid0s/
https://openzfs.github.io/openzfs-d...uning/Hardware.html#hardware-raid-controllers
> On another three servers with ZFS on a proper HW-Raid controller, I've got already 2 disk failure without any issue.
Replaced, rebuilded, not a single error.
Every few months, some subgenius drops in and thinks they know everything. No, you've actually built a house of cards on quicksand. The wind just hasn't come by yet. You're going to lose your data when it falls over and sinks - and you won't be able to migrate your disks to anything but the same make and model of RAID card. Please do not encourage others to try replicating your foolishness.
Congratulations, great behavior for a technical discussion forum!
You simply didn't even read what I wrote in the entire thread, not even the very clear bolded parts.
If you are not able to deal with a technical dialogue by discussing topics without offending, your contribution is totally useless.
So Summary, the scenarios:
1., ZFS on RAW disks
2., ZFS on HWRAID
3., ZFS+SLOG on RAW disks
4., ZFS on OpenCAS (RAW disks)
I would be interesting to see the 3., and 4., scenario benchmarks.
The idea of the OpenCAS is the same for SLOG, but more robust ( middle-ware between the raw disk and zfs, basically a "smart software cache" solution)
The difference between 1., and 4., :
[ disk ] ----> [ zfs+slog ] -----> [ filesystem ] -----> [ VM ]
[ disk ] ----> [ opencas ] ----> [ zfs ] -----> [ filesystem ] -----> [ VM ]
The OpenCAS scenario, the separate "cache" device can be a "raid" (software/hardware) disk too (example: 2x NVME in raid-1 ) - while acting a proxy between the ZFS and the RAW disks.
1., ZFS on RAW disks
2., ZFS on HWRAID
3., ZFS+SLOG on RAW disks
4., ZFS on OpenCAS (RAW disks)
I would be interesting to see the 3., and 4., scenario benchmarks.
The idea of the OpenCAS is the same for SLOG, but more robust ( middle-ware between the raw disk and zfs, basically a "smart software cache" solution)
The difference between 1., and 4., :
[ disk ] ----> [ zfs+slog ] -----> [ filesystem ] -----> [ VM ]
[ disk ] ----> [ opencas ] ----> [ zfs ] -----> [ filesystem ] -----> [ VM ]
The OpenCAS scenario, the separate "cache" device can be a "raid" (software/hardware) disk too (example: 2x NVME in raid-1 ) - while acting a proxy between the ZFS and the RAW disks.
Last edited:
>To be clear, just last week I installed two servers with ZFS, flashing the PERC H330 controllers in IT mode (HBA330), because of the absence >of battery backed-up cache.
No need "IT mode", just use "WriteThrough" mode if battery backed-up cache absent (not writeback).
No need "IT mode", just use "WriteThrough" mode if battery backed-up cache absent (not writeback).
Last edited:
Bullshit. Example : newer lsi raid-card is backward-compatible with older lsi raid-cards. I myself replaced old dead raid-card some years ago. After reboot it prompts to "import foreign configuration" (which is stored on hw-raid10 virtual drive) and all ok.
This thread started as performance comprising between native ZFS raid and ZFS on top of H.Raid and half leg is it good to run ZFS on H.Raid.
I can tell you this - How you will set up that's way it will work.
If you want H.Raid to monitor HDD, I can suggest to set ZFS checksum=off otherwise ZFS will return IO error to software on checksum mismatch because ZFS will not have how to fix it.
One of my personal setup from 'a horror story' book.
As you can see stuffs can work in different ways
I can tell you this - How you will set up that's way it will work.
If you want H.Raid to monitor HDD, I can suggest to set ZFS checksum=off otherwise ZFS will return IO error to software on checksum mismatch because ZFS will not have how to fix it.
One of my personal setup from 'a horror story' book.
In one VM (external) I have limit in HDD space. I tried other things but I ended with creating ZFS ( in VM ) from files and files are accessed through SAMBA. Samba server is not local too
Files in samba server -> VM ZFS import pools
This way system is working for about 3 years. I`m planning to replace it. But as for this journey I had couple times to restore from backup due checksum problems. I do snapshot every 10 minutes to minimize the gap.
As you can see stuffs can work in different ways
What a strange recommendation!
I want my data delivered to me to be intact. If it got damaged by any means I want at least to know this fact - and not hidden!
Of course: your mileage may vary. Perhaps there may be circumstances where this may help reading "half destroyed" data. But for nomal usage on a daily basis this sound really like a bad advise.
Just my 2 €¢...
If you want to know then yes, keep it on. In the same situation others FS will continue to operate.
But keep in mind your VM will be broken or you movie will be half way watchable.
Hi,
Sitting in the same boat here ;-)
Did you also examined ya tests with dells fastpath feature ?
https://www.dell.com/support/manual...7396f8-b45d-4c95-86fd-ff00a09ede61&lang=en-us
Hi!
Yes, we always use No Read Ahead on the controller, only Write-Through caching with battery backed-up controllers.
I’d like to take this opportunity to point out that we’ve had 3 servers in production with this configuration for the past 16 months, and not a single issue has emerged.
This applies both to the hosts (zfs status, etc.) and to all the running VMs (around 80 VMs from 40 different clients).
If there really were a risk, as many claim, we would certainly have encountered issues over these months.
Therefore, I can confirm that the configuration is absolutely stable.
On the physical server dedicated to backups with PBS, since we don’t have an enterprise-grade controller available (as we do on the main servers), we chose instead a Raidz2 setup with disks fully managed by ZFS.
Bye!
Thankyou for this.. I am moving to proxmox from vmware. I have always used raid 6 on my h720pmini for my r730xd and h710p mini on my r720.. For people who complain about bit rot.. that is why there is a thing call partol read. I am thinking of doing my 8 disk raid 6 with single disk zfs.. sounds like this may be fine.. Plan is No Read Ahead, write-trouhg raid 6.. I saw mention of disabling checksum on zfs.. This would be interesting to look into.. I have been using enterprise HW always
Yes, you're (often) able to detect it, but you are not always able to heal it. I've had raid punch holes in all RAID controller variaties, yet never with ZFS.
It has autocorrect features.. I don't know I have been running raid in enterprise servers since 2005 and never have had bit rot.. Just have not been using xfs. Are we saying that xfs on top of a single raid vd will corrupt the data. If so then it is an xfs limitation or issue here.. A file system should never corrupt data. I have had a disks fail plenty of times.. but rebuilds fine I have never had bit root.. I am using enterprise controllers it would be different if I was using an ebay raid controller.
The term, risk, doesnt mean what you think it means. there's a RISK of a meteor falling on you. the fact it didnt happen doesnt lower that risk.
Same goes for the term absolutely. see Dunning-Krueger effect.
The difference between a RAID controller and a filesystem integrated RAID (eg, ZFS) is that ZFS is aware of fault at the FILE SYSTEM level. RAID does not. RAID can (and does) lead to silent corruption that you as the operator would not be aware of until you access the resource(s) impacted. Typical use patterns follow the 80/20 rule (80% of requests are for only 20% of the data) this may take years. or never.
The RISK of this happening is relatively low (think 1 in 10^14 iops) which means you may not ever see it happen. but it can and does.
Depends on the raid controller. we are talking about enterprise battery backed up raid controllers that does patrol read cycles to check every block on the devices for issues. When it sees an issue it fixes the issue.. We are not talking about a controller off amazon that is in a home built system. Sounds like zf's scrubs doesn't it? It does same exatc thing.. I would think having HW raid doing theses checks and xfs checking fs would be a good combination not a bad. I am talking using single vd device in the zfs not use its SW raid functionality. Plan is to leverage compression dedupe, and caching. Will keep HW raid with HW controller. I talked to a Linux subject matter expert at work who has been doing zfs for 20 years and he was like how would putting a zfs fs on a vd be an issue?
Last edited:
I would not say it is dangerous per se, I only think it adds an unnecessary point of failue. Simply because it is added hardware. Sure, the same thing is true for an HBA card, but not to the same extent, since it is less complex.
But my personal money quote from the linked blog post on how HW RAID is not dangerous with ZFS is this:
While some hardware RAID cards may have a "pass-through" or "JBOD" mode that simply presents each disk to ZFS, the combination of the potential masking of S.M.A.R.T. information, high controller cost, and anecdotal evidence that any RAID mode is about 5% slower than non-RAID "target" mode results in zero reasons for using a hardware RAID card with ZFS.
Why spend money on BBU RAID, when for less money you can get a PLP SLOG?
But my personal money quote from the linked blog post on how HW RAID is not dangerous with ZFS is this:
While some hardware RAID cards may have a "pass-through" or "JBOD" mode that simply presents each disk to ZFS, the combination of the potential masking of S.M.A.R.T. information, high controller cost, and anecdotal evidence that any RAID mode is about 5% slower than non-RAID "target" mode results in zero reasons for using a hardware RAID card with ZFS.
Why spend money on BBU RAID, when for less money you can get a PLP SLOG?
I forgot about this thread and that your question (and almost everything else I just said) was covered earlier in the thread
THIS.
Well key is the fact that i already have production h710p and h720p cards.. I am hesitant flashing to IT mode.. I plan to use a single vd with a single zfs vdev.. Disabiling No read ahead disables the controllers ability to anticipate future reads and cache them.. Wright Throuhg means the data is directly writen to the storage device and acknowledged to the host system only after write is completed without using the controllers cache for buffering writes. My plan is to use a enterprise Intel Optane Data center P3700 PCI nvme for slog when im done with my testing in about a month.. I want to leverage arc caching. on my esxi server i had this intel P3700 with a slog vmdk on a nested trunas vm for testing. I gave it like 128gigs of memory for arc cache and it worked very well on my 10gig network to my windows pc with 10gig networking also between my servers with a 10gig dac direct connect..
You can see on the raid controller it does checks and patrol reads to look for block corruption and mismatches kinda how scrubs work on zfs..
I would think this would actually be not a problem.. Granted the community may not have enterprise hardware like the original poster had. Another concern is if I have patrol read going on would it be best to turn off scrubbing in zfs since I am not using the built in raid and since I already have that functionality as noted above in my perc. I have had probably 100 power outages over the years, I have never had any controller issues even with hard shutdowns. Granted this was on vmware datastores, and ext3 and xfs on Linux. The intel card is a solid state card also so it would not lose data as with slog.. Also I read even if you lose a slog it does not damage the pool. I of corse am setting up my ups with NUT to do graceful shutdown of proxmox during an outage seems it had tools that will work better with my ups than my scripted way in esxi.
Example of patrol read running on my perc controller from tty log it cycles trough every block like zfs does with its checks.
06/28/25 3:00:00: prDiskStart: starting Patrol Read on PD=00
06/28/25 3:00:00: prDiskStart: starting Patrol Read on PD=01
06/28/25 3:00:00: prDiskStart: starting Patrol Read on PD=02
06/28/25 3:00:00: prDiskStart: starting Patrol Read on PD=03
06/28/25 3:00:00: prDiskStart: starting Patrol Read on PD=04
06/28/25 3:00:00: prDiskStart: starting Patrol Read on PD=05
06/28/25 3:00:00: prDiskStart: starting Patrol Read on PD=06
06/28/25 3:00:00: prDiskStart: starting Patrol Read on PD=07
06/28/25 3:00:00: EVT#18129-06/28/25 3:00:00: 39=Patrol Read started
ÿ06/28/25 3:57:51: prCallback: PR completed for pd=05
06/28/25 4:10:50: prCallback: PR completed for pd=01
ÿ06/28/25 4:11:59: prCallback: PR completed for pd=06
06/28/25 4:13:02: prCallback: PR completed for pd=07
ÿ06/28/25 4:20:19: prCallback: PR completed for pd=03
06/28/25 4:22:08: prCallback: PR completed for pd=04
ÿ06/28/25 4:32:36: prCallback: PR completed for pd=02
06/28/25 6:40:02: prCallback: PR completed for pd=00
06/28/25 6:40:02: PR cycle complete
06/28/25 6:40:02: EVT#18130-06/28/25 6:40:02: 35=Patrol Read complete
06/28/25 6:40:02: Next PR scheduled to start at 07/05/25 3:00:00
Last edited: