ZFS vs Hardware RAID

[LnxBil]

Which raid controllers do you see, usually ?

Normally, the default built in ones like P4xx on HP, Perc on Dell, MegaRaid 3008 on non-branded cards all on 1 or 2 HE dual socket servers with at most 2 disks (also some diskless stations). Yet as I said, we almost always use FC-based SAN storage for everything.

 

[mir]

Normally, the default built in ones like P4xx on HP, Perc on Dell, MegaRaid 3008 on non-branded cards all on 1 or 2 HE dual socket servers with at most 2 disks (also some diskless stations). Yet as I said, we almost always use FC-based SAN storage for everything.

Why don't you go all in a simply skip the extra layer of things that can brake and simply use diskles servers pxe-booting over FC? Your servers will likely be of any use if the FC-based SAN is down anyway.

 

[LnxBil]

Why don't you go all in a simply skip the extra layer of things that can brake and simply use diskles servers pxe-booting over FC? Your servers will likely be of any use if the FC-based SAN is down anyway.

Easy: Customers want this.

 

[mir]

Easy: Customers want this.

Customer is always right

 

[morph027]

At least they think so

 

[Rhinox]

- flexibility

Could you make an example where ZFS/mdam is "flexible" and hw-raid not?

- costs (the LSI CacheCade/Nytro thing is friggin' expensive)

About as expensive, as RAM + CPU for your server. People tend to "forget" that in case of ZFS/mdadm all parity-calculation must be done by CPU, and RAM is used for caching. HW-raid controller has dedicated resources for this.

- well proven to scale out heavily (just google for ZFS best practice with 99 disks )

You can equally find hw-raid arrays with hundreds of disks.

- not lost data since we are using it, even with hardware failures (we'd lost huge amount of data to a failed raid controller once)

Lost hw-raid controller once, but not a single bit of data. Simply replaced it for similar one, booted, done.

- SNAPSHOTS! (our backup loves it)

I prefer to leave snapshots to filesystem.

... probably many more...

The same for hw-raid too...

 

[morph027]

Could you make an example where ZFS/mdam is "flexible" and hw-raid not?

I'm not bound to any vendor locks. Just plug and play. Take another machine, plug the disks, boot any live system with ZFS built in, ready to go.

Add another pair of mirrors to a stripe set? No problem. Set a mirror level of more then two? Works.

About as expensive, as RAM + CPU for your server. People tend to "forget" that in case of ZFS/mdadm all parity-calculation must be done by CPU, and RAM is used for caching. HW-raid controller has dedicated resources for this.

Not really, our Nytro Controllers and the CacheCade licenses are expensive as hell. 1000€ per Controller ... that's a lot of memory. And my CPU does not blame about resilvering....it's just XOR

You can equally find hw-raid arrays with hundreds of disks.

Sure, but are they built with this purpose in mind?

Lost hw-raid controller once, but not a single bit of data. Simply replaced it for similar one, booted, done.

Thats the point. I do need the exact same controller...doubling the price.

I prefer to leave snapshots to filesystem.

ZFS is a filesystem

The same for hw-raid too...

Thats right.

In conclusion, i feel better with not using hardware raid. Others do. That's ok. I guess, there are even workloads/circumstances which might lead me to deploy a hardware controller (ZFS needs a certain level of knowledge, cant' demand this from every customer). I love diversity. There are needs for all solutions.

 

[Rhinox]

...Add another pair of mirrors to a stripe set? No problem. Set a mirror level of more then two? Works.

This of course can be done by hw-raid controller too...

Not really, our Nytro Controllers and the CacheCade licenses are expensive as hell. 1000€ per Controller ... that's a lot of memory. And my CPU does not blame about resilvering....it's just XOR

Of course, you can select the most expensive raid-controller in the world. But why would you do that? You can get controller with the same chipset (LSI SAS 2208) for ~200 and for the price you get 2 cores, supercap, cache, and 8x SAS-ports.

Sure, but are they built with this purpose in mind?

Of course they are.

Thats the point. I do need the exact same controller...doubling the price.

No, you do not need exact the same controller. Just any type with the same chipset (i.e. Intel, IBM, Dell, Fujitsu, etc, they all use LSI). BTW, even motherboard could fail and you'd have to replace it too. It costs much more, and is not so easy to replace.

I love diversity. There are needs for all solutions.

Me to. That's why I'm using both hw- as well as sw-raid.

 

[Alessandro 123]

No, you do not need exact the same controller. Just any type with the same chipset (i.e. Intel, IBM, Dell, Fujitsu, etc, they all use LSI). BTW, even motherboard could fail and you'd have to replace it too. It costs much more, and is not so easy to replace.

I'm not really sure about this.
DELL usually customize the controller firmware a lot. Firmware updates are not the same as LSI.
I don't think you could replace a DELL PERC with an LSI MegaRAID (same chipset) without issue.

 

[morph027]

Thats what happened to use...same controller, different revision....

 

[Rhinox]

Not only firmware is modified, hardware too (i.e. IBM has vertical ports and supercap, original LSI has horizontal ports and bbu). And yet I could replace LSI with IBM or Intel. Trick is, configuration is not stored in controller, but on drives. That way arrays are recognised and it does not matter what vendor-specific version of firmware you have in controller...

BTW, customisation is done outside of firmware. You could flash IBM/Intel/Dell controller with original LSI-firmware and yet after flashing it still reports as Dell PERC or IBM ServeRaid...

 

[Alessandro 123]

Did you replace a branded card with the LSI one with success ? How many times ?

 

[Rhinox]

I replaced once original LSI (iirc, it was 9266-8i type) with re-branded IBM M5016. Reconnected SFF8087/SAS cables, and everything worked without touching configuration.

btw I asked LSI-support some time ago about replacing controller. They said their chips are backward compatible. So if the old raid-controller had LSI SAS-2108, it can be safely replaced by any with the same or newer chip (SAS-2208). But you have to check if the new controller has necessary features unlocked (i.e. raid6)...

 

[morph027]

The LSI things are really good. I was impressed a lot with all their features and performance boosters like CacheVault/CacheCade ...

But when i tried ZFS on an old lab machine, i was impressed too

However, as i said, there always will be scenarios which will suit either controllers or software raid.

Also i saw really strange setups build by people which can't free their minds cause ages ago somebody told them that raid controllers are the messias. There was a customer which ran a virtual Ceph cluster (inside vms with virtual disks) on top of ESXi on top of a RAID. Unbelievable.

 

[alexskysilk]

This has been litigated to death on here and many other places; RAID and ZFS are NOT ANALOGOUS and are not meant to be. Yes, ZFS has the ability to perform RAID functions but its just a side effect; its real strength is the fact that it is a filesystem AND an LVM, which means there is no separation between file allocation and block allocation.

RAID provides block storage, which means it needs to have a file system placed on it to be of use. the RAID subsystem is completely disconnected from the file system, which means it is unaware of any file system faults. ZFS is not limited to block level fault, it is aware of file level fault and can drill down to see if the cause is block level or file level related, and address both. In other words, IF you have the option to run ZFS (or BTRFS, or GPFS, etc) on bare block level devices you will have a MUCH improved control over your data reliability; It also means that higher level features stemming from CoW functions (snapshots, versioning, ZFS send, etc) are present as a file level feature but is embedded in the block level so there is no waste or lost performance due to their use.

The comparison discussion about performance, portability, etc is not really pertinent because they are apples and oranges.