mdadm

In addition: 9 users to 10 will not update raid firmware. "mdadm" is selfupdating, when you update the kernel, mdadm is automatically updated. And kernel updates are much more common than firmware upgrade (firmware upgrade most of the time requires a boot cd or similiar, most of the time Debian is not supported, thus you can't upgrade from PVE, you have to reboot to the upgrade cd, and so on. Or you have to use the DELL Update from the BIOS, that 9 times to 10 is not connecting properly to their FTP server)
 
Even with ZFS if you don't use ZIL stored on SSD with power loss protection.
Even with low-end HW RAID (that you are supporting) and forcing writeback cache (like most users does because without that, it's a pain)

Anyway, this issue is solved in kernel 4.4 by using a journal (feature made by facebook AFAIK)



This is the same for any hw raid that you are supporting. In addition, any hw raid will put your data at risks much more than mdadm, because you are relying to a closed source firmware, where bugs are discovered only after failures and so on.



That's false.
If you use cheap hardware, is not an mdadm failure, but your failure with bad hw planning.

What you have described also applies to hw raid, but hw raid is supported and mdadm not. This is non-sense.
hw raid expose data to much more risks than mdadm. What if hw card fails? what if you have to move disks to a different server with dropped support to that hw card (like all DELL/HP does with every generation change) ?

I have many many server with PERC H700 that is unsupported, untested, unusable on any subsequent hardware. As DELL tends to change the platform every 2 years, evety 2 years you'll hit this issue. If you put an H700 to an R730, you won't get any support from DELL.
Also, my H700 is not compatible (due to a different and proprietary form-factor) to any recent DELL server. Same vendor but incompatible hardware.

Forcing users to use hw raid expose to more risk than using mdadm.

And, anyway, users should be free to make their own decisions. But you can't say that with mdadm you'll loose data and with hw raid not.
I'm working with servers and PC from about 2001. I've never ever lost a single bit on server with mdadm (even after unclean shutdown, that are very rare, by using UPSs). On the other half, i've lost about 2 PB of data (in multiple servers, multiple times) due to some issues with hw raid (bad chip, bad firmware, card broken, card that started to write garbage, depleted battery not detected by the raid card and thus with write-back still enabled, unclean shutdown that totally destroyed the raid configuration, unclean shutdown that destroyed some data even with a new battery (less than 3 moths) ) and so on.


At no point are they stopping any one using it, it might only be a few lines of code, however if Proxmox took every approach with a feature they won't support but people want it would just be a huge mess. Proxmox provide the option to install ontop of a custom Debian install.

Installing via the packages are as close to an ISO as your going to get, and upgrades e.t.c are identical, unless they slapped a big not supported Term's and Conditions that had to be accepted every time you used Software Raid in the installer there will always be someone that uses it and then wants support, their are plenty of business out their that make it very difficult to use a custom setup, for example Cisco Software / VM's will do hard checks against the hardware it run's on, I would say of all the software I have had experience with deploying Proxmox is very open.
 
t
At no point are they stopping any one using it, it might only be a few lines of code, however if Proxmox took every approach with a feature they won't support but people want it would just be a huge mess. Proxmox provide the option to install ontop of a custom Debian install.

Installing via the packages are as close to an ISO as your going to get, and upgrades e.t.c are identical, unless they slapped a big not supported Term's and Conditions that had to be accepted every time you used Software Raid in the installer there will always be someone that uses it and then wants support, their are plenty of business out their that make it very difficult to use a custom setup, for example Cisco Software / VM's will do hard checks against the hardware it run's on, I would say of all the software I have had experience with deploying Proxmox is very open.

I agree. But motivations due to lack of support to mdadm are very feasible. they are supporting a much worse system
 
What is the system they support which you would say is much worse than mdadm?

As wrote above, any hardware raid. There is almost no reason, in 2017, to stick to an hardware card for multiple reasons

mdadm would be a good compromise, not the definitive solution, but a compromise. Every part of mdadm is better than any hardware card.
 
As wrote above, any hardware raid. There is almost no reason, in 2017, to stick to an hardware card for multiple reasons

mdadm would be a good compromise, not the definitive solution, but a compromise. Every part of mdadm is better than any hardware card.
I think your forgetting a hardware raid card requires absolutely no support in the Proxmox installer, as the installer just manages it as if you was installing directly to one single disk, what your asking is for them to make a system available to install and configure mdadm. Which again is something they decided not to support, but give you all the power to do manually.

Also your statement using a HBA, half of your "con's" for Hardware Raid card then would also be exactly the same for a HBA card.
 
Please expose at least one drawback of mdadm that is not present in any hw card.
The implementation passes pointers to userspace data to the underlying storages individually when using O_DIRECT (cache=none in qemu) causing the write job for each disk to read the buffer from userspace independently allowing any unprivileged user to cause it to write different data to each disk for the same write request by simply racing some writes into the buffer in a second thread - reading it back immediately puts the raid into degraded state.

https://bugzilla.kernel.org/show_bug.cgi?id=99171
 
yes totally serious
Really? You said "every" (sic!) part of mdadm is better than hw-raid. Ok, so let's discuss a few things:

System cpu-load (especially during array-reconstruction): lower with mdadm, or with hw-raid?
Or OS-support: better with mdadm, or hw-raid?
Array reconstruction: easier on mdadm, or hw-raid?
Boot-loader support: easier with mdadm, or hw-raid?
Arrays: easier done with partitions (mdadm) or with the whole disk (hw-raid)?
System cpu-load: does it impact more mdadm, or hw-raid?

Etc, etc. I could think of many reasons when hw-raid is better (and many reasons when sw-raid is better). I have been using both hw-raid as well as mdadm/fakeraid/zfs for many years, but I do not dare to say one of them is definitely better in every aspect...
 
And, anyway, users should be free to make their own decisions.

Sure, you can make your own decisions - simple install using the Debian installer and you get mdadm support if you want it.

As developers, we are also free to make our own decisions (based on our experience) ...
 
System cpu-load (especially during array-reconstruction): lower with mdadm, or with hw-raid?

negligible with modern hardware. I've rebuild a 20TB RAID-6 during peak hours in our backup server and during 6 heavy, parallel, rsync backups. I'm running the same rsync every night. There was absolutetly no delay. each backup took about 9 hours during the whole rebuild, exactly the same time that they take without a rebuild.

Today backup (with no rebuild) took 9 hours and 11 minutes.

What about ZFS ? It's the same. But ZFS is supported........

Or OS-support: better with mdadm, or hw-raid?
Absolutetly better with mdadm, that's kernel native. hw-raid requires some drivers.

Array reconstruction: easier on mdadm, or hw-raid?
Very similiar, mdadm is able to start rebuild automatically, just remove and plug the disk.

Boot-loader support: easier with mdadm, or hw-raid?
It's the same.

Arrays: easier done with partitions (mdadm) or with the whole disk (hw-raid)?
You can (you should) use the whole disks as raid member, even with mdadm

System cpu-load: does it impact more mdadm, or hw-raid?

As wrote above: negligible.

Etc, etc. I could think of many reasons when hw-raid is better (and many reasons when sw-raid is better). I have been using both hw-raid as well as mdadm/fakeraid/zfs for many years, but I do not dare to say one of them is definitely better in every aspect...

Ok, but these "drawbacks" are the same with ZFS. But ZFS is supported. PVE already support a software raid , the one inside ZFS.
But ZFS in small hardware is not suggested (it's an enterprise filesystem developed for high-end hardware). Using ZFS on low-end hardware, without ECC, without tons of memory, without a ZIL (used as writeback) is not suggested. In these cases, mdadm is far better as you'll loose all the cool things (like bit-rot protection) coming from ZFS. You can't have bit-rot protection without ECC memory. You don't have any "writeback" cache or "power-loss" protection without ZIL on SSD.

When all of this is missing, mdadm is better. All of this is alwyas missing with low-end devices and in this cases your are almost forced to use hwraid
 
negligible with modern hardware.
Yes, for raid1/0/10. But did you ever try reconstruction of raid5/6 array with ~20 drives?

Absolutetly better with mdadm, that's kernel native. hw-raid requires some drivers
AFAIK, mdadm is supported only by linux. HW-raid is supported by any major OS. In every case, you need driver (either as part of linux-kernel sources, OS-installation disk, or as extra sw-package).

Very similiar, mdadm is able to start rebuild automatically, just remove and plug the disk.
Only if you have enough cpu-power. Try to reconstruct array on heavily loaded db-server with no core reserved for mdadm. And look what it does to your i/o...

It's the same.
No, and you know it very well. For lilo/grub to work with mdadm, you need special configuration. And you need some tweak in bios (to be able to boot from 2nd disk if the 1st fails). For hw-raid array it is like a normal single disk. No special precaution is necessary.

Ok, but these "drawbacks" are the same with ZFS.
Come on, you can not compare mdadm with zfs! Yes, there are some drawbacks common for mdadm/zfs, but in case of zfs, it is more than generously compensated by benefits mdadm can not provide. *IF* I'm using software-raid these days, it's zfs then. But there are still cases when I prefer good hw-raid...
 
Did it generage "negligible" load? I doubt. Calculating parity with that much drives takes some cpu-time. And what's even worse, if you do not have cpu/core reserved exclusively for mdadm, other apps will be competing for cpu-time and slowing down i/o. Not problem for lightly/moderately loaded server with plenty of cpu-time to spare, but big problem for production server with heavy load...
 
ZFS doesn't have to recalculate parity?
It's the same, but ZFS has also to calculate the checksum, thus is heavier than mdadm in this regard.

It's the same. Not everyone are interested (or can safely use) in ZFS features, thus mdadm could be a lighter alternative
 
Again and this is my last post here in this thread: please use mdraid if it fits into your use case, as its easy to configure for you, just do it.

But accept that we do not add it our installer as it does not fit for us for obvious reasons.
 
wow.

ZFS doesn't have to recalculate parity?
It's the same, but ZFS has also to calculate the checksum, thus is heavier than mdadm in this regard.

It's the same. Not everyone are interested (or can safely use) in ZFS features, thus mdadm could be a lighter alternative

just wow.

Its one thing to stubbornly continue to repeat the same mantra, regardless of its technical or any other merit; its another to loudly insist that others accept your point of view. I accept that you consider yourself an authority on storage, but unless you can convince others (more specifically, the devs of this particular software package who have proven their qualifications by providing you a product you are using and not the other way around) of your bone fides and why they are superior I see no reason to pay any attention to you.

I willl summarize your argument thusly: I like mdadm and so you (Proxmox devs) must support it. the devs (in a great show of patience) explained that you are free to deploy it if you must but mdadm is not production quality software, and therefore not within the scope of the supported product.

Why is any further discussion even necessary?
 
ZFS doesn't have to recalculate parity?
It's the same...

I did not compare mdadm vs zfs, but mdadm vs hw-raid, because your statement was:
...Every part of mdadm is better than any hardware card.
That's simply not true. Both mdadm and hw-raid (and zfs-raid) have advantages and disadvantages. Period.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!