mdadm array - How to use as storage?

M

MGSteve

New Member
Nov 1, 2023
16
1
3
Ok, Ok, I know from reading copious threads, mdadm is not officially supported, however I have a good reason for choosing it.

I've got an Dell R640 with 7 NVMe drives and BOSS card with 2xM2 SSD for the OS. The R640 have no hardware RAID support for the NVMe drives and only RAID-1 support from the onboard software RAID under Linux.

I don't want RAID1, so access is via the AHCI driver.

If I benchmark 1 drive on it's own, under Ubuntu 22.04 I get a read of around 2g/s and write of 3g/s. On a Raid 10 array of 6 drives & 1 spare, I gets 5gb/s read & write.

Under ZFS ZRaid2 under Proxmox I get around 2g/s read & write. This is a massive performance hit that I do not want to take, hence the use of mdadm.

I've setup the mdadm array in the console on the Proxmox server, but how do I get Proxmox to show it as a storage location?

I'm not interested in ZFS, so please don't reply saying to use that instead. Once you add a VM, the benchmark speed using ssd-benchmark was 300mb/s! Dire.

I know I can reinstall using debian and install onto the Raid array, which I'd rather not do as it makes the BOSS card useless and I like the idea of separating the OS from the VM data.

I have the array mounted in /mnt/vm_store, but can of course move it to a better location.

Regards

Steve.
 
Once you have md device, you have two choices:
a) Create an LVM based storage object for PVE
b) Create a Directory based storage object for PVE

https://pve.proxmox.com/wiki/Logical_Volume_Manager_(LVM)
https://pve.proxmox.com/wiki/Storage:_Directory

One is a block storage, the other is file based. There are many articles on pros and cons for each. Just use /dev/md as target.
If you decide to go for LVM, you will need to get rid of the mount and filesystem you placed on it.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Jeremy E and slidermike
MGSteve said:
Ok, Ok, I know from reading copious threads, mdadm is not officially supported, however I have a good reason for choosing it.
Click to expand...

Same - I'm trying to mount a drive that came out of a UNAS-Pro (it was the only drive in the pool, so it should be configured as RAID1 even though its target is RAID5).

MGSteve said:
I've setup the mdadm array in the console on the Proxmox server, but how do I get Proxmox to show it as a storage location?
Click to expand...

How, please enlighten me obewan... I cannot get mdam to install, I get:

root@pve:~# apt install mdadm
Package mdadm is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
Error: Package 'mdadm' has no installation candidate

I'm an old-school Red Hat user, so I don't know how to find out what other package 'refers' to mdadm.
There is no /sbin/mdadm, though there is in an Ubuntu LXC but I want to mount this disk/drive/volume under PVE and use it in another LXC.

[EDIT] I've since found that trying to setup md in this Ubuntu LXC is pointless as there's no md_mod in the underlying hypervisor (PVE).

MGSteve said:
I know I can reinstall using debian and install onto the Raid array, which I'd rather not do [SHORTENED]
Click to expand...

I already have PVE installed and various LXC/VMs running so don't want to have to destroy what I've already built just to install again over the top of Debian.

Jeremy.
 
Last edited:
Jeremy E said:
How, please enlighten me obewan... I cannot get mdam to install, I get:
Click to expand...
Try:
Code: 
apt update
apt-cache policy mdadm

Check your repositories:
Code: 
cat /etc/apt/sources.list
ls -l /etc/apt/sources.list.d/

You should have these enabled:
Code: 
Types: deb
URIs: http://deb.debian.org/debian/
Suites: trixie trixie-updates
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb
URIs: http://security.debian.org/debian-security/
Suites: trixie-security
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Cheers

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: waltar and news
Thanks bbgeek17:

root@pve:~# apt update
Hit:1 http://deb.debian.org/debian trixie InRelease
Get:2 http://security.debian.org/debian-security trixie-security InRelease [43.4 kB]
Get:3 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Get:4 http://security.debian.org/debian-security trixie-security/main amd64 Packages [269 kB]
Get:5 http://security.debian.org/debian-security trixie-security/main Translation-en [166 kB]
Get:6 http://download.proxmox.com/debian/pve trixie InRelease [4,357 B]
Fetched 531 kB in 1s (730 kB/s)
All packages are up to date.
root@pve:~# apt-cache policy mdadm
mdadm:
Installed: (none)
Candidate: (none)
Version table:
4.4-11 -1
-1 http://deb.debian.org/debian trixie/main amd64 Packages
root@pve:~# cat /etc/apt/sources.list
root@pve:~# ls -l /etc/apt/sources.list.d/
total 16
-rw-r--r-- 1 root root 178 Apr 23 09:43 ceph.sources
-rw-r--r-- 1 root root 365 Nov 19 2025 debian.sources
-rw-r--r-- 1 root root 162 Apr 23 09:39 proxmox.sources
-rw-r--r-- 1 root root 175 Apr 23 09:39 pve-enterprise.sources
root@pve:~# cat /etc/apt/sources.list.d/debian.sources
Types: deb
URIs: http://deb.debian.org/debian/
Suites: trixie trixie-updates
Components: main contrib non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb
URIs: http://security.debian.org/debian-security/
Suites: trixie-security
Components: main contrib non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
root@pve:~# apt install mdadm
Package mdadm is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Error: Package 'mdadm' has no installation candidate

What am I missing?
 
APT can see the package, but its priority is -1, which prevents it from being selected for installation. Something in your APT configuration is assigning mdadm a negative pin priority.

You can try:
Code: 
grep -RniE 'mdadm|Pin-Priority: *-1|Pin:.*(version|release)' \
    /etc/apt/preferences /etc/apt/preferences.d/ \
    /etc/apt/apt.conf /etc/apt/apt.conf.d/ 2>/dev/null

apt-config dump | grep -iE 'Default-Release|Pin'
apt-cache policy


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: waltar
bbgeek17 said:
You can try:
Click to expand...
root@pve:~# grep -RniE 'mdadm|Pin-Priority: *-1|Pin:.*(version|release)' \
/etc/apt/preferences /etc/apt/preferences.d/ \
/etc/apt/apt.conf /etc/apt/apt.conf.d/ 2>/dev/null
/etc/apt/preferences.d/block-packages.pref:2:Pin: release *
/etc/apt/preferences.d/block-packages.pref:3:Pin-Priority: -1
root@pve:~# apt-config dump | grep -iE 'Default-Release|Pin'
root@pve:~# apt-cache policy
Package files:
-1 /var/lib/dpkg/status
release a=now
-1 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
release o=Proxmox,a=stable,n=trixie,l=Proxmox Debian Repository,c=pve-no-subscription,b=amd64
origin download.proxmox.com
-1 http://security.debian.org/debian-security trixie-security/non-free-firmware amd64 Packages
release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=non-free-firmware,b=amd64
origin security.debian.org
-1 http://security.debian.org/debian-security trixie-security/contrib amd64 Packages
release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=contrib,b=amd64
origin security.debian.org
-1 http://security.debian.org/debian-security trixie-security/main amd64 Packages
release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=main,b=amd64
origin security.debian.org
-1 http://deb.debian.org/debian trixie-updates/main amd64 Packages
release v=13-updates,o=Debian,a=stable-updates,n=trixie-updates,l=Debian,c=main,b=amd64
origin deb.debian.org
-1 http://deb.debian.org/debian trixie/non-free-firmware amd64 Packages
release v=13.7,o=Debian,a=stable,n=trixie,l=Debian,c=non-free-firmware,b=amd64
origin deb.debian.org
-1 http://deb.debian.org/debian trixie/contrib amd64 Packages
release v=13.7,o=Debian,a=stable,n=trixie,l=Debian,c=contrib,b=amd64
origin deb.debian.org
-1 http://deb.debian.org/debian trixie/main amd64 Packages
release v=13.7,o=Debian,a=stable,n=trixie,l=Debian,c=main,b=amd64
origin deb.debian.org
Pinned packages:
root@pve:~#

Ummm, I didn't notice the -1 (or understand it's meaning) earlier, and don't know how to change this (or how it got like this if it's abnormal)... (softly) help?
 
Last edited:
Jeremy E said:
Ummm, I didn't notice the -1 (or understand it's meaning) earlier, and don't know how to change this (or how it got like this if it's abnormal)... (softly) help?
Click to expand...
Uhh, I wonder if my attempts (I encountered too many hurdles) to pin a later NUT than what comes with Trixie has altered things I wasn't aware of.
 
yes, you are pinning every repository at -1 which blocks every upgrade..
 
  • Like
Reactions: Johannes S
fabian said:
yes, you are pinning every repository at -1 which blocks every upgrade..
Click to expand...
Great, thanks (for that information) - I didn't make these changes intentionally, how do revert them?

root@pve:~# cat /etc/apt/preferences.d/block-packages.pref
Package: *
Pin: release *
Pin-Priority: -1
root@pve:~#

Should I just remove this 'block-packages.pref'?

[EDIT] I've moved it to ~/ and apt-cache policy now tells me most things are at 500; holy crap I have a lot of updates that have been missed to install.

Thanks all! (PHUK-I-PHEAL-STOOPEED)
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back