HowTo: Proxmox VE 7 With Software RAID-1

Apr 29, 2016
10
4
23
36
Introduction

The operating system of our servers is always running on a RAID-1 (either hardware or software RAID) for redundancy reasons. Since Proxmox VE 7 does not offer out-of-the-box support for mdraid (there is support for ZFS RAID-1, though), I had to come up with a solution to migrate the base installation to an mdraid RAID-1 and thought this might be useful for others as well. Therefore, I am sharing this as a howto here.

High-level explanation

Basically, I install PVE7 on /dev/sda, then create a degraded(!) software RAID-1 on /dev/sdb3, move the existing PV from /dev/sda3 to /dev/md0 and then join the mdraid /dev/md0 consisting only of /dev/sdb3 at first with /dev/sda3.

Step-by-step instructions

Code:
# Install mdadm
apt install mdadm
 
# Create identical partition layout to /dev/sda on /dev/sdb without copying labels/UUIDs
sfdisk -d /dev/sda > part_table
grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
 
# Create a degraded RAID-1 on /dev/sdb3
mdadm --create /dev/md0 --level 1 --raid-devices 2 /dev/sdb3 missing
 
# Create LVM PV and add to existing VG
pvcreate /dev/md0
vgextend /dev/pve /dev/md0
 
# Move data from /dev/sda3 to /dev/md0
pvmove /dev/sda3 /dev/md0
 
# IMPORTANT! Wait for the process to finish, then remove /dev/sda3 from the VG
vgreduce /dev/pve /dev/sda3
 
# Add /dev/sda3 to RAID-1
mdadm --manage --add /dev/md0 /dev/sda3
 
# Byte copy EFI and BIOS boot partitions
dd if=/dev/sda1 of=/dev/sdb1
dd if=/dev/sda2 of=/dev/sdb2
 
# Install GRUB on all disks
grub-install /dev/sda
grub-install /dev/sdb
 
# Update initramfs
update-initramfs -u -k all
 
# Wait for RAID-1 sync to finish
watch cat /proc/mdstat
 

ness1602

Well-Known Member
Oct 28, 2014
292
37
48
Serbia
Shouldn't you use UUID or something else than disk name, because the name can change during restart. or when adding disks?
How are you doing scrubbing and error reparing?
 
Apr 29, 2016
10
4
23
36
By default mdraid scans all disks and partitions for MD superblocks (metadata) on boot. It then checks for a match of the array name or UUID (depending on your config in /etc/mdadm/mdadm.conf) of the RAID (not the disk UUID!). Therefore, it does not matter what device name the disk receives after a reboot.

Regarding scrubbing and error reparing: When installing mdadm, a monthly cronjob will be added as well to ensure consistency.

Code:
root@proxmox:~# cat /etc/cron.d/mdadm
#
# cron.d/mdadm -- schedules periodic redundancy checks of MD devices
#
# Copyright © martin f. krafft <madduck@madduck.net>
# distributed under the terms of the Artistic Licence 2.0
#

# By default, run at 00:57 on every Sunday, but do nothing unless the day of
# the month is less than or equal to 7. Thus, only run on the first Sunday of
# each month. crontab(5) sucks, unfortunately, in this regard; therefore this
# hack (see #380425).
57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi
 
Last edited:

afi

New Member
Feb 16, 2021
2
0
1
36
France
Hello, I come here to throw a bottle into the sea.

It is not possible to apply the mdadm procedure for Proxmox ve 7.1-8 after the step :

Code:
# Move data from / dev / sda3 to / dev / md0
pvmove / dev / sda3 / dev / md0

We have an error message:

Code:
  Insufficient free space: 127871 extents needed, but only 127839 available
  Unable to allocate mirror extents for pve / pvmove0.
  Failed to convert pvmove LV to mirrored.

We have tested on several PVE 7.1-8 servers it is identical.

We tested with the old updated PVE version 6.4-13 and it works fine!

We don't understand, probably an LVM problem maybe, but why it doesn't work anymore?

Code:
root @ PVE7: ~ # pvdisplay
  --- Physical volume ---
  PV Name / dev / sda3
  VG Name pve
  PV Size <499.50 GiB / not usable 2.98 MiB
  Allocatable yes (but full)
  PE Size 4.00 MiB
  Total PE 127,871
  Free PE 0
  Allocated PE 127871
  PV UUID HLHwID-HmkC-JFdX-HkXd-ngXk-Sboc-p2fK42

  --- Physical volume ---
  PV Name / dev / md0
  VG Name pve
  PV Size 499.37 GiB / not usable <1.94 MiB
  Allocatable yes
  PE Size 4.00 MiB
  Total PE 127,839
  Free PE 127839
  Allocated PE 0
  PV UUID XQUEDm-swYp-j102-H0x1-kDih-qYPq-6vsGTF

root @ PVE7: ~ # pvmove / dev / sda3 / dev / md0
  Insufficient free space: 127871 extents needed, but only 127839 available
  Unable to allocate mirror extents for pve / pvmove0.
  Failed to convert pvmove LV to mirrored.
root @ PVE7: ~ #

How should we proceed ?

Thanks for your help.
 

Emilien

Member
Jan 23, 2022
121
7
18
Italy
Introduction

The operating system of our servers is always running on a RAID-1 (either hardware or software RAID) for redundancy reasons. Since Proxmox VE 7 does not offer out-of-the-box support for mdraid (there is support for ZFS RAID-1, though), I had to come up with a solution to migrate the base installation to an mdraid RAID-1 and thought this might be useful for others as well. Therefore, I am sharing this as a howto here.
....
I suggest to make a raid1 of boot partition sda2 with sdb2.

I had also many troubles with UEFI boot bios menu entry, at the end I overwrite the UEFI bootloader fallback with proxmox UEFI:
cp /boot/efi/EFI/proxmox/grubx64.efi /boot/efi/EFI/BOOT/BOOTx64.EFI
*todo on both boot partition sda2 and sdb2.

Best regards,

Emilien
 

afi

New Member
Feb 16, 2021
2
0
1
36
France
I finally started from a debian 11 installation, without LVM in EXT4.

20GB for system in raid 1 (ext4) mdadm for /
A little swap...
and the rest in raid 1 madadm (ext4) for /var/lib/vz

Yes for the raid you have to copy the *.EFI file on the two VFAT partitions for the UEFI disks (Sda1 and Sdb1) and of course create the two entries in the bios (if the UUID disks are not cloned) .
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!