Install on DELL R630 with Hardware RAID

[jHorrocks]

I have a DELL Poweredge R630 with the current setup:

CHIP: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 24 cores
RAM: 64GB
STORAGE: 10x 1.2TB SAS drives for RAID 10 DELL PERC H730 Mini gives me around 6TB in RAID 10


I will format this server and install proxmox. The confussion is from watching so many different youtube vdo's, reading blogs etc... in how people are installing proxmox that now i am unsure the BEST way to install proxmox on a DELL server with hardware RAID.

Option 1:
In the DELL PERC RAID admin section of the server do I add all disks to RAID 10 like I have at the moment and then when I install proxmox, install it as xfs on the full Hardware RAID drive that setup will see

Option 2:
In the DELL PERC RAID admin section of the server do I not add any RAID configuration and then when I install proxmox, install xfs as RAID and select all the 10 drives (so basically not use hardware RAID)

Option 3:
Add an SSD to the server and install proxmox on this drive and then when i launch the web ui, create a new xfs volume using the SAS drives



If option 3, then which option do I choose from the DELL PERC RAID admin section, do I leave all disks single disks or do I select Hardware RAID 10.


Long question I know but I see many people install different ways so not clear on the BEST Way.

Any advice would be greatly apprciated.

Regards

J

 

[Richard]

I have a DELL Poweredge R630 with the current setup:

CHIP: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 24 cores
RAM: 64GB
STORAGE: 10x 1.2TB SAS drives for RAID 10 DELL PERC H730 Mini gives me around 6TB in RAID 10


I will format this server and install proxmox. The confussion is from watching so many different youtube vdo's, reading blogs etc... in how people are installing proxmox that now i am unsure the BEST way to install proxmox on a DELL server with hardware RAID.

Option 1:
In the DELL PERC RAID admin section of the server do I add all disks to RAID 10 like I have at the moment and then when I install proxmox, install it as xfs on the full Hardware RAID drive that setup will see

Option 2:
In the DELL PERC RAID admin section of the server do I not add any RAID configuration and then when I install proxmox, install xfs as RAID and select all the 10 drives (so basically not use hardware RAID)

Option 3:
Add an SSD to the server and install proxmox on this drive and then when i launch the web ui, create a new xfs volume using the SAS drives



If option 3, then which option do I choose from the DELL PERC RAID admin section, do I leave all disks single disks or do I select Hardware RAID 10.


Long question I know but I see many people install different ways so not clear on the BEST Way.

Any advice would be greatly apprciated.

Regards

J

Everything of the above will work, what is the best option depends also on the expectations and needs you have. Assuming you plan a stand alone server and you want to have data security I would (and this is last but not least also a matter of personal approach, i.e. what you are most familiar with) select Option3 and for data Option2, i.e. use Hardware Raid 10 for VM disk Storage type "Directory" or "LVM thin" (NOT ZFS! This must not be combined with hardware RAID, even not when running it in jbod mode).

 

[jHorrocks]

Everything of the above will work, what is the best option depends also on the expectations and needs you have. Assuming you plan a stand alone server and you want to have data security I would (and this is last but not least also a matter of personal approach, i.e. what you are most familiar with) select Option3 and for data Option2, i.e. use Hardware Raid 10 for VM disk Storage type "Directory" or "LVM thin" (NOT ZFS! This must not be combined with hardware RAID, even not when running it in jbod mode).

Ok I think I get it. Reading on the internet looks like I cannot use DELL's PERC hardware raid controller at all right? Even if I select JBOD.

I remember somewhere seeing someone's post that they changed the DELL RAID controller to a basic PCI HD controller (Sorry I am a software engineer not a hardware guy so not sure what you call it) so that the SSD is the install disk for proxmox and then once in the admin web UI of proxmox create a zfs with the disks not used by proxmox or DELL raid, is this correct?

Greatly appreciate your help on this as I dont want to switch over to proxmox and then realise later I have the base server configured wrong.

Thanks buddy for your previous reply, just a few clarifications would be helpful.

 

[Richard]

Reading on the internet looks like I cannot use DELL's PERC hardware raid controller at all right? Even if I select JBOD.

When you want to use zfs - but zfs is not a must, use Hardware RAID and on top of it "LVM[-thin]" or "Directory" Storage is not a problem.

I remember somewhere seeing someone's post that they changed the DELL RAID controller to a basic PCI HD controller (Sorry I am a software engineer not a hardware guy so not sure what you call it) so that the SSD is the install disk for proxmox and then once in the admin web UI of proxmox create a zfs with the disks not used by proxmox or DELL raid, is this correct?

Correct!

 

[jHorrocks]

Thanks Richard I understand 100% now, appreciate your time.

The only reason I would like ZFS is that the replication from the online production server can sync easy with ZFS but I have tested with LVM thin but it doesnt allow replication so how do I sync Production server to Office server with LVM Thin. I know I can backup but just one VM has an image for /home directory at 3TB so taking a backup of that will kill the system right?

 

[jHorrocks]

I might have found a solution:

As per the following link, it looks like I can disable hardware RAID controller on a DELL R630 to have single disks so that means I can install proxmox to a Enterprise SSD and then in the proxmox webUI see each individual HDD disk so that I can create ZFS from these disks.... is this correct?

https://www.dell.com/community/Powe...sabling-RAID-controller-PERC-5-i/td-p/2556627

 

[Dunuin]

I might have found a solution:

As per the following link, it looks like I can disable hardware RAID controller on a DELL R630 to have single disks so that means I can install proxmox to a Enterprise SSD and then in the proxmox webUI see each individual HDD disk so that I can create ZFS from these disks.... is this correct?

https://www.dell.com/community/Powe...sabling-RAID-controller-PERC-5-i/td-p/2556627

It depends...often those raid controllers create a JBOD or single disk raid0. This is not what you want with ZFS. You want it to operate in initiator-target mode (IT-mode) so a dumb HBA without any abstraction layer between ZFS and the disks. See here why using HW raid with ZFS isn'T a good idea: https://openzfs.github.io/openzfs-docs/Performance and Tuning/Hardware.html#hardware-raid-controllers

So you better verify that the raid is really entirely disabled.

There are some PERCs that can be crossflashed from IR-mode to IT-mode, but it looks like your PERC 730 Mini is not one of them: https://fohdeesha.com/docs/perc.html

 

[jHorrocks]

Thanks for you reply. This is great Into so I will look into this and get beck to you all.

The problem is i dont have my DELL R630 at my office to test as it is production so I am assuming things will work until I get my hands on it and do some tests.

 

[skillet54]

I was wondering if you found a solution to this. I have a Dell R530 with the same raid controller. I updated all my firmware and was able to switch it to HBA mode in the BIOS. I'm seeing my 7 HDD's in Proxmox but not my 1 SSD. I've used the lsblk command but it doesn't show there either. Did you have any luck? Thanks

 

[JeanS72]

I am having Dell Poweredge R720 with Perc H710 Mini. I made all HDDs as Raid0 invidually (i.e. each HDD as own Raid0 Virtual Disk). On Proxmox I made ZFS - Raid-Z2. How risky this setup is?

However, VMs on R720 are not so critical, I can redo them if there comes issues -> is it worth to do anything?

 

[rosmaniac]

Not all Dell PERC hardware RAID controllers are created equal. Because of that fact, the blanket, generic, advice is to just not use a hardware RAID controller of any kind, since the EXACT model and EXACT firmware version matters.

The H700, for instance, cannot be used in 'IT' mode at all, although you can disable caching on a disk-by-disk basis and run RAID0. Not recommended, but does work. I did that for a little while on an R515 with an H700 so that I could get some of the advantages of ZFS (replication and snapshotting, to be precise) until I could get bigger drives and an H200 in it; I made that choice fully aware of the risks involved and the recommendations against it. It worked long enough and well enough for what I needed to do at that particular time. But it is NOT recommended.

H710 is a different beast entirely, and CAN be flashed to IT mode. While I have not tried it, I don't believe disk pass-through (as opposed to single-disk RAID0 or even what some controllers call JBOD) is available with stock firmware. Art of Server sells IT-Mode H710's on eBay and talks about them on his youtube channel; you can do this flash yourself, but it is a tedious process; what AoS is charging is cheap compared to the time it takes to do the work yourself, plus you keep your original controller if needed.

The H730 of the OP is much more flexible, with disk passthrough non-RAID mode (concurrent with other disks in RAID volumes) available, like the H200 in IR mode will do. The passthrough mode exposes the raw, low-level disk with all SMART data and other direct ioctls available, just like an IT mode HBA.

The H200 will do a true passthrough with no caching and no RAID while allowing for a simultaneous RAID1 boot volume; I posted yesterday about my fun migrating from an H700 to an H200 with this exact configuration. Again, with the H200 in IR mode you get all SMART data, no caching, direct initiator target access to non-RAID disks, no difference can be seen at all from the ZFS point of view. From ZFS's point of view, you have an IT-mode controller plus a RAID controller on one board. (the kernel driver for the H200 in IR mode is mpt3sas, not megaraidsas like with the H700).

root@nessie:~# lspci -v -s 03:00.00
03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
    DeviceName: Integrated SAS                    
    Subsystem: Dell PERC H200 Integrated
    Flags: bus master, fast devsel, latency 0, IRQ 38
    I/O ports at fc00 [size=256]
    Memory at df2b0000 (64-bit, non-prefetchable) [size=64K]
    Memory at df2c0000 (64-bit, non-prefetchable) [size=256K]
    Expansion ROM at df100000 [disabled] [size=1M]
    Capabilities: [50] Power Management version 3
    Capabilities: [68] Express Endpoint, MSI 00
    Capabilities: [d0] Vital Product Data
    Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [138] Power Budgeting <?>
    Kernel driver in use: mpt3sas
    Kernel modules: mpt3sas

root@nessie:~# sas2ircu list
LSI Corporation SAS2 IR Configuration Utility.
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2009-2013 LSI Corporation. All rights reserved.


         Adapter      Vendor  Device                       SubSys  SubSys
 Index    Type          ID      ID    Pci Address          Ven ID  Dev ID
 -----  ------------  ------  ------  -----------------    ------  ------
   0     SAS2008     1000h    72h   00h:03h:00h:00h      1028h   1f1eh
SAS2IRCU: Utility Completed Successfully.
root@nessie:~# sas2ircu 0 display
LSI Corporation SAS2 IR Configuration Utility.
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2009-2013 LSI Corporation. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS2008
  BIOS version                            : 7.11.10.00
  Firmware version                        : 7.15.08.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 39
  Concurrent commands supported           : 2607
  Slot                                    : 0
  Segment                                 : 0
  Bus                                     : 3
  Device                                  : 0
  Function                                : 0
  RAID Support                            : Yes
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
IR volume 1
  Volume ID                               : 79
  Status of volume                        : Okay (OKY)
  Volume wwid                             : 065fe6b7f840e93b
  RAID level                              : RAID1
  Size (in MB)                            : 915200
  Physical hard disks                     :
  PHY[0] Enclosure#/Slot#                 : 1:0
  PHY[1] Enclosure#/Slot#                 : 1:1
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 0
  SAS Address                             : 4433221-1-0700-0000
  State                                   : Optimal (OPT)
  Size (in MB)/(in sectors)               : 915715/1875385007
  Manufacturer                            : ATA
  Model Number                            : SAMSUNG MZ7LM960
  Firmware Revision                       : 104Q
  Serial No                               : S2TZNX0HB13926
  GUID                                    : 5002538c404766e0
  Protocol                                : SATA
  Drive Type                              : SATA_SSD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 1
  SAS Address                             : 4433221-1-0600-0000
  State                                   : Optimal (OPT)
  Size (in MB)/(in sectors)               : 915715/1875385007
  Manufacturer                            : ATA
  Model Number                            : SAMSUNG MZ7LM960
  Firmware Revision                       : 104Q
  Serial No                               : S2TZNX0HB15247
  GUID                                    : 5002538c40476e7f
  Protocol                                : SATA
  Drive Type                              : SATA_SSD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 2
  SAS Address                             : 4433221-1-0500-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 915715/1875385007
  Manufacturer                            : ATA
  Model Number                            : SAMSUNG MZ7LM960
  Firmware Revision                       : 104Q
  Serial No                               : S2TZNX0HB14272
  GUID                                    : 5002538c4047683a
  Protocol                                : SATA
  Drive Type                              : SATA_SSD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 3
  SAS Address                             : 4433221-1-0400-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 915715/1875385007
  Manufacturer                            : ATA
  Model Number                            : SAMSUNG MZ7LM960
  Firmware Revision                       : 104Q
  Serial No                               : S2TZNX0HB15245
  GUID                                    : 5002538c40476e7d
  Protocol                                : SATA
  Drive Type                              : SATA_SSD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 4
  SAS Address                             : 4433221-1-0300-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 915715/1875385007
  Manufacturer                            : ATA
  Model Number                            : SAMSUNG MZ7LH960
  Firmware Revision                       : 304Q
  Serial No                               : S45NNE0M310866
  GUID                                    : 5002538e000d13de
  Protocol                                : SATA
  Drive Type                              : SATA_SSD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 5
  SAS Address                             : 4433221-1-0200-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 915715/1875385007
  Manufacturer                            : ATA
  Model Number                            : SAMSUNG MZ7LM960
  Firmware Revision                       : 104Q
  Serial No                               : S2TZNX0HB14270
  GUID                                    : 5002538c40476838
  Protocol                                : SATA
  Drive Type                              : SATA_SSD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 6
  SAS Address                             : 4433221-1-0100-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 915715/1875385007
  Manufacturer                            : ATA
  Model Number                            : SAMSUNG MZ7LM960
  Firmware Revision                       : 104Q
  Serial No                               : S2TZNX0HB14318
  GUID                                    : 5002538c40476868
  Protocol                                : SATA
  Drive Type                              : SATA_SSD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 7
  SAS Address                             : 4433221-1-0000-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 915715/1875385007
  Manufacturer                            : ATA
  Model Number                            : SAMSUNG MZ7LM960
  Firmware Revision                       : 204Q
  Serial No                               : S3B8NX0J300605V
  GUID                                    : 5002538c4055717c
  Protocol                                : SATA
  Drive Type                              : SATA_SSD

Device is a Enclosure services device
  Enclosure #                             : 1
  Slot #                                  : 9
  SAS Address                             : 5e4ae02-0-9015-8600
  State                                   : Standby (SBY)
  Manufacturer                            : DP
  Model Number                            : BACKPLANE  
  Firmware Revision                       : 1.07
  Serial No                               : 1B100OF
  GUID                                    : N/A
  Protocol                                : SAS
  Device Type                             : Enclosure services device
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#                              : 1
  Logical ID                              : 5d4ae520:90158600
  Numslots                                : 9
  StartSlot                               : 0
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.
root@nessie:~#

But the only solution that is recommended and supported by Proxmox is the IT mode HBA non-RAID controller. Doesn't mean the other solutions won't work, just means they won't be unsupported. Do this at your own risk, in other words. But it's a bit of a catch-22: Dell won't support PERCs reflashed to IT mode, either.

EDIT: the sas2ircu utility can be obtained directly from LSI or you can use the packages for 'buster' from the hwraid.le-vert.net package repository. Yes, I know PVE 7.4 is bullseye; the buster sas2ircu will work fine until the maintainer gets around to updating the bullseye repository with packages. Just download and install the one package, no need to enable the whole repository to get it.

As to the specific risks of a group of RAID0 single-disks, as I understood the risks ZFS can't know the actual data status of the disk media (solid state or spinning platter) if there is any form of write caching going on. With direct disk access ZFS can at least request write caching to be disabled; that may not be the case with a single-disk RAID0. Disable the write cache manually in the LSI configuration utility. And read and understand the openzfs.github.io page linked above, as it contains a lot of good information on risks beyond just the write cache issue.

 

[_gabriel]

On Proxmox I made ZFS - Raid-Z2. How risky this setup is?
However, VMs on R720 are not so critical, I can redo them if there comes issues -> is it worth to do anything?

why ZFS if not critical ?

 

[JeanS72]

I am having 13 * 600GB 2.5" HDDs in this ZFS Raid-Z2. Performance is surprisingly good even if these are old HDDs.